The Burgle Bug Fairness Principle

Beeminder’s bug classification system is like so:

1. Bitty Bugs are barely bothersome. [1]
2. Baneful Bugs make Beeminder blatantly wrong, [2] but not in any breach-of-contract way, unlike…
3. Bum-steer Bugs which may make you derail by leading you astray about the state of your graph, [3] or, worse:
4. Bamboozle Bugs making our marketing mendacious or fallacious. [4] And finally, the truly unconscionable,
5. Burgle Bugs which would charge you money you didn’t implicitly agree to pay! [5]

The Burgle Bug Fairness Principle is about the most severe kind. It goes like this: If we have a bug that incorrectly charges someone and we catch it, we can just refund it. If the user catches it — before we say anything to them — then we have to refund twice the amount we wrongly charged.

This doesn’t apply to things like non-legit derailments, even those caused by Bum-steer Bugs. Because even for an egregious Bum-steer Bug, the derailment did happen and we do explicitly ask “was this legit?” so there’s not much burgliness to it nor too much possibility for it to go unnoticed. (And to be clear, we do refund those!)

We’re blogging about this because it really irks us that the double-refund thing is not standard for all businesses. If someone makes an error in their own favor, it’s not remotely enough to be like “Oh, haha, sorry, here’s the money back that we would’ve happily stolen if you hadn’t noticed!”

Game Theory

“…we have to refund twice the amount we wrongly charged”

Ok, normal people can stop reading now. You know what the Burgle Bug Fairness Principle is and the fairness of it is perfectly intuitive so the rest of this post is just belaboring the point for the sheer academic fun of it.

The first academic question is what’s the justification for double refunds? Why not triple or just refund plus 10% or something? To directly go full-nerd on this, the refund should be multiplied by the reciprocal of the probability of the customer noticing — maybe calculated as the fraction of customers that notice. Of course we’re Bayesians so by the Principle of Indifference we could take 1/2 as the prior and thus double-refunds as a starting point.

Or we could just say double refunds because of focality AKA Schelling-pointiness and simplicity and not instantly glazing over the eyes of everyone we’re trying to convince about the Burgle Bug Fairness Principle!

But where does that “reciprocal of the probability” come from? Suppose we want to fairly redress the burglary in the sense of giving back all the money we burgled, in expectation. If 1 out of n users notice and we refund each of those users n times what we stole then overall, on average, we’ll have refunded 100% of the ill-gotten booty. Generalizing that means multiplying the reward — or the fine, from the burglar’s perspective — by one over the probability of getting caught.

(Should I pause again to clarify that we’re not intentionally stealing anyone’s money despite referring to ourselves as burglars and to burglary literally 17 times in this post? The last one we know of — at least where a user noticed before us — was years ago [5] and it’s far more common that our bugs short-change Beeminder rather than users.)

Of course our intention is to more-than-refund users who notice as well as exactly-refund everyone who didn’t notice. So we’re sticking with double refunds for those who notice before we do, rather than massive payouts for rarely noticed bugs. That still means giving back more than all of any wrongfully charged money.

 

Footnotes

[1] Examples of Bitty Bugs include #3539 and #3480. (Hover over any of these links to see the actual changelog entries from us fixing them.)

[2] Baneful Bugs include #3552 and #3541. Most of our bug fixes — grep “#bugfix” at beeminder.com/changelog — are probably somewhere in between Bitty and Baneful.

[3] A borderline Bum-steer Bug was #3567 which involved slightly misquoting a deadline.

[4] We designated #3547 and #3548 Bamboozle Bugs because there were paid premium features that couldn’t be used simultaneously.

[5] Examples of Burgle Bugs include #1001, #1084, and #2027. Also maybe barely #2035 and #2466. Interestingly, #700, #1290, and especially #3626 are negative Burgle Bugs, AKA Moneyfire Bugs. (Brainstorming b-words: bungle bugs, bezzle bugs, bloodbath bugs, beneficent bugs, bankbreaker bugs? But the real point of the bitty→baneful→bum-steer→bamboozle→burgle hierarchy is to classify severity from a user perspective. If we have a 💸 bug, that’s just Baneful if we’re failing to charge for premium and Bamboozle if we’re failing to charge for derailments — “taking your money” being the fundamental service Beeminder offers! — and we can further prioritize based on business reasons as needed.)

 

Thanks to Luke Barone-Adesi for the game theory discussion, Smitop for helping scrounge up examples of Burgle Bugs, Benjamin Fox for pointing out “Legal Systems Very Different From Ours” (see also the Astral Codex Ten review) which dives deep on the question of probability multipliers for deterring crimes like burglary, and to the others in the forum thread where this post originated.

Image credit: Faire Soule-Reeves.