Until today Beeminder had a fundamental design flaw that was baked in from literally day one. The first line of code for what would become Beeminder was to draw a line on a graph in Mathematica from a target weight to a goal weight. But weight fluctuates, I thought to myself. Or maybe I said it out loud to Bee, to which she would’ve nodded sagely at the seemingly innocuous fact. So, we figured, let’s add some thickness to that line and say you just need to keep your weight measurements on that thicker line or below it. We can color it yellow and call it the Yellow Brick Road.
After killing baby Hitler, I’d say the next most important use for a time machine is to burst in on us at that moment and explain that, sure, weight fluctuates but you can’t just add thickness to either side of the line like that! The line drawn on the graph has to be the critical edge, the bright line that you can’t cross.
Do you see the problem with how we did it? We defined the yellow brick road — the path from where you are today to your goal — in terms of the centerline but then allowed a fudge factor known as “the wrong lane”. In the so-called wrong lane, you’d be off from the goal you set for yourself but within a window that Beeminder still considered “on the road”. Besides violating a fundamental tenet of Beeminder it yielded a cascade of bugs and confusion and flat-out wrongness. Like how if you dialed in a goal to do X things by time T, Beeminder enforced something slightly but infuriatingly different.
What does all this mean for actual users?
For one thing, let me make that last example more concrete. Until now, if you wanted to, say, write 50,000 words in November, you’d end up only having written 48,333 words by the night of the 30th if you were edge-skating. Now there’s no more falling behind into that spare lane-width of leeway. It means we don’t need three numbers telling us the distance to this lane vs. that lane, just the one clear number that tells us where we are from today’s requirements.
Besides quashing whole categories of bugs and greater consistency and general elegance, though, you should notice surprisingly little! We’ve worked our butts off to make the transition silky smooth. We’ve even lovingly crafted the aesthetics of the new graphs to mimic, at least for now, the old version with lanes on a road.
So why are we telling you all this? Well, for one thing, there are so many moving parts that we want to warn you in case we break things (again [2]). And just because this is the blog of record for building Beeminder and this is a big deal for us, even if most of the work has involved minimizing the impact for you. But it should also mean paving the way for big new features in the future. Mostly, we’re really excited and wanted to share!
Can we get back to the excruciating details?
Sure! The really insidious problems arose when you changed the steepness of your yellow brick road. (And if you’re really just tuning in, that ability to change what you’ve committed to without defeating the whole point of a commitment device is another defining feature of Beeminder and you might want to read “The Road Dial and the Akrasia Horizon”.) Another fundamental tenet of Beeminder is letting you build up safety buffer and always being crystal clear about how much time you have before you’ll be officially off track on your goal, i.e., off your road. We color-coded that safety buffer — red for a beemergency, orange for beemergency eve, blue for 2 days of safety buffer, and green for 3+ days. But the colors were fundamentally based on the regions of the graph: green for the good side of the road, blue for the good lane, orange for the wrong lane, red for being off the road (you have to claw your way back on by your deadline — that’s what a beemergency means).
“We have finally peeled off the layers of duct tape and solved the root of these problems”
For the colors based on days of safety buffer and the colors based on location on the graph to match, the width of the road was computed such that you’d go from the good side (green) to the good lane (blue) to the wrong lane (orange) to the wrong side (red) each a day apart. The width had to be wider when the road was steeper and narrower when the road was shallower. Technically, if the road was flat then it should’ve had no thickness at all. But that was a big problem if you scheduled a flat spot on your road for a vacation. It would mean that you could be skating the edge of your road, in the wrong lane, and then when your flat spot mercifully arrived, the road would shrink out from under you and you’d have to do an extra day’s work to get back onto it. That was obviously no good so we made a special case: the width of the road on a flat spot was the max of the widths on either side of the flat spot. Fine. Except if you happened to schedule not a fully flat spot but just an almost flat spot, you’d be bitten in the butt by the shrinking road problem after all and it would all be a big bubbling pot of magic putrescence. (Maybe I’m overstating it now; we’re just so relieved to have finally peeled off the layers of duct tape and solved the root of these problems.)
More generally, you just couldn’t rely on the colors Beeminder showed you to correspond with your actual amount of safety buffer on your goal. Going into “the wrong lane” was perpetually messing that up on anything but the simplest graphs.
The only way out of the mess we’d built for ourselves was to redesign how the yellow brick road worked. We did this exceedingly carefully and painstakingly with the hopeful end result that casual users — in particular those with simple linear graphs — will hardly notice a difference.
It started with a more careful separation of concerns between what we call Beebrain and Beebody. We gradually got more and more logic — timezones, inferring the start of the yellow brick road, another misguided bit of magic involving weight loss leniency — out of Beebrain so it could focus just on drawing the graphs and computing statistics. The number of clusterfudgesicles — byproducts of forcing “lanes of the road” to work — that we had to gradually untangle was mildly staggering. [1]
Then there was killing off exponential roads, generalizing the “no-mercy” derails setting, fixing the infamous do-less loophole, killing off auto-widening roads, and killing off custom lane widths — all of which were prerequisites for the yellow brick half-plane. And of course a million backward-compatibility issues.
In parallel with all of that we were also keeping busy porting Beebrain from Python to Javascript. That’s about 16,000 new lines of code (including the forthcoming visual road editor), though a mercifully large chunk of it is about to be obliterated as soon as the last old-style graph is converted to yellow brick half-plane. Because, ultimately, behind the scenes, yellow brick half-plane is a dramatic simplification of how Beeminder works.
UPDATE: There’s another small change we need to mention, per the Pareto Dominance Principle: When you create a goal now, it defaults to just one day of initial safety buffer instead of two. (You can still pick any amount you like — this is just about the default if you don’t pick.) It’s a little more natural this way, to have the razor road start exactly where you are today, with your initial datapoint right on it.
UPDATE: On this day, June 18, in the year of our Lord 2020, the last old-style non-YBHP lane-ridden graph was converted to the Yellow Brick Half-Plane New World Order. Praise be! Also we finally thought to add before-and-after pictures to this post.
Footnotes
[1] The story of the parameter known as “edgy” is just gobsmacking and epitomizes the elaborate ball of duct tape we backed ourselves inside of by not having a yellow brick half-plane from the start. It’s too far into the weeds to be worth proper blogging but I’m including our dev notes about it in this footnote for posterity and just to marvel at:Suppose you start a pushups goal of 3/day but want the first 3 due on the first day. This means starting the centerline of the yellow brick road above the initial datapoint so the initial datapoint starts on the edge of the road. So an “edgy” goal is one where the initial point is on the bottom (or top, depending on “yaw”) edge of the road. That means moving the start of the road up (or down) by one lane width compared to where it would be if it started at the initial datapoint. But we have a chicken and egg problem: We need the road function in order to compute the lane width and we need the initial point, (tini,vini), to compute the road function. Which is to say, we need (tini,vini) to compute the lane width but we need the lane width to compute (tini,vini), to shift it.
Solution (sort of): The lane width is generally equal to the daily rate of the road. So we can have the road start at the initial datapoint but one day earlier. I.e., (tini,vini) -= (86400,0). Then when we know the road function we can move tini forward a day and vini by the appropriate amount, vini = rdf(tini+86400).
The only problem with this approach is that it only puts the initial datapoint at the edge if the rate of the first segment of the road is the same as the rate of the most recent datapoint, since it’s the rate there that determines the overall lane width. Trying to do this Really Right gets very messy or even impossible without introducing worse problems than the initial point not being on the actual edge.
I think the right solution to the ‘edgy’ confusion is to get rid of the edgy parameter — goals are always edgy. To get the edgy=false behavior, just use the road matrix to specify one initial flat day. And in fact goals should always have an initial road row that says rate 0 up till yesterday or today (because otherwise if you add a datapoint before the first datapoint then you’ll make the road change).
[later] Bethany has now convinced me that edginess should not be in Beebrain’s purview. The start of the road is always given explicitly by tini and vini. When you create a goal that should be edgy you know the initial rate (e.g., the generous estimate that the user provides for Set-A-Limit [now do-less] goals) so you can can bump vini up by that amount. So we’re going to do that! Phew!
[2] Here’s a story from last week as I related it in a daily beemail the next day:
Bee and I had a fun night last night. While working on the Yellow Brick Half-Plane transition (did you see the news that all new goals are YBHP now? key milestone! we couldn’t turn back now even if we wanted to!) we accidentally shrank the roads on 1200 graphs before they were ready which caused about 100 of them — the edge-skaters — to instantly derail. So we had to painstakingly fix them and email everyone to apologize and assure them that they could ignore the accidental legit check they got, etc etc. (We think there was only one person who was about to derail anyway and was able to opportunistically get out of it due to our screw-up. For everyone else we got things fixed up in time.) About 9 person-hours of work from one tiny bug. I guess we’ve seen a lot worse!
Image credit: Faire Soule-Reeves. Thanks to Kevin Lochner who, years ago, first said “Isn’t this yellow brick road more of a yellow brick half-plane?” to which we probably said “kind of but that sounds stupid” and thanks to @insti for first articulating the problem with lanes as we’d implemented them and suggesting the theoretically elegant solution of defining the road in terms of the critical edge. Biggest thanks to Uluç Saranlı — see his Beeminder blog post for even nerdier details on yellow brick half-plane. Last but not least, thanks to another Beeminder superfan, Kenn Hamm (@kenoubi on Beeminder), for a putting a bounty on this which we can now finally claim!