Really, Ariely?

Beeminder-relevant study turns out to be fake, womp womp. Ongoing discussion in the forum.

I originally wrote a draft of this post five years ago when Dan Ariely was first exposed as a fraud I mean alleged to be a fraud. But then I sat on it because it didn’t seem Beeminder-relevant enough. [1]

Well, yesterday, Data Colada dropped a new bombshell: Ariely’s famous study of commitment devices from 2002 is also fake, or, choosing our words carefully, the data was at least heavily tampered with. Also it doesn’t replicate.

The paper that’s presumably about to be retracted described two studies:

• Study 1: Students with externally imposed, evenly spaced deadlines did better than students who could pick their own deadlines. (Also that students did pick earlier-than-necessary deadlines, but not earlier enough. The idea was that students know they need commitment devices and in fact they need them more than they think.)
• Study 2: For a paid proofreading task, it’s the same story: people perform best with evenly spaced hard deadlines, worst with just one deadline for everything at the end, and in between for using a commitment device to commit to their own chosen deadlines.

Embarrassingly for me, I was entirely fooled by Ariely in 2008 [2] when I first read his popular book, Predictably Irrational, though I learned shortly after that (2009 or so) that he was dishonest, for reasons I can’t share publicly. Even more embarrassingly for me, I was still kind of defending the paper on commitment devices as late as two years ago:

In hindsight, I imagine the best parts of Predictably Irrational were describing other people’s research, not Ariely’s. Although, hmm, there’s at least the 2002 experiment where [Ariely] showed that students were willing to use, and got value from using, a commitment device for homework deadlines. That still seems valuable. I don’t know.

And later:

Despite Ariely’s involvement, this study still seems like solid research to me. Just has to be taken with a massive grain of salt now, obviously.

That was from a Beeminder Book Brigade for Annie Duke’s Thinking In Bets (and a later post on the science of Beeminder). I went on to say how I despise Ariely. He really is such a slimeball charlatan, it makes me furious.

But since one of Ariely’s coauthors sued Data Colada for 25 million dollars, let me carefully hedge and say that all that’s been shown is that the data for the commitment device study was either fabricated or severely tampered with and the metadata has “Last saved by Dan Ariely”. So whatever probability that implies on “Ariely faked it” is all I’m alleging. Plus the sky-high prior from the previous scandals. And of course I meant to say “alleged slimeball charlatan”. Alleged by me.

The good news is that there are still many Ariely-unaffiliated studies showing that commitment devices work, some of which even replicate:

• Deposit contracts for smoking cessation (Smoking Sticks and Carrots)
• Locking away money to prevent impulsive spending
• Monetary commitment devices for gym attendance
• Workplace data entry (discussed in footnote 1 of The Type Bee Personality)
• Sobriety incentives
• Weight loss

Related Reading

• Beeminder’s “Fake Data is the Devil” post (if you submit fake data to Beeminder, you’re basically Dan Ariely!)
• Andrew Gelman: “I am shocked, shocked to learn that this 2002 paper by Dan Ariely is based on dubious data and doesn’t replicate”
• Yet another example of Ariely being a bullshitting bullshitter, disparaging dentists on NPR
• Andrew Gelman on Ariely in 2021 about how Ariely willfully repeated on NPR the claim about honesty pledges after admitting it was bogus (not admitting to fraud, just that it didn’t replicate)
• Ariely’s response in 2021 speaks volumes in what it doesn’t say: no walking us through whatever clusterforkery happened with the datafile, no speculation about why/how the insurance company would have given him false data, what coincidences or conspiracies explain how it turned out that the fake data happened to be overwhelming evidence for the thesis of the paper which everyone, Ariely included, now admit was wholly bogus, etc etc. Mostly he emphasizes how this was “more than ten years ago”.
• Ariely’s reponse to the latest which says only that he was “made aware” his paper “contained serious anomalies” and that he lost his records and remembers nothing, but he supports retraction of the paper.

Footnotes

[1] I’ll relegate the part that’s no longer timely to this foonote. The story back in 2021 was that Ariely coauthored a study in 2012 about how some obviously insignificant thing about the when you get someone to sign an honesty pledge has some hugely significant effect on how much they cheat. This paper was a huge deal, published in Proceedings of the National Academy of Sciences, taught in MBA courses around the world, etc etc. Of course it didn’t replicate. Then Dan Ariely was caught basically red-handed fabricating the data for one of the studies (yes, for a paper about cheating — the irony is off the charts here) that all this was based on. Here’s my friend Chris Hacking’s hilarious summary of how it went down:

1. Take data set
2. Not big enough, duplicate all values with random variation (forget to round or match font).
3. Assigning experimental vs control group somehow too much like work to do randomly, somehow fuck that up in an obviously non-random way.
4. Measuring dependent variable is too much like work, generate it randomly (forget to round or match font).
5. Change headers on your data when it accidentally doesn’t show the result you wanted and you were too lazy to notice before.
6. Publish.

THE SCIENTIFIC METHOD!

[2] I was pretty oblivious to the replication crisis back then. I think most of us were but I remember my friend David Reiley (an economist and friend of Beeminder from the very early days) surprising me by admitting that there was a whole class of social science papers he simply dismissed as likely BS. It seemed kind of shockingly anti-science at the time but succeeded at planting important seeds of doubt for me.