Jonathan Altman

• Coding Horror: Groundhog Day, or, the Problem with A/B Testing

  • This posting deals largely with 37 Signals' concept of having opinionated software, but from the opposite end. "Phil wasn't making these choices because he honestly believed in them. He was making these choices because he wanted a specific outcome -- winning over Rita -- and the experimental data told him which path he should take. Although the date was technically perfect, it didn't ring true to Rita, and that made all the difference. That's the problem with A/B testing. It's empty. It has no feeling, no empathy, and at worst, it's dishonest. As my friend Nathan Bowers said: A/B testing is like sandpaper. You can use it to smooth out details, but you can't actually create anything with it. The next time you reach for A/B testing tools, remember what happened to Phil. You can achieve a shallow local maximum with A/B testing -- but you'll never win hearts and minds. If you, or anyone on your team, is still having trouble figuring that out, well, the solution is simple."
  • Posted: Tue Jul 20 15:30:52 UTC 2010

Click here for all of my pinboard.in bookmarks.