Why we don’t run A/B tests

A/B testing is supposed to be the rational way to make product decisions: ship two variants, measure which one performs better, keep the winner. It looks like science. It often isn’t.

The problems we kept running into

Most A/B tests at our scale have too little statistical power to be meaningful. Tests that are powered enough run for so long that something else in the app or the world changes underneath them, contaminating the result. And the metric you’re measuring is almost never the metric you actually care about — you optimize for clicks because clicks are countable, and the deeper thing you wanted (trust, retention, taste) drifts in some direction you never measured.

What we do instead

We design two options, pick the one that’s more honest about what the app is, ship it confidently, and listen carefully to what the next six weeks of reviews and support tickets tell us. We’re willing to be wrong sometimes in exchange for not having to be statistically certain all the time.

It’s slower in one sense and faster in every other.