Stop peeking at your A/B test
Stopping an A/B test the moment it looks significant fills your changelog with fake wins.
Day three of your test. The new pricing page is up 30% and your tool says it’s significant. You ship it and post the screenshot. A month later, signups are exactly where they were. Congrats, you found a ghost.
Evan Miller explained why in his 2010 essay How Not To Run an A/B Test. Significance math assumes you fixed the sample size in advance. If you instead keep checking and stop as soon as the result looks significant, the reported significance levels become meaningless. In his worst-case example, with a significance check after every single visitor on a change that does nothing at all, the test wrongly declares a winner 26.1% of the time. That’s more than five times the 5% error rate you thought you signed up for.
The fix is boring discipline. Decide the sample size before you start (Miller has a free sample size calculator), change one thing so you know what caused any difference, and don’t stop early because the dashboard turned green. Looking is fine; stopping because of what you saw is the problem. If your traffic can’t reach that sample size in a few weeks, skip the test and make a judgment call.
The example
- Instead of
- “Day 3: variant B is up 30% and the tool says significant. Ship it!”
- Try
- “Planned 4,000 visitors per variant before launch. Day 3 looks great. We decide at 4,000.”
Before you flush
Before your next test, write down the one change, the metric and the sample size you’ll wait for. Don’t call it until you hit that number.