A/B Testing

A/B testing (split testing) randomly shows two or more variants to users and compares which produces better outcomes on a defined metric. The standard way to learn what actually works rather than what feels right.

Common failure modes: testing too small a change (a button-color test on traffic too thin to detect a 0.1% lift), calling a winner too early (significance flashes are not signals), and ignoring sample size (running for 7 days because that's the sprint).

Always compute sample size before launching. The two-proportion formula with 95% confidence and 80% power tells you exactly how many users per variant you need. Most "winners" that don't replicate in production were called below that threshold.