TL;DR
A/B testing is comparing two (or more) variants of a page or element to pick the one that performs better (e.g. higher CTR, conversions). Key: clear hypothesis, one thing changed, sufficient sample size, and statistical significance.
Who this is for
- Marketers and product owners
- People responsible for CRO (Conversion Rate Optimization)
- Shop and lead-page owners
Keyword (SEO)
a b testing, ab test conversions, website ab testing, conversion optimization testing
What is an A/B test?
- Variant A (control) – current version
- Variant B – changed version (e.g. different CTA, headline, layout)
- Traffic split – e.g. 50/50 of users
- Metric – e.g. CTA clicks, sign-ups, purchases
Rule: change one thing. If you change several at once, you won’t know what drove the result.
When does A/B testing make sense?
- ✅ You have traffic (at least hundreds of conversions per variant per month)
- ✅ You have a clear hypothesis (e.g. “green button will get more clicks”)
- ✅ You can wait 1–4 weeks for results
- ❌ Symbolic traffic – result won’t be statistically meaningful
- ❌ Changing everything at once – that’s a new page, not A/B
Step-by-step process
1. Hypothesis
State: “Changing X will increase Y because Z.”
Example: “Changing CTA from ‘Submit’ to ‘Get a quote’ will increase form completions because it clearly says what the user gets.”
2. Goal and metric
- Primary metric – e.g. form conversion
- Secondary metrics – e.g. time on page, bounce (to avoid hurting UX)
3. Variant design
- Only one element different (e.g. button text, color, position)
- Variant B must work correctly on all devices
4. Duration and sample size
- Use a sample-size calculator (e.g. A/B test calculator)
- Account for seasonality – don’t end the test on a long weekend
- Usually at least 1–2 weeks, often 2–4
5. Analysis
- Statistical significance (e.g. 95% confidence)
- Don’t stop the test early when it “looks clear” – that’s peeking
Tools
- Google Optimize (discontinued) – replaced by VWO, Optimizely, AB Tasty
- GA4 + Google Tag Manager – custom experiments (redirect or content change)
- CRO tools – often include significance calculator and reports
Pitfalls
- Peeking – checking results repeatedly and stopping early
- Too small sample – “win” for B may be random
- Multi-variant effect – testing many things (A/B/C/D) needs a larger sample
- Ignoring segments – e.g. mobile vs desktop may behave differently
FAQ
What is statistical significance?
The probability that the observed difference is not due to chance. 95% = we treat the result as real with 5% risk of error.
Can I test more than 2 variants?
Yes (A/B/n), but you need a proportionally larger sample and longer run. For simplicity, sequential A/B tests are often better.
What if the result is a “tie”?
No significant difference is still a result – stay with variant A (or the cheaper/simpler one). Don’t implement the change “by gut feel”.