📈
Real-world scenario
Tunde (Lagos growth analyst): Did variant B actually win?

Tunde is a growth analyst at Flutterwave's checkout team in Lagos. Two weeks ago, the product team shipped a new checkout flow (variant B) on 50% of traffic. The other 50% sees the old design (variant A). Now Tunde has data: 2,500 users in each variant, and variant B shows 14% conversion vs. A's 10%. That's a 40% lift! But Tunde's boss asks: "Is this real improvement, or just luck?"

Tunde can't just eyeball the difference and celebrate. With real checkout data, randomness matters. Maybe if he ran it again, variant B would be worse. His job: prove variant B's win is unlikely to be by chance, quantify the lift with confidence, and decide: keep it or keep testing?

5,000
Total Sessions
14.0%
Variant B Conv.
43%
Observed Lift
What you'll learn
  • Understand what A/B testing actually proves (and doesn't)
  • Compute conversion rates and lift from raw event data
  • Use chi-squared tests to compute p-values and test statistical significance
  • Interpret confidence intervals: when is a difference real?
  • Recognize sample size effects: why 500 rows is not enough
  • Spot common pitfalls: peeking, Simpson's paradox, multiple testing
  • Decide when to stop testing and scale the winner
📥
Flutterwave checkout A/B test dataset
5,000 rows of checkout events with variants, devices, conversions, and order values

What is A/B Testing?

A/B testing (or split testing) divides users into two groups: one sees variant A (control), the other sees variant B (treatment). You measure an outcome (conversion, click-through, session length) and ask: Is the difference in outcome statistically significant?

Why not just compare? Because random noise always exists. With 100 users, you might see 11% convert with variant A and 14% with B—but that could be luck, not a real improvement. With 10,000 users per variant, a 1% difference is usually real. A/B testing is the scientific method for product decisions.

Key insight: A/B tests don't prove variant B is "better forever." They prove: given your sample, the observed difference is unlikely to have happened by chance (p < 0.05). If you run the test again with new users, the direction might flip. But at sufficient scale, a real winner emerges.

Computing Conversion Rates and Lift

Start with event-level data: each row is a user session. Count conversions by variant, compute rates, then calculate lift (the % improvement).

Load data and compute conversion rates
Conversion rate by variant with 95% confidence intervals.
Bar chart showing Variant A at 9.8% conversion (95% CI: 8.8–10.8%) and Variant B at 14.0% (95% CI: 13.0–15.1%). B is higher.
What this shows: Variant A converts 9.8% of sessions; Variant B converts 14.0%. The confidence intervals (error bars) don't overlap, which is a good sign. But overlapping CIs don't mean no difference—we need a formal significance test to be sure.

Statistical Significance & P-Values

A difference in conversion rates doesn't automatically prove causation. Random variation creates noise. To test if variant B is truly better, we ask: If A and B were identical, how likely would we see a difference this large (or larger) by chance alone? That probability is the p-value.

The standard threshold is p < 0.05 (5% chance of observing the data if there's no real difference). When p < 0.05, we say the result is statistically significant — the difference likely reflects a real improvement, not noise.

Chi-squared test for independence
P-value shrinking as sample size grows (per arm: 500, 1k, 2k, 5k sessions).
Line chart showing p-value declining from 0.43 at n=500, to 0.18 at n=1k, to 0.0003 at n=2k, and nearly 0 at n=5k. Demonstrates that larger samples yield stronger evidence.
What this shows: With 500 users per arm, the p-value is 0.43 (43% chance of seeing this difference if A=B). Not significant. At 2,000 users per arm, p drops to 0.0003 — now we're confident variant B is better. This is why sample size matters: larger samples give stronger statistical power.

Why Sample Size Matters: The Power of Large N

Statistical significance depends heavily on sample size. A 1% difference in conversion with 100 users per arm is almost never significant. The same 1% difference with 100,000 users per arm is highly significant. Why?

Larger samples reduce random variation. The standard error of a proportion is inversely proportional to √n. Doubling your sample size cuts the standard error by ~29%, making it easier to distinguish signal from noise. This is why large-scale A/B tests in tech companies run for weeks or months: they need sufficient power to detect real but modest improvements.

Rule of thumb: For a 1-2% difference in conversion rates, you need ~1,000–10,000 users per variant. For a 10% difference, 500 users per arm might suffice. Always compute sample size requirements before launching a test.

Segmented Analysis: Variant Performance by Device

A/B test results often hide important variations. Variant B might win overall, but lose on mobile. Segment your analysis by key dimensions: device, geography, user segment. This reveals interactions where a variant excels in one subgroup but fails in another.

Conversion rate by variant and device type.
Grouped bar chart showing Mobile: A=9.6%, B=13.6%; Desktop: A=10.2%, B=14.8%. Variant B leads in both segments.
What this shows: Variant B wins on both mobile and desktop. This is consistent. If we'd seen B win on desktop but lose on mobile, we'd need a deeper investigation: maybe B's mobile UX is broken, or a mobile-specific cohort has different preferences.

Beyond Conversion: Order Value Matters

Conversion rate is one metric, but revenue is what scales companies. Two variants can have the same conversion rate but different order values. Always measure order value (or revenue per session) alongside conversion.

Distribution of order values (converted sessions only) for both variants.
Histogram of order values in Naira. Mean ₦18,033, Median ₦13,134. Right-skewed distribution with long tail to ₦156k. Both variants show similar patterns.
What this shows: Order values are lognormal: most orders cluster below ₦20k, but a few large orders (₦100k+) exist. The mean (₦18,033) exceeds the median (₦13,134), showing the right-skew. When comparing variants on order value, use median or a robust test, not just means—outliers can distort the picture.

Common A/B Testing Pitfalls

1. Peeking: Stopping Early When You See "Winning"

The problem: You run a test, and after 3 days you see variant B ahead with p=0.04. You declare victory. But you violated the statistical contract—you were supposed to run the test for 2 weeks at fixed sample size. Stopping early inflates false positives. This is called "peeking."

The fix: Decide sample size in advance. Don't glance at results daily. Wait until you reach the target N, then analyze once.

2. Simpson's Paradox: Aggregation Hides Reversals

The problem: Variant B wins overall, but loses in every segment (mobile, desktop, each country). How is this possible? It's Simpson's Paradox: the segment sizes differ. Variant B got more traffic from a high-converting segment, making it look good in aggregate, but it actually underperformed.

The fix: Always segment your analysis. If aggregate results contradict segment results, investigate segment sizes and composition.

3. Multiple Testing / Repeated Testing

The problem: You run 20 A/B tests. Even if all variants are identical, you expect ~1 to be "significant" at p<0.05 by chance (multiple testing problem). Report all tests, adjust p-value thresholds (e.g., use p<0.01 for multiple tests), or use Bonferroni correction.

The fix: Decide on your metrics and variants in advance. Don't fish for significance by testing dozens of combinations post-hoc.

4. Confounding Variables

The problem: You test a new checkout flow. Traffic is randomly split... but variant B is deployed only on weekdays, and variant A runs 24/7. Weekday traffic is higher-quality (office workers), so B looks better. But the improvement isn't the checkout flow; it's the timing.

The fix: Ensure randomization is clean. Use A/B testing platforms that enforce date-agnostic rollout. Control for time-of-day, day-of-week, and other confounds in analysis.

When to Stop: Declaring a Winner and Scaling

Three scenarios:

  • Significant and practical: p < 0.05 AND lift > 2% (or your business threshold). Scale the winner. You've found a real, meaningful improvement.
  • Significant but too small: p < 0.05 BUT lift < 1%. The difference is real, but not worth the engineering effort to deploy. Archive the test and move on.
  • Not significant: p ≥ 0.05. You have two choices: run longer (more power) or declare no difference and either keep variant A or roll a coin. Don't pick B just because it's numerically higher.

Once you declare a winner, set a monitoring dashboard. A/B test improvements sometimes regress over time due to habituation, seasonality, or bugs. Track the winning variant in production for 2–4 weeks post-launch.

📋
Test your knowledge
What does a p-value of 0.03 mean in an A/B test?
There is a 3% probability that variant B is the true winner.
If A and B had the same conversion rate, we'd see a difference this large or larger 3% of the time by random chance.
Statistically significant evidence of a difference exists at the 5% level.
97% of users prefer variant B.
You see variant B with 12% conversion vs. A with 11% after 100 users per arm. Should you scale B?
Yes—B is higher, so it's better.
No—with only 100 users per arm, this difference is very likely due to random chance. Run the test longer for statistical power.
Only if p < 0.5.
Yes—a 1% lift compounds over time.
Variant B wins overall (15% vs. 12%), but loses on mobile (10% vs. 13%) and wins on desktop (16% vs. 11%). What should you investigate?
Variant B is best—ignore the segments, they're too small to matter.
Simpson's Paradox: the traffic mix differs between mobile and desktop. Check if B is getting more desktop traffic, which inflates the overall win.
The test is invalid because results contradict.
Always scale the winner; segment analysis is overcomplicated.
You run 20 independent A/B tests and find 2 with p < 0.05. What could be going on?
2 variants are genuinely better; scale both.
Multiple testing problem: even if all variants are identical, ~1 test (5% of 20) will be "significant" by chance. Report all tests and apply multiple-testing correction.
10% of your tests have false positives; ignore the rest.
Always run only 1 A/B test at a time to avoid this.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Master A/B testing and drive real decisions.

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →