The E-Commerce Experimentation Playbook — A/B Tests That Actually Decide

Every model, feature, banner, ranking change, and pricing rule in e-commerce is ultimately judged by one thing: an experiment. Yet most companies run tests that are statistically doomed before launch, then argue about the results for two weeks. This playbook covers the statistics you need, the e-commerce-specific traps the textbooks don't mention, and the organizational discipline that separates experimentation cultures from dashboard cultures.

1. Why Experimentation Is the Highest-Paid Skill Nobody Teaches

Here's an uncomfortable industry fact: at companies with mature experimentation cultures (Booking, Amazon, Airbnb), the majority of tested ideas fail to beat control. Microsoft's published number is roughly two-thirds of ideas showing no significant improvement or actively hurting metrics. This means:

• Intuition is worse than a coin flip at predicting what works — even expert intuition
• A company that ships without testing is shipping a majority of neutral-to-negative changes
• The person who can reliably separate real wins from noise controls what ships — that's leverage

The math is straightforward. If a typical successful test moves conversion by 1–3%, and your business does $500M GMV, being right about ten decisions a year is worth tens of millions. Experimentation literacy is how you get to be right.

2. Designing a Test That Can Actually Succeed

Every good experiment is fully specified before launch. The non-negotiable elements:

3. Power Analysis — The Step Everyone Skips

Power is the probability your test detects an effect that actually exists. Run an underpowered test and a real 2% win looks like noise — you kill a good idea and never know it. The industry standard is 80% power at 5% significance, and the sample size formula every analyst should be able to run:

# Sample size per variant for a conversion-rate test
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline_cvr = 0.032        # 3.2% conversion
mde           = 0.05         # want to detect a 5% relative lift
target_cvr    = baseline_cvr * (1 + mde)

effect = proportion_effectsize(target_cvr, baseline_cvr)
n = NormalIndPower().solve_power(effect, power=0.8, alpha=0.05)
print(f"~{n:,.0f} users per variant")   # → ~190,000 per variantpython

That number surprises people every time: detecting a 5% relative lift on a 3.2% baseline needs ~190k users per arm. This is why the most important output of power analysis is often the decision not to run the test — or to redesign it:

• Too little traffic? Test a bigger change (bigger MDE), test on a higher-baseline metric (add-to-cart instead of purchase), or use CUPED (next section)
• Minimum Detectable Effect (MDE) is a business question: "What's the smallest lift that pays for this feature's maintenance?" If a 1% lift justifies it but you can only detect 10%, your test can only confirm miracles

4. CUPED — Free Speed for Your Tests

CUPED (Controlled-experiment Using Pre-Experiment Data) is the highest-ROI statistical upgrade in experimentation, standard at Microsoft, Netflix, Booking, and Airbnb, and still weirdly unknown outside them. The idea: much of the variance in your metric comes from stable user differences — heavy buyers buy a lot in both control and treatment. If you know each user's pre-experiment spend, you can subtract the predictable part and test only the residual:

# CUPED-adjusted metric in ~5 lines
import numpy as np

# y = metric during experiment, x = same metric pre-experiment
theta = np.cov(y, x)[0, 1] / np.var(x)
y_cuped = y - theta * (x - x.mean())

# variance reduction = corr(x, y)² — e.g. corr 0.6 → 36% less variance
# → ~36% smaller sample needed for the same powerpython

For purchase metrics with returning customers, correlations of 0.5–0.7 are common — that's 25–50% shorter tests for free, forever, on every test. Two caveats that the tool-vendor blogs undersell:

• New users have no pre-period — CUPED does nothing for them (use θ·(0 − mean) or fall back to unadjusted). In acquisition-heavy funnels, the gain shrinks accordingly
• The pre-period must be unaffected by the treatment — always use data strictly before randomization

5. Peeking, Fishing, and the Other Cardinal Sins

Peeking

Checking your test daily and stopping when p < 0.05 sounds harmless. It isn't: with daily peeks over a four-week test, your real false-positive rate inflates from 5% to roughly 25–30%. A quarter of your "wins" are noise, and they'll mysteriously fail to replicate in the annual metric review. Fixes, in order of preference:

1. Don't peek. Fix the duration, look at the end. Monitoring dashboards for guardrail disasters is fine — deciding early because primary looks good is not.
2. Sequential testing (mSPRT, always-valid p-values — what Statsig/Eppo/Optimizely use) lets you peek legally at the cost of somewhat lower power.
3. Bayesian monitoring with a pre-agreed decision rule — fine, as long as the rule really is pre-agreed.

Metric fishing

Twenty metrics at 5% significance ≈ one false positive per test guaranteed. If the primary metric is flat but "conversion among Android users in Riyadh on weekends" is up 12%, that's not a finding — that's the reason the primary metric exists. Subgroup effects are hypotheses for the next test, never conclusions from this one.

Sample Ratio Mismatch (SRM) — the silent test killer

You configured 50/50 but got 50.8/49.2. A chi-square test says that's wildly improbable — something is corrupting randomization (bot filtering hitting one arm, redirects dropping users, caching). SRM invalidates the whole test, and mature platforms check it automatically:

from scipy.stats import chisquare

observed = [100_480, 98_320]          # users per arm
stat, p = chisquare(observed)
if p < 0.001:                        # conventional SRM threshold
    print("SRM detected — do NOT read the results. Debug the assignment.")python

6. E-Commerce-Specific Traps

These are the ones generic A/B testing guides miss — and they're exactly where e-commerce tests go wrong:

7. Reading Results Like an Operator

The test ended. Here's the discipline for the readout:

1. Check SRM first. If assignment is broken, stop.
2. Check guardrails second. A conversion win with a latency regression or margin hit is a business decision, not an automatic ship.
3. Read the confidence interval, not the p-value. "+2.1% [+0.3%, +3.9%]" tells you the plausible range of impact; "p=0.02" tells you almost nothing actionable. Ship decisions should weigh the whole interval against costs.
4. Flat is information. A well-powered flat result kills a roadmap item cheaply — that's a win. Log it in a searchable registry so the idea doesn't resurrect in six months with a new sponsor.
5. Annualize honestly. The classic sin: "2% lift × 52 weeks = 104% growth." Effects regress, novelty decays, tests interact. Mature orgs discount projected annual impact by 50–70% — and validate with periodic holdback groups (5% of users kept on the old experience for a quarter).

8. The Organizational Layer

The statistics are the easy half. These are the practices that make experimentation compound:

• A written test plan per experiment — hypothesis, primary metric, MDE, duration, decision rule — filed before launch. One page. This single artifact kills 80% of post-hoc arguments.
• A results registry — searchable history of every test, including flat and negative ones. Institutional memory is the real moat; without it you re-test the same ideas every 18 months.
• Kill criteria with teeth — agree in advance what result kills the feature. If no result can kill it, don't bother testing; ship it and save the traffic.
• Separate "learning tests" from "shipping tests" — a painted-door test (fake button measuring clicks) answers demand questions in days without building anything.
• Track your win rate. If 80% of your tests "win," your process is broken — the industry base rate says most ideas don't work. A suspiciously high win rate means peeking, fishing, or testing only trivially safe changes.

9. The One-Page Checklist

Before launch:

• ☐ Hypothesis with a mechanism ("because...")
• ☐ One primary metric + guardrails, written down
• ☐ Power analysis: sample size & duration for a business-justified MDE
• ☐ Full-week duration, fixed in advance
• ☐ CUPED pre-period data wired up (if returning users matter)
• ☐ Interference check: does treatment consume shared inventory/price/capacity?
• ☐ Decision rule: what result ships, what result kills

During:

• ☐ SRM monitored automatically
• ☐ Guardrail disasters monitored; primary metric NOT used for early stopping

After:

• ☐ SRM → guardrails → CI-based readout, in that order
• ☐ Novelty check (week 1 vs later)
• ☐ +30-day re-read for returns/cancellations
• ☐ Registry entry filed — win, flat, or loss

Sources & Further Reading:
GrowthBook: A/B Testing Guide • Statsig: CUPED Test Duration • AB Tasty: Statistical Models Compared • Kohavi's ExP Platform papers