A/B test analysis funnel
From experiment completion through results analysis and ship decision.
Thousands of A/B tests run and get abandoned because nobody ever finishes the analysis. This template enforces the full decision loop: from experiment completion through statistical rigor, business-impact thresholds, and rollout strategy. It's the difference between running tests and learning from them.
The two critical gates — statistical significance and business impact — prevent two common mistakes: shipping a meaningless difference because it "feels good" or killing a real win because the effect was small. Both are questions the funnel forces you to answer explicitly.
When to use this template
- Experimentation governance — onboard new teams to your company's test-and-ship standard, so nobody ships without hitting both gates.
- Test postmortems — when a shipped feature underperformed, trace back to this diagram: which gate did you skip? Often the problem was business impact was never quantified.
- Roadmap planning — show which experiments won and why, so leadership sees data-driven decisions instead of hunches.
How to adapt it
Customize thresholds and add domain-specific gates:
- Add a sensitive cohorts analysis between significance and ship: check whether the win holds for VIP customers, enterprise customers, or international regions, or if it hurts a key segment.
- Insert a cost-benefit analysis node: some tests win the metric but increase latency, infrastructure cost, or support burden. Quantify the tradeoff.
- Add a winner rollout template: after "Plan rollout", detail your gradual-ramp strategy (canary 10% → 50% → 100% with time gates between).
Drag nodes to customize the gates for your business — the editor regenerates the Mermaid code automatically.
Mermaid code
Copy it anywhere Mermaid is supported — GitHub, Notion, or your docs.
flowchart TD
A[Experiment ends] --> B[Collect results]
B --> C{Sufficient sample size?}
C -->|No| D[Extend experiment]
D --> A
C -->|Yes| E[Calculate metrics]
E --> F{Statistically significant?}
F -->|No| G[Mark as inconclusive]
G --> H[Document learnings]
F -->|Yes| I{Result aligns with goal?}
I -->|No| J[Analyze failure mode]
J --> H
I -->|Yes| K{Business impact sufficient?}
K -->|No| L[Deprioritize]
L --> H
K -->|Yes| M[Plan rollout]
M --> N[Ship to 100%]
Frequently asked questions
- What makes an A/B test result statistically significant?
- Your sample size and observed difference combine to create a p-value. If p < 0.05, the result is statistically significant (95% confident the difference is real, not random chance). Online calculators exist, but most analytics platforms (Amplitude, Mixpanel, Statsig) compute this automatically. Never ship a test that 'looks good' without hitting significance — randomness tricks you.
- Why do you need both statistical significance AND business impact?
- A test can be statistically significant but move the metric by 0.1%, creating engineering debt for no real gain. Conversely, a 3% lift that's not quite significant at p=0.08 might still be worth shipping if the volume is huge. The funnel forces both questions: Is it real? Is it worth doing?
- What should I do if a test is inconclusive?
- Extend the experiment if possible — more data clarifies the picture. If not, document what you learned (the direction was neutral, the implementation was complex, users didn't notice the change) and move on. Use those learnings to design the next experiment. Visual edits regenerate clean Mermaid, so you can update the diagram as the test runs.
- How do I handle rollout risk when shipping a winning test?
- Start with a 10% canary rollout, monitor for 24 hours (did metrics hold? did errors spike?), then 50%, then 100%. If something breaks, roll back and investigate. The 'Plan rollout' node is where you insert this gradual-ramp strategy before the final ship step.