A/B testing experiment lifecycle
States from hypothesis through rollout and analysis.
Every feature release should be an experiment: you hypothesize that a change will improve a metric, run it on a subset of users, measure the effect, and decide whether to roll it out. But many teams skip the rigor — they stop early when they see a win, forget to check significance, or lack a clear decision rule.
This template documents the disciplined path: hypothesis, design, active experiment, analysis with a clear threshold for winning, and conditional rollout. Each state transition has a purpose, and each end state — won, lost, inconclusive — triggers a specific action.
When to use this template
- Setting up experimentation culture — show your team the full lifecycle so everyone knows when a test is supposed to stop, how to measure, and what "winner" means before they code anything.
- Incident postmortems — when a launched feature caused problems, trace back to the experiment state: did it pass significance? Was the effect size large enough to matter? Would a longer test have caught the issue?
- Roadmap prioritization — document which hypotheses you're validating in each quarter, and use the state diagram to explain why some tests are still inconclusive.
How to adapt it
Extend the states and transitions to match your process:
- Add staging state — insert a state between design and active for quality assurance, ensuring the variant implementation matches the hypothesis.
- Include segments — after rollout starts, add a state for cohort-based rollout: first internal users, then beta segment, then general availability.
- Track post-launch monitoring — add a state after complete rollout to monitor for regressions, showing how long you keep the experiment metrics active.
Visual edits regenerate clean Mermaid code, so you can paste the diagram into your product requirements document or experimentation playbook.
Mermaid code
Copy it anywhere Mermaid is supported — GitHub, Notion, or your docs.
stateDiagram-v2
[*] --> Hypothesis
Hypothesis --> DesignExperiment
DesignExperiment --> Active: Start
Active --> Analyzing: Stop
Analyzing --> Won: Winning variant clear
Analyzing --> Lost: No significance
Analyzing --> Inconclusive: More data needed
Lost --> [*]
Inconclusive --> Active: Continue
Won --> Rollout
Rollout --> Complete: 100% rollout
Complete --> [*]
Frequently asked questions
- What does an A/B testing lifecycle look like?
- You start with a hypothesis — 'bigger buttons increase clicks' — design the experiment with a control and treatment variant, run it for a period collecting data, then analyze results. If the treatment won, you roll it out to 100% of users. If it lost or was inconclusive, you either declare it a loss or run it longer and restart analysis.
- Why do we need a designed experiment state diagram?
- Product teams often run experiments in ad-hoc ways: stopping too early because they see a win, forgetting to check statistical significance, or leaving failed tests running without a clear decision. A state diagram forces you to document when you measure, when you decide, and what 'winner' actually means — usually p < 0.05 and a practical effect size.
- What is statistical significance in A/B testing?
- Statistical significance (p-value < 0.05) means the difference between your control and variant is unlikely to have happened by chance. If your test isn't significant — even if the variant looks better — you don't know whether the improvement will hold in production. More data usually required.
- What do you do when a test is inconclusive?
- Inconclusive means the results are in the gray zone: not significant but not obviously negative. You have three choices: stop and declare it a loss (prioritize learning), extend the test to collect more data and re-analyze, or increase your variant sample size and re-randomize. Most teams choose to extend and re-analyze.