Error budget state machine
Track SLO compliance and error budget depletion through states.
Site Reliability Engineers (SREs) live by error budgets: you get a fixed amount of downtime per quarter (for a 99.9% SLO, that's roughly 43 minutes) and you can spend it however you want. But once it's gone, you're in breach. This state machine models the error-budget lifecycle: starting Within Budget, degrading to Warning at 50% remaining, to Critical at 25%, to Exhausted when budget hits zero, and finally to Action Required when the SLO is breached and you must respond.
The states tell your team what to do: Within Budget means proceed normally. Warning means be cautious — defer non-critical deployments. Critical means freeze changes and focus on stabilization. Exhausted means declare an incident. Using states forces consistency: everyone knows what "Critical" means and what actions it triggers.
When to use this template
- SLO dashboards — render this state machine on your incident-response dashboard so the team always knows your budget status at a glance.
- Deployment policies — define rules that key off states: "no deployments during Critical or Exhausted" or "deployments during Warning require on-call approval". Make the policy visible so engineers see when they're blocked and why.
- On-call playbooks — document what to do at each state. Warning: check error rates and review recent changes. Critical: roll back risky changes or scale up capacity. Exhausted: declare incident, bring in incident commander, start post-mortem.
How to adapt it
Customize the thresholds and states for your organization:
- Add intermediate states — if your SLO budget decays fast, add more granular states: 80%, 60%, 40%, 20%. Each one is a checkpoint where you assess the situation.
- Add recovery transitions — if errors clear at any state, transition directly back to Within Budget rather than funneling through Critical. This lets you measure MTTR (mean time to recovery).
- Include escalation — from Action Required, add a transition to "Executive Escalation" after N hours without recovery, signaling that you need extra resources or external support.
Visual edits regenerate clean code, so you can experiment with state structures without Mermaid syntax complexity. Share this with your team so everyone understands your SLO governance.
Mermaid code
Copy it anywhere Mermaid is supported — GitHub, Notion, or your docs.
stateDiagram-v2
[*] --> Within Budget
Within Budget --> Warning: 50% budget remaining
Within Budget --> Critical: 25% budget remaining
Warning --> Within Budget: Error rate improves
Warning --> Critical: Budget depletes
Critical --> Within Budget: Errors cleared
Critical --> Exhausted: Zero budget left
Exhausted --> Action Required: SLO breach
Action Required --> Critical: Emergency fixes start
Critical --> Within Budget: SLO recovers
Frequently asked questions
- What is an error budget and why track it as states?
- An error budget is the amount of downtime or errors your SLO allows in a period (e.g., 99.9% uptime = 43 minutes of downtime per month). When you exceed that budget, you've broken your SLO. Tracking it as states — Within Budget, Warning, Critical, Exhausted — makes it clear what actions to take at each threshold. At Warning you start being cautious; at Critical you stop risky deployments; at Exhausted you declare an incident.
- Why do I need a Warning state if I'm not yet at Critical?
- Warning is an early signal. If you're at 50% budget remaining and error rates keep climbing, you'll hit Critical in hours. Seeing Warning lets ops teams increase alertness, defer risky changes, and start investigating before the SLO breaks. It's the difference between 'we have a problem' and 'we're about to have a problem'.
- How do I calculate the budget in real time?
- Record error counts (failed requests, page errors, timeouts) and uptime in a metrics system (Prometheus, Datadog, CloudWatch). Error budget remaining = (allowed errors per period) - (errors so far this period). Transitions happen automatically: when remaining budget reaches 50% or 25%, fire alerts. When budget reaches zero, declare SLO exhausted. Update the state machine in your dashboard so the team sees it.
- What happens when we transition from Exhausted back to Critical?
- It means errors cleared enough that you're no longer in active SLO breach, but the budget for the period is still empty or very low. You're recovering but not yet safe. Use this state to keep the team focused on stabilization rather than relaxing back to normal operations. Only return to Within Budget when the period rolls over or when enough time has passed that you've regenerated budget cushion.