Scheduled job monitoring state machine
Job lifecycle from scheduled through execution, failure recovery, and completion.
Every background job or cron task has a heartbeat: it runs on schedule, succeeds or fails, and if it fails, someone needs to care. But 'someone needs to care' is vague — do you retry immediately? How many times? When do you page on-call? This template lays out a seven-state lifecycle: scheduled, running, success (archived), failure, retry, alert, and a manual paused state where ops can hold the job while a dependency recovers.
The key insight is that a single failed job shouldn't page you — but three consecutive failures, or a failure of a critical job like payroll, should. The state machine forces teams to define the retry policy and escalation threshold upfront, so the monitoring loop doesn't become a surprise at 3am.
When to use this template
- Cron monitoring design — plan your retry strategy and alert threshold before building, so teams know what to expect when a job fails.
- On-call runbook — use the diagram to spec the steps: retry? check logs? page on-call? This becomes your incident response guide.
- SLA auditing — every state transition should have a time limit (e.g. "fail after 30s running"). Add timestamps to the diagram so teams see the full SLA.
How to adapt it
Customize the state names and transitions for your job types:
- Add a backoff delay — between Retry and Running, insert a time-delay state showing exponential backoff (1s, 2s, 4s, etc.) so flaky retries don't hammer the system.
- Split alert types — replace the Alert state with two branches: Alert → Slack (for non-critical jobs) and Alert → Page On-Call (for critical jobs); use separate thresholds.
- Add a manual review state — for financial or compliance jobs, add a Manual Review state between Failed and Retry, so a human approves the retry before it runs again.
Visual edits regenerate clean code, so you can sketch different retry and escalation paths in the editor without writing state-machine syntax directly.
Mermaid code
Copy it anywhere Mermaid is supported — GitHub, Notion, or your docs.
stateDiagram-v2
[*] --> Scheduled
Scheduled --> Running: Execution time reached
Running --> Success: Job completes
Running --> Failed: Error or timeout
Failed --> Retry: Retry available
Retry --> Running: Retry scheduled
Failed --> Alert: Max retries exceeded
Alert --> Paused: Manual intervention
Paused --> Running: Resume triggered
Success --> Archived: Completion recorded
Paused --> Archived: Abandoned
Frequently asked questions
- What is a scheduled job state machine?
- It models the lifecycle of a cron job or scheduled task: from the moment it's scheduled to run, through execution, success or failure, retry attempts, and alerting if it fails too many times. Teams use it to define when to retry, when to escalate to on-call, and what a 'healthy' monitoring loop looks like. Every microservice that runs scheduled work needs this diagram so on-call engineers know what to expect.
- How many retries is the right number?
- It depends on the impact. Transient failures (network blip, temporary API rate limit) warrant 2–3 immediate retries with exponential backoff (1s, 2s, 4s). Persistent failures (bad data, permission denied) should alert after 1 retry. For business-critical jobs (payroll, billing), alert immediately on failure. This diagram forces that decision; update the 'Retry available' diamond with your SLA.
- When should I page on-call for a failed job?
- Alert immediately if the job is critical (billing, fraud detection, financial reconciliation). For utility jobs (log rotation, cache warming), wait for 3 retries before alerting. For low-priority jobs (analytics import), log it and notify via Slack instead of paging. The state machine shows the decision points; fill in your own SLAs.
- What does 'Paused' mean?
- When a job fails repeatedly and the cause is known to be temporary (an external API is down for maintenance), ops can pause the job to stop alerts, then resume it once the dependency is back. Paused is also used to manually disable a job without deleting its schedule. Visual edits regenerate clean code, so you can sketch your escalation path before implementing it.
Related templates
Incident state machine
States and transitions during an incident from detection to postmortem.
SLA response state machine
Track support ticket states from submission through response and resolution SLAs.
Deployment rollback decision tree
Incident detection, severity assessment, and rollback trigger criteria.