Pod restart and recovery
Container health checks, restarts, and fallback strategies.
Containers fail: processes crash, hang, run out of memory, or encounter a dependency that is down. A production orchestrator (Kubernetes, Nomad, Docker Swarm) handles this by continuously checking the container's health and restarting it if needed. But it does not restart forever — after a handful of failed attempts, it gives up and alerts the on-call team so a human can debug.
This template maps the complete cycle: the orchestrator runs the health check, and if it passes, the pod is healthy. If it fails, the orchestrator tries to restart the container. If the restart succeeds, the health check runs again. If restarts keep failing, the pod enters a CrashLoopBackOff state and stops trying — that is the signal to the team that something is fundamentally wrong with the image or config.
When to use this template
- Kubernetes health check design — walk the team through what a successful health check means, and which failure mode (readiness vs liveness probe) triggers each branch.
- Troubleshooting CrashLoopBackOff — when a pod is stuck restarting, use the diagram to narrow the problem: is it a startup issue, or does the app crash after initialization?
- Alerting on restarts — agree on thresholds (e.g., more than 3 restarts in 5 minutes) and wire them into your monitoring so the team learns about flapping pods before they affect availability.
How to adapt it
Add your implementation details:
- Specify your health check type (HTTP GET to /health, TCP connection to port 8080, or a shell command).
- Define your restart policy (never, on-failure with max attempts, or always).
- Add a dependency check before the health check — does your app fail fast if Redis is down, or does it hang waiting for it?
- Include your escalation logic — at what restart count do you page the on-call, and do you auto-rollback to the last known-good image?
Mermaid code
Copy it anywhere Mermaid is supported — GitHub, Notion, or your docs.
flowchart TD
A[Orchestrator monitors pod] --> B{Health check passes?}
B -->|Yes| C[Pod runs normally]
C --> A
B -->|No| D{Max restart attempts?}
D -->|No| E[Kill unhealthy container]
E --> F[Start new container instance]
F --> G{Starts successfully?}
G -->|Yes| B
G -->|No| D
D -->|Yes| H[Pod stuck in CrashLoopBackOff]
H --> I[Alert on-call team]
I --> J[Inspect logs + debug]
J --> K[Fix root cause + roll new image]
Frequently asked questions
- What is a pod restart and recovery diagram?
- It shows how an orchestrator (Kubernetes, Nomad, Docker Compose) detects when a container fails its health check and automatically restarts it. Most restarts succeed — the container comes back up healthy. But repeated failures trigger a circuit breaker (CrashLoopBackOff in Kubernetes) so the orchestrator does not thrash the same failing pod forever; it alerts the team instead.
- When does a health check fail?
- When the orchestrator's probe (HTTP GET, TCP socket, or shell command) does not get a success response within the timeout. Common causes: the app crashed, the app process is hanging, or it is listening on the wrong port. The health check failing does NOT mean the pod is dead — it means the app is unhealthy and needs restart.
- What is CrashLoopBackOff?
- Kubernetes's circuit breaker for a pod that keeps crashing on startup. After a few restart attempts (default 5), Kubernetes backs off and waits increasingly longer (10s, 20s, 40s, 80s, 160s, 300s) before retrying. It stays in CrashLoopBackOff until you fix the issue and roll a new image or config.
- How do I know a pod is restarting too much?
- Look at the pod's restart count in 'kubectl get pods' — if it is increasing over time, the pod is crashing. Check the logs with 'kubectl logs <pod>' to see the error, or 'kubectl describe pod <pod>' to see the last termination reason. Set up alerting on restart_count > threshold to catch problems before they affect users.