MermaidCreator
Sign inOpen editor
All templates
State template

Adaptive rate limit backoff

Exponential backoff with jitter when hitting API rate limits.

Customize with AIUse this template

APIs rate-limit to protect themselves from overload. When you hit a rate limit (HTTP 429), the server is saying "I am busy, do not send more requests right now." The naive approach — retrying immediately — just makes the problem worse. The correct approach is to back off: wait a bit, then retry. If you get another 429, wait longer. Exponential backoff gives the server time to recover without hammering it into submission.

This state machine shows the complete flow: you send a request, get a 429, enter an exponential backoff phase, wait, then retry. If the retry succeeds, you are done. If you hit the maximum retry count or backoff time, you give up and fail fast. The key detail is jitter — a random component added to the backoff window so all clients do not retry at the same instant.

When to use this template

  • API client library design — document the retry logic your SDK implements so callers understand when their request might be delayed or fail.
  • Integration testing — simulate a rate-limited upstream API and verify your client's backoff behavior. Ensure it does not thrash and that it eventually gives up.
  • Oncall runbook — when users report "My integration keeps timing out", use this diagram to explain the backoff strategy and help them tune retry budgets (e.g., "we retry for 2 minutes maximum per request").

How to adapt it

Add specifics to your implementation:

  • Set your backoff formula (e.g., 2^attempt or 1.5^attempt), jitter range (0–1 seconds), and max backoff (e.g., 32 seconds cap).
  • Add a circuit breaker state if the upstream API is consistently 429-ing — after 5 consecutive failures, flip to a degraded mode (use cached data or a fallback) instead of retrying.
  • Include metrics logging at each state so you can track which external APIs are rate-limiting you and optimize your request patterns.
  • Add a Retry-After header check — many APIs send Retry-After: 120 to tell you exactly how long to wait instead of guessing.

Mermaid code

Copy it anywhere Mermaid is supported — GitHub, Notion, or your docs.

stateDiagram-v2
    [*] --> Idle
    Idle --> Sending: Send request
    Sending --> Success: 2xx response
    Success --> [*]
    Sending --> RateLimited: 429 response
    RateLimited --> Backing_Off: Exponential backoff\n(1 + random) * 2^attempt
    Backing_Off --> Retry: Backoff window passed
    Retry --> Sending: Resend request
    Sending --> GiveUp: Max retries exceeded
    GiveUp --> [*]

Frequently asked questions

What is adaptive rate limit backoff?
When an API responds with 429 (Too Many Requests), the client does not immediately retry — instead, it waits an increasing amount of time before each retry. This is exponential backoff with jitter: wait 1 second, then 2 seconds, then 4 seconds, then 8 seconds, adding a random fraction so clients do not all retry at the same moment and cause a thundering herd.
Why not just retry immediately?
If everyone retries immediately, the API is still overloaded and you get another 429. The server needs time to recover. By backing off, you give the server space to drain its queue, and you reduce load on the network. Coordinating retries with jitter prevents the 'thundering herd' problem where all clients retry at the same instant and re-crash the API.
What is jitter and why does it matter?
Jitter is a random delay added to the backoff window — instead of waiting exactly 2^attempt seconds, you wait (1 + random) * 2^attempt. This spreads retries across time so they do not clump. Without jitter, if 1000 clients all get 429 and wait 2 seconds, they all retry at the same instant and re-crash the API. With jitter, they retry staggered across the 2-second window.
When should I give up and stop retrying?
Set a maximum retry count (e.g., 5 retries, or a max backoff of 32 seconds) or a total timeout (e.g., 2 minutes). If the API is still rate-limiting after max retries, it is probably having a bigger problem than temporary overload — alert your team and fail fast rather than timing out the user's request.

Related templates

Flowchart

Request timeout and retry pattern

Handle slow/failing requests with exponential backoff and max retries.

State

Rate limit state machine

Track quota consumption, backpressure, and recovery states.

State

Request retry strategy

State machine for exponential backoff, jitter, and circuit breaker integration.