Webhook delivery with retries
Provider, queue, and endpoint with backoff retries.
Webhooks fail constantly in production — deploys, timeouts, overloaded receivers — and the difference between a robust integration and a data-loss incident is what happens next. This sequence diagram shows the standard answer: the provider enqueues an order.paid event, the delivery queue POSTs to your endpoint, and the alt block covers both worlds. A 2xx marks the event delivered; a timeout or 5xx triggers a backoff-scheduled retry that succeeds on attempt two.
The queue's self-message — "Schedule retry with backoff" — is the detail worth internalizing. Retry state lives in the provider's delivery infrastructure, not in your endpoint, which is why your only jobs are to respond fast and to handle duplicates.
When to use this template
- Integration runbooks — document for your on-call team what the provider does when your endpoint is down, and how long you have before events hit the dead-letter queue.
- Designing your own webhook system — if you are the provider, this is the minimum delivery semantics your customers will expect: queue, retry with backoff, and a delivered/failed status they can inspect.
- Debugging missing events — walk the diagram with real logs to determine whether the event was never sent, retried into a dead-letter state, or delivered twice and deduplicated incorrectly.
How to adapt it
Swap in your real event names and participants, then add your specifics:
- Add a signature verification step at the endpoint before the 200 OK, so the security requirement is part of the documented flow.
- Extend the failure branch with attempt 3..N and a dead-letter queue to match your provider's actual retry schedule.
- Show the async handoff inside your endpoint — respond 200, enqueue internally, process later — to document why responses stay fast.
Because visual edits regenerate clean Mermaid code, the adapted diagram is ready to paste into your README, runbook, or provider documentation.
Mermaid code
Copy it anywhere Mermaid is supported — GitHub, Notion, or your docs.
sequenceDiagram
participant P as Provider
participant Q as Delivery Queue
participant E as Your Endpoint
P->>Q: Enqueue event order.paid
Q->>E: POST /webhooks (attempt 1)
alt 2xx response
E-->>Q: 200 OK
Q->>P: Mark delivered
else Timeout or 5xx
E-->>Q: 503 Service Unavailable
Q->>Q: Schedule retry with backoff
Q->>E: POST /webhooks (attempt 2)
E-->>Q: 200 OK
Q->>P: Mark delivered after retry
end
Frequently asked questions
- How do webhook retries with exponential backoff work?
- When your endpoint times out or returns a 5xx, the provider's delivery queue schedules another attempt after a growing delay — commonly 1 minute, then 5, then 30, up to hours. This diagram shows that loop: a 503 on attempt one, a self-message scheduling the backoff, and a successful redelivery on attempt two.
- Why is there a delivery queue between the provider and my endpoint?
- Because reliable webhook systems never call your endpoint directly from the code that generated the event. Events are enqueued first, so a slow or failing receiver cannot block the provider, and failed deliveries persist for retry. Drawing the queue as its own participant makes this decoupling — and where retry state lives — explicit.
- What status code should my webhook endpoint return?
- Return 2xx quickly — acknowledge receipt, enqueue the work, and process it afterward. Anything else, including timeouts from slow synchronous processing, counts as failure and triggers a retry. That is also why retries make idempotency mandatory: use the event ID to deduplicate, because attempt two may arrive after attempt one half-succeeded.
- How do I adapt this template for my webhook provider integration?
- Rename the event to your real payload (invoice.created, push, message.received), set the participant names to the actual provider and your service, and extend the else branch with your provider's real retry schedule and dead-letter behavior. Visual edits regenerate clean Mermaid code, ready for your integration runbook.