Webhook delivery and retry pattern

Webhooks are how your application tells the outside world that something happened — a payment processed, a document uploaded, a user signed up. But networks fail. Endpoints go down for maintenance. Retries aren't optional; they're how you ensure that critical events reliably reach their destination.

This template shows the pattern every production webhook system uses: enqueue the event, attempt delivery with exponential backoff (wait 1s, then 2s, then 4s, then 8s between retries), and on final failure, move the event to a dead-letter queue where it can be inspected and replayed. Without this pattern, events silently disappear when endpoints are briefly unavailable, and downstream systems go out of sync.

The exponential backoff strategy is essential. If you retry immediately upon failure, you risk overwhelming an already-struggling endpoint and turning a 30-second outage into a cascading failure that lasts hours. By progressively increasing the wait time between attempts (1s, then 2s, then 4s, then 8s), you give the endpoint breathing room to recover while still ensuring eventual delivery. Most transient failures — database hiccups, temporary network issues, brief service restarts — resolve within the first two retries, so the exponential approach catches these without waiting too long.

The dead-letter queue is your safety net. After exhausting all retries (typically 5–10 attempts over 5–10 minutes), the event moves to a DLQ rather than disappearing or looping forever. From the DLQ, on-call engineers can inspect why delivery failed — was the endpoint permanently gone? Did the API contract change? — and decide whether to manually replay the event or discard it. Large-scale systems like Stripe, Twilio, and AWS SNS all expose DLQ-like constructs because permanent webhook delivery failure is often a sign of a deeper architecture problem that deserves investigation.

When to use this template

• Designing event delivery infrastructure — walk the team through what happens when a webhook endpoint is down: how many retries, how long to wait, when to give up, and how to surface failures.
• Debugging webhook failures — annotate the diagram with your retry counts and intervals (max_retries: 5, initial_delay: 1s, max_delay: 60s) so the team understands why a webhook succeeded on attempt 3 instead of on the first try.
• On-call runbooks — include the diagram in your post-incident review to explain how an endpoint outage caused a backlog of events in the DLQ, and how you drained it on recovery.
• Webhook provider documentation — if you're building a platform with webhooks (e.g., Zapier-like integration, notification service), share this diagram in your API docs so customers understand what reliability they can expect.

How to adapt it

Customize the retry schedule to match your reliability SLA and the criticality of your events:

• High-criticality events (payments, transfers) — increase max retries to 8 and max backoff to 300s so you exhaust all reasonable retry windows before DLQ. For payment webhooks, consider retrying every time the endpoint recovers, not just once.
• Low-criticality events (notifications, analytics) — reduce retries to 3 and backoff to 10s to fail fast and reduce queue depth. Analytics events that arrive a few minutes late are usually fine; notifications less so.
• Add event signing — insert a step where your app signs each webhook with an HMAC key, and the endpoint verifies the signature before processing, so replay attacks are prevented.
• Jitter to avoid thundering herd — if N events hit the DLQ and you replay them all at once, they can overwhelm the endpoint again. Add random jitter (±10%) to the retry delay so replayed events spread across a wider time window.

Because the visual editor writes clean Mermaid, you can sketch the lifecycle — enqueue, retry loop, success or DLQ — without wrestling with sequence syntax. Share the diagram with your customers or partners so they understand how to implement robust webhook consumers on their end.

Monitoring and observability are critical: a silent DLQ is a hidden disaster. Set up alerts when your DLQ depth exceeds a threshold (e.g., more than 100 events older than 1 hour). Log each attempt with the endpoint, timestamp, error reason, and attempt number; export these logs to your observability platform (Datadog, New Relic, CloudWatch) so on-call engineers can spot patterns (a specific endpoint failing all webhooks, a rate-limiting issue causing cascading retries, a protocol version mismatch) without manual digging. If your DLQ starts filling up, the on-call response should be automated: alert pages immediately, and a runbook tells the responder to check the endpoint health, the API contract, and whether the endpoint's rate limits were tightened.

Replay and idempotency matter too. When you decide to replay events from the DLQ, your recipient must be idempotent: if the same webhook is delivered twice (once before the outage, once on replay), the recipient should detect the duplicate and skip it, not process the event twice. Webhook IDs and idempotency keys are how you achieve this. If your webhook payload includes a stable event_id, the receiver can store {event_id -> processed_timestamp} and reject any repeat delivery of the same event ID. This is essential because in a real replay scenario, you might send the event successfully but the recipient's response got lost, so you retry and end up delivering a second time.