Dead-Letter Queues for Webhooks: Safe Replay, Idempotency, and Monitoring

Dead Letter Queues For Webhooks Safe Replay Idempotency And Monitoring
Dead-Letter Queues for Webhooks: Safe Replay, Idempotency, and Monitoring
Webhooks fail. Endpoints time out, databases go down mid-deploy, a TLS cert expires at 2am — and if your system just retries forever or drops the event, you lose data silently. A Dead-Letter Queue (DLQ) is the safety net: instead of vanishing, a failed event gets quarantined for inspection and controlled replay. Here's how DLQs actually work in production, what the monitoring numbers should really be, and how to replay events without causing a second outage.

Idempotency comes before replay, not after
A message can crash after partially succeeding — think "charged the card, then died before marking the invoice paid." If you replay that event blindly, you double-charge the customer. So before any replay logic runs, your consumer needs to check an idempotency key, almost always the provider's event_id, against a record of what's already been processed.

This isn't a hypothetical edge case. Providers like Stripe explicitly document that an endpoint might occasionally receive the same event more than once, and AWS SQS standard queues warn that a message copy can reappear if a server is unavailable during deletion. In other words, at-least-once delivery is the norm across the industry, which makes deduplication the consumer's job, not something you can outsource to the network. The practical pattern most guides converge on: verify the provider's signature, extract the event ID, insert it into a unique-indexed idempotency table, then acknowledge quickly and do the heavy processing asynchronously.

How a DLQ actually works (AWS SQS as the reference case)
The generic description — "failed messages get moved to a quarantine queue" — is accurate but glosses over the mechanics. On AWS SQS, you attach a redrive policy to your main queue that names a target DLQ and a maxReceiveCount. When a message's receive count exceeds that threshold, SQS automatically moves it to the dead-letter queue while keeping the original message ID intact. If you don't set this explicitly, the default maxReceiveCount is 10 — meaning a message survives ten failed delivery attempts before SQS gives up on it.

Retention is where a lot of teams get bitten. A DLQ's retention period should be configured longer than the main queue's — a common pattern is 14 days on the DLQ versus around 4 days on the source queue — because the age clock for a standard-queue message keeps counting from its original enqueue time, not from when it landed in the DLQ. A message that already spent a day retrying in the main queue arrives in the DLQ with three days left on a four-day retention window, not four fresh days. Some references now push this further, treating 14 days as a minimum and recommending a 30-day floor for webhook DLQs specifically, given how long payment or billing reconciliation can take to notice a gap.

A minimal CLI example of setting this up:

Code example
Copy code
aws sqs set-queue-attributes \
--queue-url https://sqs.us-east-1.amazonaws.com/123456789012/webhook-processing \
--attributes '{
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:webhook-dlq\",\"maxReceiveCount\":\"5\"}"
}'
Kafka-based systems implement the same idea differently (a dedicated dead-letter topic consumed by an error-handling process), but the principle is identical: isolate the poison message, don't let it block the queue, keep enough context to debug it later.

What "Dead-Letter Pressure" should actually alert on
It's tempting to pick alert thresholds that sound dramatic — "500 messages a minute is a critical incident" — but that number is meaningless without knowing your normal traffic volume. What the current SRE guidance actually converges on is smaller and steadier signals:

Depth: alert when DLQ depth exceeds roughly 10 events — a small standing backlog is usually the first sign that a downstream dependency or a handler bug is quietly dropping events, well before you'd notice a dramatic spike.
Age: alert when the oldest event in the DLQ has sat unreviewed for more than about an hour, since age catches the slow leak that a depth-only alert misses — one event nobody triages is a silent data-loss risk regardless of overall volume.
Trend, not isolated events: a single failed delivery is expected in an at-least-once system, but a sustained rise in failure rate, retry rate, queue depth, or dead-letter volume is the real signal worth paging on.
Beyond DLQ-specific metrics, treat the whole pipeline as observable: track success rate, failure rate by error type, retry count distribution, processing latency at p50/p95/p99, time from receipt to completion, and event age, with SLOs set on each.

Safe replay: slower than it feels like it should be
Once the underlying issue is fixed — a deploy rolled back, a downstream database restored — the instinct is to flush the entire DLQ at once. Don't. Replaying tens of thousands of events simultaneously can overwhelm the destination and trigger the very outage you just recovered from. A safer pattern:

Confirm the destination is actually healthy first — send a single test payload and check for a 2xx response before touching the queue.
Replay in small batches — on the order of 100 to 500 events — watching the destination's error rate between each batch before increasing size.
Not every event type is safe to replay blindly — classify them first, and for anything with external side effects (charges, emails, shipments), confirm the idempotency check runs before the handler acts.
This is also where the idempotency work from earlier pays off directly: because every event carries a stable ID, replaying something that actually did succeed before it hit the DLQ is safe — the dedup layer treats it as a no-op instead of a duplicate charge.

Build it yourself, or use dedicated infrastructure?
Rolling your own DLQ pipeline on SQS or Kafka is very doable — the mechanics above aren't exotic — but a production-grade version also means building a dashboard for manual inspection, wiring alerting into Slack or PagerDuty, and securing a replay endpoint so it can't be abused. That's a meaningful chunk of infrastructure work for something that isn't your core product.

A number of vendors now specialize in exactly this so you don't have to build it: Hookdeck offers a queue-first delivery model with multi-tier retries and tenant isolation, priced (for its self-serve Outpost product) at around $10 per million events, or self-hosted under Apache 2.0. Svix focuses on the sending side for platforms that need enterprise-grade outbound webhook delivery. Smaller, newer entrants like InstaWebhook target the receiving side specifically — durable ingest endpoints, delivery timelines, retry and replay, and endpoint monitoring, with an optional bring-your-own-database mode for teams that want to keep payloads in their own Postgres instance. There's no universally "best" option here — it comes down to whether you're building an outbound webhook sender, an inbound receiver, or both, and how much control you need over where the data lives.

The bottom line
A DLQ isn't an optional nicety bolted onto a webhook system — it's the difference between "we lost a payment confirmation and didn't know for three weeks" and "we caught it in an hour and replayed it safely." The parts that actually matter are boring and mechanical: dedupe on a real idempotency key, set retention longer on the DLQ than the source queue, alert on depth and age rather than dramatic-sounding volume spikes, and replay in small, monitored batches instead of one big flush.

Sources: AWS SQS documentation on dead-letter queues and redrive policies; Hookdeck, GetHook, HookRay, and Digital Applied engineering guides on webhook reliability (2026); vendor documentation for Hookdeck, Svix, and InstaWebhook.