A dead-letter queue is where messages go when your consumer cannot process them. Done well, it gives you a recoverable record of every failure. Done badly, it becomes a black hole where data disappears quietly. This guide covers the decisions that matter when designing a dead-letter queue for a Kafka pipeline.
Why you need a dead-letter queue ¶
Without a dead-letter queue, a consumer that encounters a message it cannot process has two options: retry indefinitely, which blocks the partition, or skip the message, which loses data. Neither is acceptable in most production systems. A dead-letter queue gives you a third option: park the message somewhere safe, record why it failed, and continue processing. You can then investigate and replay the failed messages when the underlying problem is fixed.
What to put in the error payload ¶
The error payload is what makes a dead-letter queue useful rather than just a dumping ground. At minimum, include the original message bytes, the exception class and message, the topic and partition the message came from, the offset, and a timestamp. We also include the consumer group ID and the application version. With this information, you can reconstruct exactly what happened and replay the message with confidence once the fix is in place.
Naming and topic structure ¶
We use a consistent naming convention: the original topic name followed by a.dlq suffix. If your source topic is payments.transactions, the dead-letter topic is payments.transactions.dlq. This makes it easy to find dead-letter topics in a large cluster and to set up monitoring rules that apply to all of them. Keep dead-letter topics on a separate retention policy from your main topics. You usually want to retain failed messages longer, not shorter.
Monitoring your dead-letter queues ¶
A dead-letter queue that nobody watches is not much better than no dead-letter queue. Set up a Prometheus alert that fires when any dead-letter topic receives a message. The alert should be informational by default, not a P1. You want to know that failures are happening, but a single failed message is not usually an incident. Set a separate alert for when the dead-letter topic lag grows continuously, which indicates a systemic problem rather than an isolated failure.
Replaying messages from the dead-letter queue ¶
Replay is where most dead-letter queue implementations fall short. The replay process needs to be documented and tested before you need it in production. We write a small replay utility for every pipeline we build: it reads from the dead-letter topic, optionally filters by error type or time range, and writes back to the original topic. The utility is part of the connector deliverable and is included in the runbook.
Dead-letter queues are one of the three most common gaps we find in pipeline audits. If you are not sure whether your current setup handles failures gracefully, the pipeline audit will surface it. The questions we hear often page covers related operational topics.