Marcus Holt spent seven years as a data platform engineer at a logistics company in Austin, Texas, where the pipeline went down every time a major carrier updated their API. The fix was always the same: a manual script, a late night, and a promise that someone would build something better. In the spring of 2019, he left to build that something. Vault Labs StreamNex started as a one-person consultancy operating out of a rented desk at a co-working space on East 6th Street, taking on Kafka audits for two fintech startups that had outgrown their batch jobs.
Consumer lag is one of the most-watched metrics in any Kafka deployment, and one of the most misread. A lag number on its own tells you almost nothing. What matters is whether that lag is growing, stable, or shrinking, and at what rate. This guide covers how to read consumer lag accurately, what causes it to grow, and how to set thresholds that page you when something is actually wrong.
Read more →A dead-letter queue is where messages go when your consumer cannot process them. Done well, it gives you a recoverable record of every failure. Done badly, it becomes a black hole where data disappears quietly. This guide covers the decisions that matter when designing a dead-letter queue for a Kafka pipeline.
Read more →Moving from batch ETL to a streaming architecture is not a refactor. It is a rethink of how your system handles time, ordering, and failure. We have run six of these migrations since 2019. The technical work is rarely the hard part. The hard part is the assumptions your existing system makes that nobody has written down.
Read more →We map your data sources, latency requirements, and failure modes before writing a line of config. Most projects start with a two-hour whiteboard session, not a proposal deck.
We run Kafka on bare metal and cloud alike. Partition tuning, consumer lag alerts, broker failover. We have been doing this since before managed Kafka was a product.
Off-the-shelf connectors break at the edges. We write custom Kafka Connect plugins in Java and Go for sources that Confluent never thought to support.
Marcus Holt is the founder of Vault Labs StreamNex. He spent seven years as a data platform engineer at a logistics company in Austin, Texas, before starting the consultancy in 2019. His background is in distributed systems and event-driven architecture, with a particular focus on Kafka internals and consumer group behaviour under load. Before founding StreamNex, he contributed to an open-source Kafka monitoring tool that is still maintained by the Apache community. Outside of work, he runs long distances on the trails around Barton Creek and reads more infrastructure post-mortems than is probably healthy. His two most-cited pieces of writing are a 2021 deep-dive on partition rebalancing and a 2023 guide to dead-letter queue design that circulated widely in the Kafka community.
Yes, but we are honest about what that means. If your team has never run Kafka in production, the first engagement usually includes more explanation and documentation than a team that already has a cluster running. We price that in. The training workshop is often a good starting point before a larger project.
We review your broker configuration, topic settings, consumer group assignments, and any connector configs you have. We replay your recent failure logs if you can share them. The output is a written report, usually 15-25 pages, with findings ranked by severity and specific remediation steps. We do not sell you on a follow-up engagement at the end.
Yes. We have worked with Confluent Cloud, Amazon MSK, and Aiven. Managed services remove some operational burden but introduce their own constraints, particularly around networking, IAM, and connector support. We know where those edges are.
On-call is included in the managed retainer. We use PagerDuty with escalation paths that your team can see and adjust. Response time target is 15 minutes for P1 alerts. We define P1 together during onboarding, because not every alert is a P1.
Yes, standard practice. We handle client configs and data schemas that are sensitive. We use a mutual NDA template that most legal teams approve without changes. If you need specific clauses, send us your version and we will review it.
Fill in the form and one of our engineers will reply within one business day. We will ask a few follow-up questions before suggesting a next step.