The Confluent Hub lists over 200 connectors. For common sources like PostgreSQL, S3, and Salesforce, the off-the-shelf options are mature and well-maintained. For everything else, the picture is more complicated. This guide covers how to evaluate an existing connector before committing to it, and what to expect if you decide to build your own.
What off-the-shelf connectors get wrong at the edges ¶
Most connectors are built for the common case. They handle standard authentication, standard field types, and standard API versions. When your source uses a non-standard auth flow, returns nested JSON that the connector does not flatten correctly, or changes its API version without notice, the connector breaks. The question is not whether the connector works in the demo. The question is whether it handles your specific source's edge cases in production.
How to evaluate a connector before using it ¶
Read the open issues on the connector's GitHub repository before you deploy it. Look specifically for issues related to your source system, your authentication method, and your message volume. Check when the last commit was made. A connector with no commits in 18 months is not necessarily broken, but it is not being actively maintained either. Run it against a sample of your production data in a staging environment, not just the example data in the documentation.
The case for building your own ¶
A custom connector makes sense when the off-the-shelf option has known issues with your source, when the source is internal and no connector exists, or when you need behaviour that the existing connector does not support, such as a specific error handling strategy or a custom transformation. The build cost is real: a well-tested connector takes two to four weeks to build properly. But a connector that breaks in production every time the upstream API changes costs more than that over its lifetime.
What a custom build actually involves ¶
A Kafka Connect connector is a Java or Go library that implements the Connector and Task interfaces defined by the Connect API. The connector class handles configuration and task assignment. The task class handles the actual data movement. The complexity is usually not in the core logic but in the error handling: what happens when the source is unavailable, when a record fails validation, when the API returns an unexpected response. That is where most of the build time goes.
Handing over a custom connector ¶
A connector that only the person who built it can maintain is a liability. Every connector we deliver includes a runbook that covers how to deploy it, how to configure it, how to monitor it, and what to do when common errors occur. The source code is the client's. We do not retain a licence. The goal is that your team can extend or modify the connector without coming back to us, though we are available if they want to.
We have built fourteen custom connectors since 2019, for sources ranging from legacy ERP systems to real-time pricing APIs. If you are evaluating whether to build or buy, we are happy to look at the specific connector you are considering. What we deliver is described in more detail on the deliverables page.