Change Data Capture is the modern default for keeping systems in sync — challenged
Hot take: CDC is not the default for keeping systems in sync. Log-tailing pipelines are at-least-once by construction. After a connector restart or a partition rebalance, consumers see duplicate events. That means you'r
Hot take: CDC is not the default for keeping systems in sync.
Log-tailing pipelines are at-least-once by construction. After a connector restart or a partition rebalance, consumers see duplicate events. That means you're building idempotent consumers regardless - whether you use CDC or not.
So you're paying for a Kafka Connect cluster, an initial snapshot phase that can hold locks on the source table at cutover, and schema-evolution coupling that ties every downstream consumer to your internal table shapes. All of that, for a guarantee you'd have to implement anyway.
For most teams: a transactional outbox table and a lightweight reconciliation job gets you the same consistency with a fraction of the moving parts. You already had to build idempotent consumers. Use them without the extra layer.
CDC earns its complexity when you have many independent consumers, need sub-second propagation, or already run Kafka. That's a scaling decision - not a starting assumption.
Is your CDC pipeline solving a problem you couldn't solve with an outbox?
https://www.morling.dev/blog/cdc-is-a-feature-not-a-product/
https://www.morling.dev/blog/you-dont-need-kafka-just-use-postgres-considered-harmful/
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.