What is Change Data Capture (CDC)?
A comprehensive guide to Change Data Capture — how it works, why it matters for database migrations, and how NoSQLSync uses CDC for zero-downtime MongoDB, DynamoDB, and Firestore replication.
What is Change Data Capture?
Change Data Capture (CDC) is a technique that tracks and captures every change made to your data — inserts, updates, and deletes — as they happen, and streams those changes to another system in real time. Instead of running periodic batch jobs that dump entire datasets, CDC continuously streams only the changed records.
How CDC Works
CDC typically works by reading the database's transaction log. In MongoDB, this is the oplog — a capped collection that records every operation applied to the primary node. In DynamoDB, this is DynamoDB Streams — a time-ordered sequence of item-level modifications. The CDC tool tails this log, extracts the changed records, and delivers them to a target system.
Log-Based CDC vs. Query-Based CDC
Log-based CDC reads directly from the database's internal transaction log (oplog for MongoDB, WAL for PostgreSQL, binlog for MySQL). It's efficient, has minimal impact on the source database, and captures every change with sub-second latency. Query-based CDC (also called polling) periodically queries the database for new or changed records using timestamps or version columns. It adds query load and has inherent latency.
CDC for Database Migration
When migrating a database, CDC plays a critical role: first, you bulk-migrate all existing data. Then, you enable CDC to capture every new write that happens during and after the bulk load. This means your application can continue writing to the source database while data flows continuously to the target. The two databases stay in sync, and you can cut over to the target when ready — with zero downtime.
MongoDB CDC with NoSQLSync
NoSQLSync connects to your MongoDB replica set or sharded cluster and tails the oplog directly. Every insert, update, and delete is captured with sub-second latency. The CDC pipeline handles MongoDB-specific structures — nested documents, arrays, ObjectIds — and maps them to the equivalent SQL types in your target database in real time.
Common CDC Use Cases
Beyond migration, CDC powers real-time analytics (streaming operational data to BigQuery or Snowflake for dashboards), cache invalidation (keeping Redis or Elasticsearch in sync with your primary database), audit logging (capturing every data change for compliance), and event-driven architectures (triggering downstream services on data changes).