CDC vs Batch ETL
Compare Change Data Capture and batch ETL for database migration. Understand the tradeoffs in latency, complexity, and data consistency.
What is Batch ETL?
Batch ETL (Extract, Transform, Load) processes data in chunks on a schedule — hourly, daily, or weekly. The source database is queried for all records changed since the last run, those records are transformed, and loaded into the target. It's simple, well-understood, and works with any database that supports queries.
What is CDC?
Change Data Capture streams every data change in real time as it happens. Instead of polling for changes, CDC reads the database's transaction log (oplog for MongoDB,binlog for MySQL, WAL for PostgreSQL). This gives sub-second latency with minimal source impact.
Latency Comparison
CDC: milliseconds to seconds. Changes appear in the target almost instantly after commit on the source. Batch ETL: minutes to hours. Changes only appear in the target at the next scheduled run. For migrations where you need the target to be an exact mirror of the source, CDC is essential.
Complexity and Infrastructure
Batch ETL is simpler — it only needs query access to the source and write access to the target. CDC requires access to the database's transaction log, which may need special permissions, replica set configuration (MongoDB), or enabling streams (DynamoDB). However, NoSQLSync manages all of this complexity for you.
When to Use Batch ETL
Batch ETL is sufficient when: you're doing a one-time migration and can afford a brief maintenance window, the target doesn't need to be in real-time sync with the source, or you're migrating a read-only dataset where no new writes occur during migration.
When to Use CDC
CDC is essential when: you need the target to stay continuously in sync with the source, you cannot afford any downtime, your application continues writing to the source during migration, or you need a rollback option (the source is always up-to-date).
NoSQLSync's Hybrid Approach
NoSQLSync combines both: batch load for historical data (fast, parallel, efficient) + CDC for ongoing sync (real-time, continuous). This gives you the speed of batch for the bulk of your data and the real-time guarantees of CDC for ongoing changes.