Live Sync (CDC)

Real-time change data capture from MongoDB oplog to your SQL target.

How CDC works in NoSQLSync

NoSQLSync uses Change Data Capture to stream real-time changes from your MongoDB oplog to your SQL target:

  • The CDC worker opens a change stream on your MongoDB collection, tailing the oplog
  • Each insert, update, or delete event is captured and transformed according to your schema mapping
  • Events are written to the SQL target via batched statements for performance
  • Resume tokens are saved to the database — if the worker restarts, it picks up exactly where it left off

Setting up MongoDB oplog access

CDC requires your MongoDB deployment to have the oplog accessible. For MongoDB Atlas, the oplog is available by default. For self-hosted MongoDB, ensure your cluster runs as a replica set — standalone instances do not expose the oplog.

Verify oplog access:

db.getSiblingDB("local").oplog.rs.find().limit(1)

Monitoring replication lag

The Live Sync screen shows each active CDC channel with real-time status:

  • Green dot — channel is running and consuming events
  • Amber dot — channel is paused
  • Gray dot — channel is stopped

On the migration detail page, the SSE log stream displays per-event timestamps. Typical latency is under 50ms from oplog entry to SQL write.

Handling schema changes mid-sync

Schema detection samples your collection when you create the migration. If the source later gains a field that sample never saw, the worker applies the migration's drift policy, chosen when you create a CDC migration and changeable afterwards via PATCH /api/migrations/:id:

  • Add the column (default) — the worker runs ALTER TABLE ... ADD COLUMN for a nullable column of the inferred type, extends the mapping, and keeps replicating. Rows written before the column existed are NULL for it.
  • Skip the field — the target schema is left alone. The field is recorded and a warning is logged, but its data is not migrated.
  • Pause the sync — replication pauses so you can review the mapping. Resume once you have decided.

Every drifted field is recorded either way, with its inferred type, a sample value, how many times it has been seen, and what was done about it. Read them from GET /api/migrations/:id/drift; they also appear as warnings in the sync log.

Automatic column addition is currently implemented for PostgreSQL, MySQL, and MariaDB targets. On other targets a drifted field is recorded and warned about, and you add the column yourself.

A one-time migration cannot alter its table between batches, so it only reports: if the bulk load sees fields the 500-document sample missed, it names them in a warning when the migration completes.

Pausing and resuming live sync

Click Pause on any active CDC channel to temporarily halt replication. The worker stops consuming from the oplog immediately. Click Resume to continue from the exact oplog position — the saved resume token ensures no data loss or duplication.

CDC vs one-time migration

Use one-time migration for snapshot copies where you don't need ongoing sync. Use Live CDC when you need:

  • Zero-downtime cutover — run both databases in parallel during migration
  • Continuous sync — keep your SQL target up to date with production writes
  • Incremental migration — bulk load history, then CDC for ongoing changes

CDC worker internals

The CDC worker runs as a BullMQ job in the cdc queue. It has 5 retry attempts with exponential back-off (10s initial delay). If the MongoDB connection drops, the worker reconnects and resumes from the last saved token. The worker never processes the same event twice — idempotent writes prevent duplicates.