Live Sync (CDC)

Real-time change data capture from MongoDB oplog to your SQL target.

How CDC works in NoSQLSync

NoSQLSync uses Change Data Capture to stream real-time changes from your MongoDB oplog to your SQL target:

  • The CDC worker opens a change stream on your MongoDB collection, tailing the oplog
  • Each insert, update, or delete event is captured and transformed according to your schema mapping
  • Events are written to the SQL target via batched statements for performance
  • Resume tokens are saved to the database — if the worker restarts, it picks up exactly where it left off

Setting up MongoDB oplog access

CDC requires your MongoDB deployment to have the oplog accessible. For MongoDB Atlas, the oplog is available by default. For self-hosted MongoDB, ensure your cluster runs as a replica set — standalone instances do not expose the oplog.

Verify oplog access:

db.getSiblingDB("local").oplog.rs.find().limit(1)

Monitoring replication lag

The Live Sync screen shows each active CDC channel with real-time status:

  • Green dot — channel is running and consuming events
  • Amber dot — channel is paused
  • Gray dot — channel is stopped

On the migration detail page, the SSE log stream displays per-event timestamps. Typical latency is under 50ms from oplog entry to SQL write.

Handling schema changes mid-sync

If your MongoDB documents gain new fields while CDC is active, the worker handles them based on your schema configuration:

  • Flexible mode: Unknown fields are added as nullable columns automatically
  • Strict mode: Unknown fields are logged and skipped — no schema change
  • JSONB fallback: New fields stored in a dedicated metadata JSONB column

Pausing and resuming live sync

Click Pause on any active CDC channel to temporarily halt replication. The worker stops consuming from the oplog immediately. Click Resume to continue from the exact oplog position — the saved resume token ensures no data loss or duplication.

CDC vs one-time migration

Use one-time migration for snapshot copies where you don't need ongoing sync. Use Live CDC when you need:

  • Zero-downtime cutover — run both databases in parallel during migration
  • Continuous sync — keep your SQL target up to date with production writes
  • Incremental migration — bulk load history, then CDC for ongoing changes

CDC worker internals

The CDC worker runs as a BullMQ job in the cdc queue. It has 5 retry attempts with exponential back-off (10s initial delay). If the MongoDB connection drops, the worker reconnects and resumes from the last saved token. The worker never processes the same event twice — idempotent writes prevent duplicates.