MongoDB Change Streams: Real-Time Change Tracking
How MongoDB Change Streams work, how they differ from tailing the oplog, and how NoSQLSync uses them for CDC replication.
What are MongoDB Change Streams?
Change Streams are MongoDB's official API for watching real-time changes on collections, databases, or entire deployments. They provide a high-level, resumable, filtered stream of change events — inserts, updates, replaces, deletes, and even DDL operations like dropping a collection.
Change Streams vs Tailing the Oplog
Change Streams are built on top of the oplog but provide a much nicer API. Benefits over raw oplog tailing: resumable via resume tokens (survive disconnections and replica set elections), filterable (watch only specific collections or operation types), durable (if your app crashes, resume from the last token), and full documents available for updates (not just the delta). The tradeoff: slightly more overhead than raw oplog tailing.
How Change Streams Work Under the Hood
When you open a change stream, MongoDB creates a cursor on the oplog that watches for matching entries. The stream emits change events as they're written to the oplog. A resume token is essentially a pointer to an oplog position — it lets you restart the stream from exactly where you left off.
Using Change Streams with NoSQLSync
NoSQLSync uses MongoDB Change Streams as its primary CDC mechanism when available (MongoDB 3.6+). For older MongoDB versions, it falls back to oplog tailing. The stream watches your selected collections and emits every change event to NoSQLSync's transformation pipeline, which maps the BSON data to SQL and writes to the target.
Change Stream Event Types
insert: a new document was created (full document provided), update: an existing document was modified (update description + optional full document), replace: an entire document was replaced, delete: a document was removed (document key provided), invalidate: the watched collection was dropped or renamed (stream must be re-established), and drop: a collection was dropped.