MongoDB Oplog Explained
Understand MongoDB's oplog — how it works, how to tail it, and how NoSQLSync uses it for real-time CDC replication to SQL databases.
What is the MongoDB Oplog?
The oplog (operations log) is a special capped collection that lives in the local database of every MongoDB replica set member. It records every write operation — inserts, updates, and deletes — in the order they were applied to the primary node. Secondary nodes use the oplog to replicate changes and stay in sync with the primary.
Oplog Structure
Each oplog entry contains: a timestamp (when the operation occurred), the operation type (i for insert, u for update, d for delete), the namespace (database.collection), and the document data. For updates, it includes the filter query and the changes applied. For inserts, it includes the full document.
Tailing the Oplog
To tail the oplog, a client opens a tailable cursor on the oplog collection — similar to tail -f on a log file. The cursor blocks until new entries arrive, then returns them. This is how CDC tools (including NoSQLSync) capture MongoDB changes in real time without polling the collections.
Oplog Size and Retention
The oplog is a capped collection with a fixed size (default 5% of free disk space on Linux). When the oplog reaches its size limit, the oldest entries are overwritten. This means CDC tools need to keep up — if a CDC tool falls behind by more than the oplog window, it loses changes and must re-sync from scratch.
NoSQLSync's Oplog-Based CDC
NoSQLSync connects to your MongoDB replica set and establishes a tailable cursor on the oplog. It filters entries for your selected collections, transforms BSON documents to SQL rows using your schema mapping, and writes them to the target database — all with sub-second latency. The system monitors oplog lag and alerts if it approaches the oplog retention window.