What is Transaction Log Processing (and Why It Matters)
What is Transaction Log Processing (and Why It Matters)
Your analytics dashboard says "last synced: 6 hours ago." Your search index is showing products that sold out this morning. A customer just got double-charged because the fraud detection pipeline was running on yesterday's transactions.
These are symptoms of the same problem: your data lives in a database, but the systems that need it are working with stale copies.
The fix has been running inside your database the whole time. Every production database maintains a sequential record of every change it makes. PostgreSQL, MySQL, MongoDB, SQL Server, Oracle. They all do it. These records exist for durability, replication, and crash recovery (the exact role varies by database), but they turn out to be something more valuable: a real-time stream of everything that happened to your data, in exact order, with full transactional context.
Transaction log processing is the practice of reading that stream and putting it to work.
What is a Transaction Log?
A transaction log is an append-only file where a database records every change before applying it to the actual data files.
The keyword is before. When you execute an INSERT, your database doesn't immediately write to the table on disk. That would be dangerous. If power failed mid-write, the table would be left in a corrupted, half-updated state. Instead, the database writes the change to a sequential log in memory, and when the transaction commits, flushes that log to disk (via fsync or an equivalent). Only after the log is safely on durable storage does the database acknowledge the write back to your application.
The actual data files get updated later, asynchronously, by a background process. This is the write-ahead pattern: log first, apply second.
If the database crashes after the log is flushed but before the data files are updated, it replays the log on startup and the data is intact. If it crashes before the flush, the transaction never committed. Your application got an error, not a silent corruption.
WAL, Binlog, Oplog
Every major database keeps an ordered, durable record of changes. What it's called and exactly what job it does varies.
PostgreSQL calls it the WAL (Write-Ahead Log). WAL data lives in the pg_wal directory as a series of 16MB segment files (configurable at database initialization via --wal-segsize). Each change gets a Log Sequence Number (LSN), a monotonically increasing position marker that tells you exactly where in the log a change occurred.
MySQL and MariaDB call it the binary log (binlog). Unlike PostgreSQL's WAL, the binlog is not the crash-recovery mechanism (InnoDB has its own redo log for that). The binlog's role is replication and point-in-time recovery. It supports three formats: statement-based (logs the SQL text), row-based (logs the actual row changes), and mixed. For Change Data Capture, row-based is the only reliable format. It records what actually changed, not what SQL was executed. Consider a statement like UPDATE users SET active = false WHERE last_login < '2024-01-01'. That might affect 50,000 rows. Statement-based logging records one SQL string. Row-based logging records all 50,000 individual row changes. Only the second version gives a downstream consumer the full picture. Worth noting: as of MySQL 8.0.34, the binlog_format setting is deprecated and row-based is becoming the only format going forward.
MongoDB calls it the oplog (operation log). The oplog is a capped collection in the local database, sized by default to 5% of physical memory (capped at 50 GB). Write operations are recorded as idempotent documents, meaning you can replay them safely without creating duplicates. MongoDB's change streams API provides a higher-level interface on top of the oplog. Like the binlog, the oplog is not MongoDB's crash-recovery mechanism. That role belongs to the WiredTiger journal, which implements the actual write-ahead pattern, recording every operation between checkpoints and replaying them on crash. The oplog exists for replication, which is what makes it the right thing to read for CDC. One important caveat: change streams require a replica set or sharded cluster. A standalone MongoDB instance does not maintain an active oplog, so there's nothing for change streams to read from.
SQL Server and Oracle have their own equivalents (the transaction log and redo logs, respectively), each with their own position tracking and decoding mechanisms.
Different names, same core idea. An ordered, durable, append-only record of every mutation. That record is the foundation of transaction log processing.
What is Transaction Log Processing?
Transaction log processing is the practice of reading a database's transaction log and converting its raw contents into structured events that other systems can consume.
Every database has a transaction log. But having a log and processing that log are different things.
These logs serve different native purposes depending on the database. PostgreSQL's WAL handles both crash recovery and replication. MySQL's binlog and MongoDB's oplog handle replication and point-in-time recovery (crash recovery is handled separately by their storage engines). But in every case, the log's native consumers are the database's own internal mechanisms. To turn it into something external systems can use, you need a processor that connects via the database's replication protocol, decodes the raw entries into human-readable events, and delivers them to wherever they need to go. Kafka, Elasticsearch, a data warehouse, another database, an S3 bucket.
That's what a transaction log processor does. The log is the source. The processor reads, decodes, and delivers.
Three Core Use Cases
Change Data Capture (CDC). Streaming row-level changes to external systems in real time. A row changes in PostgreSQL, and within milliseconds the event appears in Kafka. CDC is the most common use case driving adoption of transaction log processing today, and it's the one most teams encounter first.
Replication. Replaying changes into another database. This can be the same engine (logical replication) or a completely different one: PostgreSQL to MySQL, MongoDB to Elasticsearch. The transaction log is the source of truth. The target database is a derivative. This works because the processor normalizes events into a universal format that any destination can consume.
Archival and Recovery. Storing decoded events in durable storage so you can restore data to any specific point in time. Someone accidentally ran DELETE FROM orders without a WHERE clause at 3:47 PM? Replay the archived events up to 3:46 PM and the data is back. The log becomes a time machine for your data.
Why Transaction Log Processing Matters Now
Transaction logs have existed for decades. The databases that write them have been around since the 1970s. So why is transaction log processing on every data engineering roadmap right now?
Batch ETL Hit a Wall
For most of the 2000s and 2010s, the standard way to move data between systems was batch ETL: Extract, Transform, Load. A scheduled job runs every hour (or every night), queries the source database for everything that changed, transforms it, and loads it into the destination.
This worked when "yesterday's data" was good enough. For many teams, it no longer is. The problems stack up:
Staleness. An hourly batch means downstream systems are showing data up to 59 minutes old. For e-commerce, that means selling products already out of stock. For financial services, that means displaying incorrect balances.
Expense. Figuring out "what changed since last time" without a log means comparing entire datasets. Unless every table has a reliable updated_at column (and in practice, many don't), you're running full table scans against the production database. That costs CPU cycles, saturates network links, and inflates cloud bills.
Invisible deletes. If you query for records that exist, you won't find the ones that were deleted. Many batch pipelines miss deletions entirely, leading to ghost data that accumulates in downstream systems and never gets cleaned up.
Lost transaction context. A single business operation might insert an order, update inventory, and create a shipment record across three tables. Batch ETL treats each table independently. Downstream consumers see fragments, never the complete atomic operation.
Transaction log processing solves all four. The log contains every change (including deletes), in commit order, with full transaction context, in real time. Reading the log adds no query load to the source database and minimal overhead (though it does require resources like replication slots that need monitoring). Staleness drops from hours to milliseconds.
Real-Time Data as a Competitive Edge
Fraud detection models running on hour-old data miss patterns that real-time processing catches, like a card being used in two countries within the same minute. Search indexes that lag behind the source database show users products that no longer exist. Recommendation engines that update nightly can't respond to what a user did five minutes ago.
The gap between "real-time" and "last night's data" keeps widening as user expectations rise and the systems competing for their attention get faster.
AI and ML Pipelines Need Fresh Data
The rise of AI workloads is accelerating this shift. RAG (Retrieval-Augmented Generation) systems need knowledge bases that reflect the latest data, not last night's snapshot. Feature stores powering ML models need fresh signals. A feature computed on stale data produces stale predictions. Vector databases backing semantic search need to index new content as it's created, not hours later.
All of these systems depend on the same thing: getting changes from the source database to the target system quickly, completely, and in order. Transaction log processing is the most efficient way to feed them, because the log is already there, already ordered, and already complete. No polling. No full scans. No missed deletes.
How Transaction Log Processing Works
There are two fundamentally different ways to process a transaction log, and understanding the difference matters because they serve completely different purposes.
Two Modes: Logical and Physical
Logical decoding parses the raw log bytes into structured, human-readable events. You get individual operations: "INSERT into users table, row ID 42, name = 'Alice', email = 'alice@example.com'." These events can be filtered (only the users table), transformed (mask the email field), and routed to different destinations (orders to Kafka, users to Elasticsearch).
In PostgreSQL, logical decoding works through pgoutput, the built-in output plugin that converts WAL records into a stream of structured change messages over the logical replication protocol. MySQL uses row-based binlog parsing. MongoDB exposes change streams over the oplog.
Physical streaming copies the raw log bytes without decoding them. You can't filter, transform, or route them. They're opaque binary data, meaningful only to the exact same database engine and major version that produced them. The sole purpose is disaster recovery: stream the raw bytes to durable storage, and if the primary database dies, restore from a base backup and replay the bytes to recover to any point in time.
Most production setups need both. Logical decoding feeds downstream systems in real time. Physical streaming handles disaster recovery and full-cluster restore. They complement each other because they solve fundamentally different problems.
What a Change Event Looks Like
When a user updates their email address, the processor decodes the log entry and produces a structured event:
{
"operation": "UPDATE",
"table": "users",
"timestamp": "2025-06-10T14:30:22Z",
"transaction_id": "tx_8842",
"before": { "id": 42, "email": "alice@old.com" },
"after": { "id": 42, "email": "alice@new.com" }
}
The event tells you what changed (email), what it changed from, what it changed to, and when. The transaction_id groups this change with other changes that happened in the same database transaction, so an order insert and an inventory decrement can be treated as a single atomic unit.
This normalized format is what downstream systems receive, regardless of whether the source is PostgreSQL, MySQL, or MongoDB.
What Happens Between Capture and Delivery
Between reading the log and delivering events to their destination, a transaction log processor typically handles several responsibilities: filtering out events the destination doesn't need, transforming sensitive fields (masking, renaming), preserving transaction boundaries so consumers don't see partial commits, managing delivery guarantees (at-least-once or exactly-once), and saving its position so it can recover cleanly after a crash.
Each of these is a topic in its own right. We cover them in detail in the articles that follow: delivery guarantees, backpressure, transaction boundaries, checkpointing and crash recovery, and deduplication strategies.
What to Look for in a Transaction Log Processor
Several tools exist in this space, each with different trade-offs.
Debezium is the most established open-source option. Its primary deployment model runs on Kafka Connect, which means running Kafka, Kafka Connect, and (depending on version) ZooKeeper as infrastructure dependencies. Debezium also offers lighter-weight alternatives: Debezium Server, a standalone process that sends events directly to sinks like Kinesis, Pub/Sub, or Redis without Kafka, and the embeddable Debezium Engine for custom applications. Still, the Kafka Connect path is where most of the ecosystem and documentation lives. For teams already operating Kafka, this is manageable. For teams that aren't, even the lighter options require evaluating the trade-offs carefully.
Managed and hybrid services like AWS DMS, Fivetran, and Airbyte reduce operational burden. AWS DMS and Fivetran are fully managed. Airbyte is open-source and self-hostable, with a managed cloud tier for teams that don't want to run infrastructure. The trade-offs vary: fully managed services mean less control and potential vendor lock-in, while self-hosted options like Airbyte OSS still require you to provision and maintain the underlying infrastructure (typically Kubernetes). In all cases, when something breaks (and in data pipelines, things break), the question is how much visibility you have into what went wrong.
Custom implementations give you maximum control. The trade-off is time. Correctly handling replication slot management, schema changes, large transactions, backpressure, and crash recovery takes months of focused engineering work. Most teams underestimate this.
When evaluating any transaction log processor, the questions that actually matter are:
- What databases does it support? Is the architecture designed to add new sources, or is each database a ground-up implementation?
- What are the delivery guarantees? At-least-once? Exactly-once with specific sinks? What happens on crash?
- What's the operational footprint? A single binary, or an ecosystem of supporting services?
- How does it handle backpressure? When a destination can't keep up, does the pipeline drop events, buffer until memory runs out, or apply controlled flow management?
- Does it preserve transaction boundaries? Or does it stream events individually, leaving downstream systems to reconstruct atomicity on their own?
Where to Go from Here
Transaction log processing is a foundational layer of real-time data infrastructure. The log is already there. Your database writes it on every commit. The question isn't whether to use it, but how.
The articles that follow go deeper into each concept introduced here: how WAL, binlog, and oplog work under the hood, what Change Data Capture is and how it compares to batch ETL, how delivery guarantees work in practice, and how point-in-time recovery turns your log into a time machine for your data.
We built Remac because we needed a transaction log processor that runs as a single binary, preserves transaction boundaries, and doesn't require a Kafka cluster just to get started. Remac currently supports PostgreSQL, with MySQL, MongoDB, SQL Server, and Oracle on the roadmap.
If that sounds like a fit, take a look at remac.io.
Further Reading:
- PostgreSQL: Write-Ahead Logging (WAL). Official PostgreSQL documentation on WAL internals.
- MySQL Binary Logging Formats. Official MySQL documentation on statement, row, and mixed binlog formats.
- MongoDB Replica Set Oplog. Official MongoDB documentation on oplog structure and behavior.