Seven guarantees.
Zero exceptions.
Every transaction log processor makes implicit promises about your data. We make ours explicit, write tests for each one, and prove them under hostile conditions.
Completeness
LogicalEvery committed change is captured. Nothing slips through.
Ordering
LogicalEvents arrive in commit order. Always.
Delivery
LogicalAt-least-once delivery. Effectively-once when your sink supports idempotent writes.
Resumability
BothAfter any failure, processing resumes from the last confirmed position. No full re-sync required.
Consistency
LogicalThe target eventually matches the source at every point in time.
Liveness
BothThe pipeline makes forward progress. It never silently stalls.
Recoverability
PhysicalArchives are complete, gapless, and can restore to any point in the recovery window.
The cost of getting it wrong
Pipeline failures aren't edge cases. They're the norm.
74 failure modes. Cataloged, tested, and growing.
We've mapped every way a transaction log processor can break. Each category below represents a class of failures we actively test against. This catalog isn't exhaustive yet, and we're adding to it continuously.
Source failures
9 modesConnection drops, slot invalidation, disk pressure, DDL mid-stream, failover races, snapshot degradation.
Pipeline failures
10 modesTransaction buffer overflow, orphaned transactions, deduplication exhaustion, backpressure deadlock, thread leaks.
Sink failures
8 modesWrite timeouts, partial batch writes, schema mismatches, connection pool exhaustion, broker unavailability.
Middleware failures
20 modesFilter false-drops, transform mutations, enrichment staleness, routing conflicts, hot-reload races.
Checkpoint failures
6 modesCorruption, checkpoint-ahead-of-sink, concurrent writers, lease expiry, storage exhaustion.
Storage failures
6 modesPartial uploads, read-after-write inconsistency, credential expiry, bit-rot, prefix collision.
Recovery failures (physical)
6 modesArchive gaps, unreachable restore targets, snapshot corruption, timeline divergence, config conflicts.
Recovery failures (logical)
5 modesReplay ordering violations, sink failure during replay, event format evolution, cross-database type mismatches.
Configuration failures
4 modesInvalid hot-reload, config drift, secret exposure in logs, default value mismatches.
Total: 74 cataloged failure modes across 9 categories. This number grows with every release.
Real incidents. Real consequences.
These aren't hypotheticals. They're documented failures in production transaction log and replication systems.
Six hours of production data lost
PostgreSQL replication lag caused the secondary to fall behind. WAL segments were removed before the secondary could consume them. During recovery, an engineer accidentally deleted the primary database. Multiple backup mechanisms had silently failed, including unusable database exports and unconfigured cloud snapshots.
Keepalive flush bug forces full database re-syncs
A driver-level keepalive feature silently advanced the replication slot past the connector's tracked offset. On restart, the saved offset was behind the slot's confirmed position, forcing full re-syncs of entire databases. Root cause: position tracked in two places with no consistency guarantee between them.
CDC replication completely broken after maintenance update
A Redshift maintenance update surfaced three separate bugs that made CDC non-functional across all DMS tasks. Primary key columns were hardcoded to varchar(20) regardless of source type. Any key exceeding 20 characters caused immediate task failure.
43-second network split causes 24 hours of degraded service
A routine maintenance task severed connectivity for 43 seconds, triggering an automated MySQL failover. Both data centers accepted writes during the partition, creating divergent datasets. Reconciling these writes took over 24 hours of degraded service.
Unmanaged replication slot fills source database disk
A replication slot was not properly managed, causing transaction log files to grow until the source database ran out of storage and stopped accepting writes. Restarting DMS required a full table reload, extending downtime to several hours.
Two-day outage causes unrecoverable data gaps
A control plane outage lasted approximately 40 hours. Datasets that were not replicated across regions experienced persistent gaps. Log push features suffered unrecoverable data loss for the majority of the event.
Write-ahead log pathology causes 73-hour total outage
High read/write load on Consul triggered a pathological performance issue in BoltDB, the underlying write-ahead log storage. Deleted log entries left free pages that were never reclaimed, ballooning write times from milliseconds to seconds. A single Consul cluster supporting multiple workloads amplified the blast radius to a 73-hour total outage.
24-hour global outage with confirmed customer data loss
An unsupervised global update restarted infrastructure, disconnecting 50–60% of production Kubernetes nodes. Intake pipelines lacked disk-based persistence, so data existed only in memory or local disk. Losing nodes meant losing data. Replicated stores were unable to accept writes, causing memory buffers to overflow. Datadog confirmed "a limited but non-zero amount of customer data" was lost in unrecoverable ways.
Bulk user removal triggers replication lag cascade and repeated OOM kills
A customer removed a large number of users from their workspace, overloading the database cluster with queries. Replication lag appeared as replicas fell behind the primary. The high write load exhausted memory on the shard primary, triggering an OOM kill. The promoted replica also OOM-killed under the same load, and the cycle repeated.
Failure doesn't come one at a time.
Real outages are compound. These are some of the scenarios we actively design and test against, because single-fault tolerance isn't enough.
Schema change during active streaming
Columns are added, dropped, retyped, or renamed while events stream. The decoder must pick up each new RELATION message so that every insert, update, delete, and truncate afterward keeps decoding against the current schema.
Checkpoint store failure + slow sink
The checkpoint store goes down while a sink is experiencing high latency. The pipeline must continue processing, buffer checkpoint state internally, and never advance the source position past what has been durably persisted.
Primary failover + slot loss
The primary database fails over to a standby. If the replication slot was not synchronized, events between the old slot position and the failover point are at risk. Detection must be immediate, not discovered days later in a data audit.
Multi-sink partial delivery
An event is routed to two sinks simultaneously. The first write succeeds, but the second upload fails mid-transfer. The checkpoint state must account for this partial delivery. The answer must be deterministic, not implementation-dependent.
Credential expiry during long archive job
An authentication token expires mid-upload of a large archive segment to cloud storage. Partial uploads must be aborted cleanly, credentials refreshed, and archiving resumed without leaving orphaned data or creating archive gaps.
Discipline, not promises.
The Gold Standard for Transaction Log Processing isn't a tagline. It's an engineering commitment.
We took our testing philosophy from SQLite, the most widely deployed database in the world. SQLite ships hundreds of lines of test code for every line of production code.
We don't need that ratio, but we share the same responsibility: we handle other people's data. A transaction log processor that silently drops events, delivers them out of order, or fails to resume after a crash is worse than having no processor at all.
Engineering Principles
- Every cataloged failure mode is a tracked test target; coverage is measured, not assumed.
- No bug is fixed without a failing test that reproduces it first.
- Failure test cases outnumber success test cases 70/30 for I/O code.
- Crash recovery is tested by SIGKILLing the process mid-stream and verifying it resumes from its last durable checkpoint with no loss.
- Network partitions, latency, and disconnects are injected between the pipeline and every dependency with a real fault-injection proxy.
- A flaky test is treated as a production bug, investigated and fixed, never skipped.
- Tests are the specification. If a behavior is not tested, it is not guaranteed.
Deploy once. Forget the pipeline exists.
Your data arrives where it needs to be, in order, with delivery guarantees — and your team never thinks about it again.