The Outbox Pattern — Reliable Event Publishing in Microservices

The 03:00 AM Ghost in the Transaction

At exactly 03:14 UTC on a Tuesday, our consistency model collapsed. We were running a standard distributed commerce flow: a user places an order, the order-service persists the record to a PostgreSQL RDS instance, and immediately after repository.save(order), we fired a message to Kafka via a standard KafkaTemplate.send() call.

The logs showed a TransactionSystemException. A momentary network blip prevented the database commit from completing within the allotted timeout, but the Kafka producer had already received an ACK from the broker. We essentially told the manufacturing downstream that an order existed, but the database rolled back. We sold inventory we didn't officially own.

This is the classic "Dual-Write Problem." You cannot atomically commit to a relational database and a message broker simultaneously without an XA transaction—which no sane engineer wants to manage in a high-throughput microservice environment. If the DB fails after the message is sent, you have a ghost message. If the message fails after the DB commits, you have a silent data loss.

Anatomy of a Failed Sync: The Timeline

To understand why simple retry logic fails, look at the sequence of events during our outage.

1. T+0ms: order-service starts a @Transactional block. 2. T+45ms: INSERT INTO orders ... completes in the DB (uncommitted). 3. T+50ms: kafkaTemplate.send("orders-topic", payload) is called. 4. T+110ms: Kafka broker ACKs the message. 5. T+115ms: The service attempts to commit the Postgres transaction. 6. T+215ms: Postgres detects a deadlock or a transient connection failure. 7. T+220ms: Transaction rolls back.

The result: Kafka has the event, the DB does not. Downstream services start picking up the "Order Created" event, query the order-service for details via REST, and receive a 404 Not Found.

The root cause isn't "bad code"—it's the fundamental impossibility of distributed consensus across heterogeneous storage systems without a unified coordinator.

Implementing the Persistence-First Outbox

To solve this, we moved the "publishing" of the event into the same ACID transaction as the business logic. Instead of calling Kafka directly, we write the event to a dedicated outbox table in the same PostgreSQL schema.

CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(255) NOT NULL,
    aggregate_id VARCHAR(255) NOT NULL,
    type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    processed BOOLEAN DEFAULT FALSE
);

By doing this, the event and the order are either both committed or both rolled back. There is no middle ground. The service code now looks like this:

```java
@Service
@RequiredArgsConstructor
public class OrderService {
    private final OrderRepository orderRepository;
    private final OutboxRepository outboxRepository;
    private final ObjectMapper mapper;

@Transactional public void placeOrder(OrderRequest request) { // 1. Persist the business entity var order = Order.builder() .status(OrderStatus.CREATED) .customerId(request.customerId()) .total(request.total()) .build(); orderRepository.save(order);

// 2. Persist the event in the same transaction var event = OrderCreatedEvent.from(order); var outboxEntry = OutboxEntry.builder() .id(UUID.randomUUID()) .aggregateType("ORDER") .aggregateId(order.getId().toString()) .type("ORDER_CREATED") .payload(mapper.valueToTree(event)) .build(); outboxRepository.save(outboxEntry); } } ```

Bridging the Gap with Debezium and CDC

Writing to the outbox table is only half the battle. We still need to get that data into Kafka. While you could write a background poller (Select/Update loop), that approach scales poorly and puts unnecessary read pressure on the primary database.

We opted for Change Data Capture (CDC) using Debezium. Debezium acts as a Kafka Connect source that tails the PostgreSQL Write-Ahead Log (WAL). It treats every INSERT into the outbox table as an event stream.

+-----------------+      +-----------------+      +------------------+
| order-service   |      | PostgreSQL      |      | Debezium (Kafka  |
|                 |      |                 |      | Connect)         |
|  [ Transaction ]|----->| [ orders ]      |      |                  |
|                 |      | [ outbox ] <-----------|-- Tailing WAL    |
+-----------------+      +-----------------+      |        |         |
                                                  +--------|---------+
                                                           |
                                                  +------------------+
                                                  | Kafka Cluster    |
                                                  |                  |
                                                  | [ orders-topic ] |
                                                  +------------------+

Our Kafka Connect configuration for the connector identifies the specific table to watch:

{
  "name": "orders-outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db.production.internal",
    "database.dbname": "orders_db",
    "table.include.list": "public.outbox",
    "topic.prefix": "cdc",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.table.field.event.id": "id",
    "transforms.outbox.table.field.event.key": "aggregate_id",
    "transforms.outbox.table.field.event.payload": "payload",
    "transforms.outbox.route.topic.replacement": "orders.events"
  }
}

By leveraging EventRouter Single Message Transformation (SMT), Debezium automatically unwraps the Postgres-specific CDC envelope and routes the JSON payload directly to the orders.events topic.

The Cost of Guaranteed Delivery: Idempotency

One uncomfortable truth about the Outbox pattern is that it guarantees at-least-once delivery. If Debezium reads the WAL, pushes to Kafka, but the Kafka Connect offset commit fails, it will re-read and re-push the same event.

We learned this the hard way when a consumer service started processing duplicate payments. The fix was mandatory idempotency on the consumer side. Every consumer must track processed message IDs.

In our payment-service, we use a simple uniqueness constraint in the database to discard duplicates:

@KafkaListener(topics = "orders.events")
@Transactional
public void onOrderCreated(OrderCreatedEvent event, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) String eventId) {
    if (processedEventRepository.existsById(eventId)) {
        log.warn("Duplicate event detected: {}", eventId);
        return;
    }
    
    // Process business logic...
    
    processedEventRepository.save(new ProcessedEvent(eventId));
}

Hard-Won Performance Lessons from Production

When we first deployed this, we saw a noticeable spike in database CPU usage and disk I/O. Tracking it down revealed two issues that almost derailed the migration.

First, the outbox table grows indefinitely if you don't prune it. Debezium reads the WAL, not the table itself, so the records remain in the table after being "captured." We initially tried a Cron job to delete rows where created_at < NOW() - INTERVAL '1 day', but at high volumes (1,000+ orders/minute), the mass deletion caused massive VACUUM pressure in Postgres, locking the table and spiking p99 latency.

The realization: You don't actually need to keep the rows after they are in the WAL. However, Postgres needs the rows to exist long enough for the replication slot to advance. We switched to a more surgical pruning strategy, deleting in small batches (500 rows/transaction) during off-peak hours, which dropped our p99 from 480ms back down to 90ms.

Second, the Write-Ahead Log (WAL) level must be set to logical. If you're on AWS RDS, this requires a reboot and setting rds.logical_replication = 1. This increases the WAL size significantly. We had to increase our disk throughput (IOPS) by 30% to handle the additional write amplification of double-writing every business event.

Why We Won't Go Back

The Outbox pattern isn't free. It adds architectural complexity (Kafka Connect), increases storage overhead, and demands idempotent consumers. However, the peace of mind it provides is unmatched. Prior to this, we spent roughly 4 hours a week manually reconciling "lost" orders between the DB and Kafka. Since the full adoption of the CDC-based Outbox, that number has dropped to zero.

If your system relies on "fire and forget" messaging after a database commit, you aren't running a reliable system—you're running an eventually inconsistent system where "eventually" is a gamble. The Outbox pattern turns that gamble into a guarantee by shifting the reliability burden from the application code to the database's own transaction logs.

The final shift in our perspective was treating the database not just as a store for current state, but as the primary source of truth for the communication intent. Once you accept that the outbox table is as critical as your users or orders table, the operational overhead becomes a standard part of maintaining a healthy, resilient distributed system.