Event-Driven Architecture with Spring Boot and Kafka — Building Reactive Distributed Systems

The Choreography of Decoupled States

When you move from synchronous REST-based microservices to an event-driven architecture (EDA), you aren't just changing a network protocol; you are shifting the source of truth from a database state to a log of transitions. In a traditional request-response model, the OrderService calls the InventoryService. If the inventory database is locking a row or the network jitters, the order fails. This is temporal coupling.

A reactive distributed system built on Spring Boot and Kafka eliminates this by treating events as first-class citizens. The OrderService emits an OrderPlaced event and immediately moves on. It doesn't care who consumes it. The system becomes a set of autonomous actors reacting to a distributed, append-only commit log.

    [ Producer: Order Service ] 
               |
         (Topic: orders.v1)
               |
    +----------+----------+
    |                     |
[ Consumer A ]       [ Consumer B ]
(Inventory)          (Analytics)

At 10,000 feet, the architecture is a Directed Acyclic Graph (DAG) of state transitions. Kafka acts as the ultimate buffer, absorbing spikes that would otherwise collapse downstream services. However, the complexity doesn't disappear; it shifts to the edge. You trade the "Distributed Monolith" problem for the "Eventual Consistency" problem. Your system must now handle out-of-order delivery, schema evolution without downtime, and the reality that every consumer might see the same message three times.

Reliable Producers and the Outbox Pattern

The most common failure point in Spring Boot-Kafka systems is the "Dual Write" problem. An engineer writes code that saves an entity to PostgreSQL and then calls kafkaTemplate.send(). If the database commit succeeds but the Kafka broker is unreachable (or the pod restarts before the send returns), your system is now inconsistent. The downstream services will never know that order exists.

To solve this, you must implement the Transactional Outbox pattern. Instead of sending directly to Kafka, you write the event to a dedicated outbox table in the same local database transaction as your business logic. A separate process (like a Debezium connector or a scheduled Spring task) then polls this table and publishes to Kafka.

When configuring the KafkaTemplate, you must tune for durability over raw throughput if your data is financial or transactional. This means setting acks=all and min.insync.replicas=2.

```java
@Configuration
public class KafkaProducerConfig {

@Bean public ProducerFactory<String, OrderEvent> producerFactory() { Map<String, Object> config = new HashMap<>(); config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker-1:9092"); config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class); // Critical for consistency config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); config.put(ProducerConfig.ACKS_CONFIG, "all"); config.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE); config.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120000); // 2 mins return new DefaultKafkaProducerFactory<>(config); } } ```

The enable.idempotence=true setting is non-negotiable in modern versions. It assigns a producer ID and sequence number to every batch, ensuring that if a retry occurs due to a network glitch, the broker discards the duplicate. Without this, a 300ms network timeout could result in double-billing a customer.

Schema Evolution and the Avro Contract

If you send JSON over Kafka in a distributed system, you are inviting a production outage. JSON is too flexible; a developer in the Payments team might rename user_id to customer_id, and suddenly the Shipping service—which hasn't been touched in six months—starts throwing NullPointerExceptions and crashing in a loop.

You must use a Schema Registry (Confluent or Apicurio) and a binary format like Avro or Protobuf. In a Spring Boot environment, this means the OrderService fetches the schema from the registry at startup (or on demand) and serializes the message into a compact binary format. The consumers download the schema and deserialize it.

A hard-won lesson: Always use Forward Compatibility. This ensures that new versions of a schema can still be read by old consumers. If you deploy your services in a rolling fashion, you will inevitably have v2 producers and v1 consumers running simultaneously. If your schema change isn't backward/forward compatible, your v1 consumers will start hitting the Dead Letter Queue (DLQ) immediately upon the first v2 message arrival.

Consumer Idempotency and the Offset Trap

Kafka guarantees "at-least-once" delivery by default. It does *not* guarantee "exactly-once" delivery unless you use the Transactional API, which carries significant performance overhead. For most high-scale systems, the better approach is to make your consumers idempotent.

In Spring Kafka, consumers track their progress via offsets. If a consumer processes a message but crashes before committing the offset back to Kafka, the new consumer instance will re-process that same message.

To handle this, use a "Unique Command ID" or the Kafka record's key as a guard in your database. Before processing, check a processed_messages table:

```java
@KafkaListener(topics = "orders.v1", groupId = "inventory-service")
@Transactional
public void consumeOrder(OrderEvent event, Acknowledgment ack) {
    String messageKey = event.getOrderId();
    
    if (idempotencyRepository.existsById(messageKey)) {
        log.warn("Duplicate message detected: {}", messageKey);
        ack.acknowledge();
        return;
    }

// Business Logic inventoryService.reserveStock(event); // Guard against future duplicates idempotencyRepository.save(new ProcessedMessage(messageKey)); ack.acknowledge(); } ```

Notice the Acknowledgment ack. In production, you should set ack-mode: manual_immediate. If you rely on the default auto-commit, Spring will commit the offset as soon as the listener method returns, even if your database transaction fails later in the pipeline. By using manual acknowledgments, you ensure the offset is only moved forward when you are certain the data is persisted.

Poison Pills and Non-Blocking Retries

A "Poison Pill" is a message that a consumer cannot process (e.g., malformed data or a logic bug). If a consumer encounters a poison pill, it will retry indefinitely, blocking the entire partition. This is a common cause of "Kafka Lag" alerts.

Do not use a simple try-catch and log the error; you will lose data. Instead, implement a Non-Blocking Retry with DLT (Dead Letter Topic).

Spring Kafka offers the DefaultErrorHandler and DeadLetterPublishingRecoverer. If a message fails after $N$ attempts, Spring captures the exception, attaches the stack trace to the message headers, and publishes it to a topic_name.DLT. This allows the consumer to move to the next message while developers investigate the failure in the DLT.

One specific optimization we implemented in a high-throughput system: Exponential Backoff. Retry 1: 1 second Retry 2: 10 seconds Retry 3: 60 seconds Then move to DLT.

This prevents your service from hammering a database that is already struggling. When the database recovers, the service continues processing new messages, and you can later "replay" the DLT messages once the root cause is fixed.

Rebalancing and the Partition Count

The performance of your distributed system is ultimately capped by your partition count. If you have a topic with 3 partitions, you can only have 3 active consumer instances in a group. Scaling to 10 pods won't help; 7 will sit idle.

However, adding partitions later is dangerous if you rely on Key-based ordering. Kafka maps keys to partitions using hash(key) % partition_count. If you increase the count from 10 to 20, the mapping changes. An OrderUpdated event might end up on Partition 14, while the original OrderPlaced was on Partition 2. This destroys the temporal guarantees for that specific Order ID.

The rule of thumb: overestimate your partition count. Start with at least 12 or 24 partitions even for low-traffic topics. This gives you room to scale your Spring Boot pods horizontally without re-partitioning and breaking your ordering logic.

Observability and the p99 Latency Shift

Debugging a synchronous system is easy: look at the stack trace. Debugging an event-driven system requires distributed tracing (OpenTelemetry/Zipkin). You must propagate the traceId through Kafka headers. Spring Cloud Sleuth (now Micrometer Tracing) does this automatically by injecting headers into the ProducerRecord.

In one production migration, we observed our p99 latency drop from 480ms to 90ms. This wasn't because the code was faster, but because we stopped waiting for downstream dependencies. The OrderService returned 202 Accepted almost instantly. The "perceived" latency for the user was gone, but the "system" latency—the time until the inventory was actually updated—remained the same.

This shift requires a change in how you monitor health. You no longer care about the response time of the POST /order endpoint as much as you care about Consumer Lag. If the lag on orders.v1 is increasing, your system is falling behind, even if your API responses are lightning-fast. Monitor the records-lag-max metric in Prometheus. If it exceeds a threshold (e.g., 10,000 messages), trigger an alert to scale your consumer deployment.

The sharp takeaway for any architect is this: Event-driven systems are not about speed; they are about resilience and autonomy. By decoupling services through a durable log, you ensure that a failure in a secondary system like "Email Notifications" cannot prevent a primary system like "Checkout" from doing its job. You pay for this in complexity, but for any system requiring high availability and scale, the trade-off is mandatory.