Spring Boot + Kafka — Build a Real-Time Messaging System

Kafka is not a queue, and pretending otherwise will hurt you

Most Spring Boot + Kafka tutorials treat Kafka as a fancy queue: producer pushes, consumer pulls, done. That mental model gets you to "hello world" and then ruins your first production incident. Kafka is an *append-only log*. Consumers track their own position. Messages are not deleted on consumption. Every operational decision flows from that.

This article is the working knowledge I rely on: the producer and consumer config that I actually ship, the three things that bite everyone, and the boundaries between "Kafka can do this" and "you should let Kafka do this".

The mental model that matters

A topic is a partitioned, append-only log. Each partition is an ordered sequence of messages. A consumer group is a set of consumers that collectively read every partition exactly once — Kafka assigns partitions to consumers within the group.

Three implications:

• Ordering is per-partition. Messages within one partition are ordered; across partitions, there is no global order. If you need ordering by some key (e.g. per user), key your messages by that field so all messages for one user land in one partition.
• Throughput scales by adding partitions and consumers in the group, up to one consumer per partition. Beyond that, extra consumers sit idle.
• A consumer can re-read. Offsets are just numbers; rewind, replay, fast-forward.

The producer configuration I ship

spring:
  kafka:
    bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092
    producer:
      acks: all
      retries: 5
      properties:
        enable.idempotence: true
        max.in.flight.requests.per.connection: 5
        compression.type: zstd
        linger.ms: 10
        batch.size: 65536

Why each setting:

• acks: all — wait for replication to all in-sync replicas before considering the send successful. The alternative (acks: 1) loses data when the leader fails between accepting the write and replicating it. The throughput cost is small; the durability win is large.
• enable.idempotence: true — the producer assigns a sequence number to each message; the broker deduplicates retries. Without it, a network blip plus a retry can produce duplicates.
• linger.ms: 10 — wait up to 10ms to batch messages. The throughput improvement from batching is dramatic; the latency cost is small for most apps.
• compression.type: zstd — zstd compresses better than gzip and is faster than snappy. There is rarely a reason not to use it on modern brokers.

The consumer configuration I ship

spring:
  kafka:
    consumer:
      group-id: orders-service
      auto-offset-reset: earliest
      enable-auto-commit: false
      max-poll-records: 50
      properties:
        isolation.level: read_committed
    listener:
      ack-mode: manual
      concurrency: 3

Why:

• enable-auto-commit: false plus ack-mode: manual — you commit the offset only after the message is fully processed. Auto-commit happens on a timer regardless of processing success, which means a crash mid-processing causes message loss.
• max-poll-records: 50 — bounded batch size so one consumer cannot starve the others by holding the poll for too long.
• concurrency: 3 — three consumer threads per JVM. Match this to the number of partitions you want this instance to handle.

The listener that does not lose messages

@KafkaListener(topics = "order.placed", groupId = "billing-service")
public void onOrderPlaced(
    ConsumerRecord<String, OrderPlaced> record,
    Acknowledgment ack) {
  try {
    billing.invoice(record.value());
    ack.acknowledge();
  } catch (DuplicateInvoiceException dup) {
    // Already processed (we are idempotent). Safe to commit.
    ack.acknowledge();
  } catch (Exception e) {
    // Do not acknowledge. The message will be re-delivered.
    log.error("Failed to process {}", record.key(), e);
    throw e;
  }
}

Three deliberate choices:

• Acknowledge only on success. If processing throws, the offset is not advanced; the message will be re-delivered on the next poll.
• Treat duplicates as success. Re-delivery is normal; the handler must be idempotent. Without that, re-delivery causes double-spending.
• Throw on failure. Combined with a DefaultErrorHandler configured below, this routes poison messages to a dead-letter topic instead of looping forever.

The dead-letter topic — non-negotiable

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
  var recoverer = new DeadLetterPublishingRecoverer(template,
      (rec, ex) -> new TopicPartition(rec.topic() + ".dlt", -1));
  return new DefaultErrorHandler(recoverer,
      new ExponentialBackOffWithMaxRetries(3));
}

After three retries with exponential backoff, the message goes to <topic>.dlt. A separate consumer (or a human) inspects the DLT, decides why it failed, and either fixes the code or republishes the message.

Without a DLT, a single poison message blocks the partition forever. Every consumer in the group retries it on every poll. The consumer lag graph climbs vertically. The on-call gets paged.

Idempotency at the handler is the real durability story

acks: all plus enable.idempotence gives you "at least once" delivery. The handler being idempotent gives you the *effect* of exactly-once. The pattern:

@Transactional
public void invoice(OrderPlaced event) {
  if (invoiceRepo.existsByOrderId(event.orderId())) return;
  invoiceRepo.save(Invoice.from(event));
}

A unique constraint on order_id in the invoice table backstops the application-level check. Double-deliveries become a no-op instead of a duplicate.

This is the part that "exactly once with Kafka transactions" tries to solve at the protocol level. In practice, Kafka transactions are operationally heavy (transactional producers, read-committed isolation, exactly-once semantics setup) and idempotent handlers are a five-line change. Pick the simple option.

What Kafka is bad at

A few cases where reaching for Kafka is the wrong instinct:

• Request/response. Kafka is not RPC. Use HTTP or gRPC.
• A few messages per minute. The operational overhead is not justified. Use a real queue (SQS, RabbitMQ) or even a database table.
• Low-latency, narrow fan-out. A direct HTTP call beats Kafka for sub-100ms latency.
• Messages with strict per-message authorization. Kafka's auth model is per-topic, not per-message.

Kafka is the right answer when you have *high throughput, durable, fan-out* event streams. Pick a different tool for everything else.

What to monitor

Three numbers tell you whether your Kafka setup is healthy:

• Consumer lag per partition (kafka_consumer_lag). Should be near zero in steady state; rising lag means the consumer is falling behind.
• DLT message rate. Non-zero is fine if you have a process for it; rising is a code bug.
• Producer error rate. Should be effectively zero; non-zero means the broker is rejecting writes, usually a partition/leader issue.

Wire these to dashboards on day one. Kafka problems are silent until they are catastrophic; the lag graph is your early warning.

The honest summary

Kafka pays off when you commit to its model: durable log, idempotent handlers, manual commits, dead-letter topics, monitored lag. Adopted halfway, it gives you the operational complexity without the durability guarantees. The configuration above is what I ship by default — every setting has a reason, and removing any one of them re-introduces a class of bug I have already paid for.

What this guide consolidates

The 'real-time apps with Kafka masterclass' page has been merged into this canonical Kafka tutorial. The streaming and consumer-group material from the absorbed page now lives in the second half of this guide, so producer setup, consumer semantics and stream processing read as one continuous story.

Build Real-Time Apps with Spring Boot and Kafka: A Masterclass

The Illusion of Request-Response in High-Volume Streams

A common architectural trap for teams moving from REST to Kafka is treating the message broker like a distributed database or a synchronous pipe. The most visible symptom is the Synchronous Producer Blocking anti-pattern. Engineers often wrap kafkaTemplate.send() in a blocking .get() call or a .thenAccept() that waits for a metadata acknowledgement before proceeding with the HTTP request.

// DON'T DO THIS: Turning an asynchronous stream into a slow synchronous pipe
@PostMapping("/v1/events")
public ResponseEntity<String> sendEvent(@RequestBody Event payload) {
    try {
        // This blocks the Tomcat thread until the Kafka broker acks the write
        SendResult<String, Event> result = kafkaTemplate.send("telemetry-topic", payload).get(5, TimeUnit.SECONDS);
        return ResponseEntity.ok("Dispatched: " + result.getRecordMetadata().offset());
    } catch (Exception e) {
        return ResponseEntity.status(500).build();
    }
}

When your broker latency spikes—perhaps during a partition leader rebalance or a disk I/O burst on the Kafka nodes—this implementation causes thread pool exhaustion in your Spring Boot application. If your server.tomcat.threads.max is 200 and Kafka acknowledgement takes 1 second, your entire API goes down at just 200 requests per second.

The correct approach is to embrace Fire-and-Forget with Async Callbacks. You should leverage Kafka’s internal buffering and the CompletableFuture returned by KafkaTemplate to handle failures without holding the caller hostage.

// DO THIS: Non-blocking dispatch with decoupled error handling
@PostMapping("/v1/events")
public void sendEvent(@RequestBody Event payload) {
    kafkaTemplate.send("telemetry-topic", payload.getId(), payload)
        .whenComplete((result, ex) -> {
            if (ex != null) {
                log.error("Failed to append to log for ID: {}", payload.getId(), ex);
                // Handle via dead-letter-topic or local retry buffer
            } else {
                log.debug("Offset {} assigned to partition {}", 
                    result.getRecordMetadata().offset(),
                    result.getRecordMetadata().partition());
            }
        });
}

By decoupling the HTTP response from the Kafka acknowledgement, you allow the RecordAccumulator to batch records. In one high-throughput environment monitoring vehicle telematics, switching from synchronous .get() calls to asynchronous batching increased producer throughput by 14x while dropping p99 API latency from 480ms to 92ms.

The Partitioning Fallacy and Global Order Obsession

Newcomers often assume that Kafka guarantees total ordering across a topic. When they realize it only guarantees order within a partition, they frequently resort to the Single Partition Bottleneck. They set partitions: 1 to maintain sequence, effectively turning a distributed system into a single-threaded queue.

WRONG TOPOLOGY (Single Partition):
[Producer] -> [Topic: Orders (P0)] -> [Consumer A (Active)]
                                   -> [Consumer B (Idle - wasted scale)]
                                   -> [Consumer C (Idle - wasted scale)]

If you have 10 consumers but only 1 partition, 9 consumers will sit idle. The correct way to maintain strict ordering for business logic (e.g., ensuring OrderCreated is processed before OrderShipped) is Consistent Hashing by Business Key.

Configure your producer to use a specific key, usually an orderId or userId. Kafka’s default partitioner hashes this key to ensure all events for that ID land in the same partition.

# application.yml correct configuration
spring:
  kafka:
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all # Ensure data safety over raw speed
      properties:
        enable.idempotence: true
        max.in.flight.requests.per.connection: 5

With enable.idempotence: true, the producer attaches a sequence number to every batch, allowing the broker to detect and discard duplicates caused by network retries. This gives you exactly-once delivery semantics at the producer level without sacrificing the horizontal scalability of multiple partitions.

The Manual Acknowledgement and Hidden Rebalance Loops

A pervasive error in Spring Kafka consumers is the Over-Aggressive Manual Ack. Developers often disable auto-commit and try to manually acknowledge every single record in a loop, thinking it makes the system more "reliable."

// THE DANGEROUS PATTERN: Manual acks in a tight loop
@KafkaListener(topics = "transactions", groupId = "accounting-group")
public void listen(ConsumerRecord<String, Transaction> record, Acknowledgment ack) {
    try {
        process(record.value());
        ack.acknowledge(); // High frequency commits kill broker performance
    } catch (Exception e) {
        // What now? If you don't ack, the consumer might stall or loop
    }
}

Frequent commits (e.g., every 1ms) create immense pressure on the __consumer_offsets topic. Worse, if your process() logic takes longer than max.poll.interval.ms, Kafka assumes the consumer has died and triggers a rebalance. In a cluster with 50 partitions and 10 nodes, a rebalance "storm" can halt processing for 30+ seconds.

The production-grade solution involves Batch Listeners with Controlled Commits. Instead of processing one-by-one, process in chunks and let the container manage the offsets.

@KafkaListener(
    topics = "transactions", 
    groupId = "accounting-group",
    batch = "true",
    properties = {
        "max.poll.records=500",
        "fetch.min.bytes=1024"
    }
)
public void listen(List<ConsumerRecord<String, Transaction>> records) {
    // 1. Bulk DB insert (Atomic)
    repository.saveAll(records.stream().map(this::mapToEntity).toList());
    
    // 2. Spring Kafka will commit the highest offset in the batch automatically
    // after this method returns, provided AckMode is BATCH (the default).
}

By using batch = "true", you reduce the overhead of the consumer group protocol. If the service crashes midway through a batch, the next consumer will pick up from the last *successfully committed* batch. This guarantees at-least-once delivery.

Why Try-Catch in Listeners Is Not Error Handling

Most developers implement error handling by wrapping the listener body in a try-catch block and logging the error. This is a Poison Pill Anti-pattern. If a record is malformed (e.g., a JSON parsing error), your code will log it, "swallow" the error, and move to the next offset. You have now lost data silently.

// DON'T DO THIS: Silent failure
@KafkaListener(topics = "payments")
public void handle(Payment p) {
    try {
        validate(p);
    } catch (Exception e) {
        log.error("Failed to process payment"); // Data is gone forever
    }
}

In a real-time system, you need a Non-Blocking Retry and Dead Letter Topic (DLT) strategy. Spring Kafka provides a DefaultErrorHandler (or DefaultErrorHandler in 2.8+) that can move a message to a .DLT topic after $N$ failed retries, allowing the main consumer to keep moving while you investigate the failure.

@Bean
public DefaultErrorHandler errorHandler(KafkaOperations<Object, Object> template) {
    // Retry 3 times with a 1-second backoff
    FixedBackOff backOff = new FixedBackOff(1000L, 3);
    
    // Dead Letter Publishing Recoverer sends the failed message to "payments.DLT"
    DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
    
    return new DefaultErrorHandler(recoverer, backOff);
}

This configuration ensures that "poison pills"—messages that will never succeed due to schema mismatch or logic bugs—don't block the entire partition. One production cluster I managed had a partition blocked for 12 hours because a single malformed message caused a NullPointerException that wasn't handled, causing the consumer to restart, fetch the same message, and crash again in an infinite loop.

Transactional Integrity Across DB and Kafka

The most difficult challenge in Spring Boot Kafka apps is the Dual-Write Problem. You save an entity to PostgreSQL, and then you send an event to Kafka. If the database commit succeeds but the Kafka send fails (or the app crashes in between), your state is inconsistent.

// THE FRAGILE DUAL-WRITE
@Transactional
public void processOrder(Order o) {
    repo.save(o); // DB Commit
    kafkaTemplate.send("order-topic", o); // Might fail!
}

Using JTA or Two-Phase Commit (2PC) is a performance nightmare. Instead, use Kafka Transactions coordinated by Spring’s KafkaTransactionManager. This allows you to bind the Kafka producer’s transaction to the database transaction.

```java
@Bean
public KafkaTransactionManager<String, Object> kafkaTransactionManager(
        ProducerFactory<String, Object> pf) {
    return new KafkaTransactionManager<>(pf);
}

// In the service: @Transactional("kafkaTransactionManager") public void processOrder(Order o) { // Both or neither will commit repo.save(o); kafkaTemplate.send("order-topic", o); } ```

When this is enabled, Kafka marks the message as "uncommitted." Consumers configured with isolation.level: read_committed will not see the message until the transaction finishes. This solves the atomic consistency problem without the latency overhead of XA transactions.

The "Read-Your-Writes" Trap with Consumer Lag

If your architecture relies on a "Store in DB -> Send to Kafka -> Consumer updates Cache" flow, you will eventually hit the Temporal Inconsistency wall. Developers often expect the consumer to be nearly instantaneous. In reality, during traffic spikes, consumer lag might grow to several seconds.

If your UI redirects the user to a page that reads from the "Cache" immediately after the "Store in DB" call, the user will see stale data.

Hard-won lesson: Never use Kafka as a "synchronous" update mechanism for a UI that requires immediate consistency. If the user needs to see their change immediately, return the state directly from the API that handled the write. Use Kafka for downstream side effects (analytics, search indexing, secondary notifications) where eventual consistency (e.g., a 200ms delay) is acceptable.

The sharp takeaway for high-performance Kafka systems: Scale horizontally by increasing partition counts early, handle failures via DLTs rather than local try-catch blocks, and never trade the broker's natural asynchronicity for the false comfort of a blocking call. Real-time apps aren't just fast; they are resilient and decoupled.