Build Real-Time Apps with Spring Boot and Kafka: A Masterclass

Introduction

REST is fine when one service asks another a question. But for real-time apps — order pipelines, notifications, live dashboards, fraud detection — you need a stream of events that any number of services can subscribe to. That's what Kafka was built for.

This masterclass walks through a complete event-driven app in Spring Boot: order events flow from a REST endpoint through Kafka to multiple consumers (inventory, notifications, analytics) in real time.

Who this is for

Spring Boot developers building their first event-driven system, and anyone who has hit the limits of synchronous REST between microservices.

The architecture

   POST /orders ─► order-service ──► [orders topic, 6 partitions] ──►
                                                  │
                          ┌───────────────────────┼───────────────────────┐
                          ▼                       ▼                       ▼
                   inventory-service     notification-service       analytics-service
                   (consumer group A)    (consumer group B)         (consumer group C)

Each consumer group reads the same stream independently. Adding a new consumer doesn't touch the producer.

Step 1 — Local Kafka in 30 seconds

services:
  kafka:
    image: bitnami/kafka:3.7
    ports: ["9092:9092"]
    environment:
      KAFKA_CFG_NODE_ID: 1
      KAFKA_CFG_PROCESS_ROLES: controller,broker
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER

docker-compose up -d and you have a single-node Kafka cluster in KRaft mode — no ZooKeeper.

Step 2 — Add Spring Kafka

<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka</artifactId>
</dependency>

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all
      properties.enable.idempotence: true
    consumer:
      group-id: inventory
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties.spring.json.trusted.packages: "com.masterlab.*"

Step 3 — Producer: publish an event

```java
public record OrderPlaced(String orderId, String userId, BigDecimal total, Instant at) {}

@Service @RequiredArgsConstructor public class OrderProducer { private final KafkaTemplate<String, OrderPlaced> kafka;

public void publish(OrderPlaced evt) { kafka.send("orders", evt.orderId(), evt); } } ```

Step 4 — Consumer: react in real time

```java
@Component
public class InventoryListener {

@KafkaListener(topics = "orders", groupId = "inventory") public void onOrder(OrderPlaced evt, Acknowledgment ack) { inventoryService.reserve(evt.orderId()); ack.acknowledge(); } } ```

Enable manual acks for at-least-once delivery:

spring.kafka.listener.ack-mode: manual

Partitions, keys and ordering — the rule you must memorize

Kafka guarantees order per partition, not per topic. Two events with the same key always go to the same partition, so:

• Key by orderId → all events for one order are processed in order.
• Key by userId → all events for one user are processed in order.
• No key → round-robin, no ordering.

Pick the partition key that matches your *business* ordering requirement, not whatever feels natural.

Consumer groups and scaling

Add more pods of inventory-service and Kafka rebalances partitions across them. With 6 partitions:

• 1 pod → 1 consumer handles 6 partitions
• 3 pods → 3 consumers, 2 partitions each
• 6 pods → 6 consumers, 1 each (max parallelism)
• 7 pods → 6 consumers active, 1 idle (you can't beat the partition count)

Always provision partitions for your peak parallelism, not today's load.

Exactly-once with the outbox pattern

"At-least-once + idempotent consumer" works for most cases. For strict exactly-once across DB + Kafka, use the outbox pattern:

1. In the same DB transaction as your business change, insert a row into an outbox table. 2. A separate poller reads new rows and publishes to Kafka. 3. Mark the row as sent.

This avoids the dual-write problem entirely.

Common errors and fixes

• SerializationException: Unknown magic byte — producer and consumer disagree on the serializer. Standardize on JSON or Avro and configure both sides.
• Consumer reads the same message forever — your listener throws before ack.acknowledge(). Either fix the bug or send to a dead-letter topic after N retries.
• Rebalance storms — max.poll.interval.ms is too short and slow processing makes Kafka think the consumer is dead. Either speed up processing or raise the interval.
• Lag grows under load — not enough partitions. Adding partitions breaks key→partition mapping; do it in a maintenance window.

Source code

GitHub repo includes docker-compose.yml, three Spring Boot services and a load-test script that pushes 10k orders/sec on a laptop.

Related tutorials

• [Spring Boot + Kafka Tutorial](/blog/spring-boot-kafka-tutorial)
• [Kafka + ZooKeeper Docker Setup](/blog/kafka-zookeeper-docker-quick-setup)
• [Spring Boot Microservices Architecture](/blog/spring-boot-microservices-architecture-explained)

<!-- AFFILIATE-SECTIONS-V1 -->

Tools used in this tutorial

The setup below uses the following tools. Versions matter less than the role each one plays, so swap freely as long as you keep the responsibilities the same.

• Kafka
• Spring Boot
• Event-Driven
• Real-Time
• Java 17+
• Maven / Gradle
• Docker

Real-world production context

In production, a Spring Boot service almost never runs as a single JAR on a developer laptop. It is packaged as a container image, deployed on a managed platform, and fronted by a load balancer that handles TLS, health checks and autoscaling. Database connections go through a pool sized for the instance type; secrets are injected from a managed secret store rather than checked into Git; logs and metrics stream to a central backend so on-call engineers can answer "is the system healthy?" in seconds. This is why most teams pair Spring Boot with a cloud platform from day one — the JVM is great, but the operational surface around it is what keeps the service alive.

Recommended tools & deployment options

Once the tutorial works on your machine, the next question is *where do I run this for real?* These are the platforms most backend teams reach for, and the trade-offs between them:

• [DigitalOcean](https://www.digitalocean.com/) — the simplest path from a working container to a public URL. App Platform deploys directly from a Dockerfile, managed databases and Redis are one click away, and pricing is predictable. A common way to deploy the setup in this tutorial is using a cloud provider like DigitalOcean when you want to ship quickly without learning a full cloud SDK.
• [AWS](https://aws.amazon.com/) — the default for enterprise workloads. ECS Fargate or EKS run containers without you managing servers, RDS handles the database, and CloudWatch covers logs and metrics. In production environments, developers typically host this on AWS or a similar cloud platform when they need fine-grained IAM, multi-region failover, or deep integration with other AWS services.
• Docker — the packaging format every modern deploy target understands. Build once, run the same image locally, in CI and in production.
• Kubernetes (managed: EKS, DOKS, GKE) — the right choice once you have more than a handful of services, need rolling updates, autoscaling and policy-driven networking. For a single service it is usually overkill; for a microservices estate it quickly pays for itself.

A VPS or managed cloud service is required to run this architecture end-to-end — the local docker-compose setup is for development, not for serving traffic.

Next steps & related tutorials

Keep the momentum going with the next tutorial in this learning path:

• [Next: API Rate Limiting in Spring Boot with Bucket4j and Redis](/blog/api-rate-limiting-in-spring-boot)
• [Next: Building REST APIs with Spring Boot: A Complete Guide](/blog/building-rest-apis-with-spring-boot)
• [Next: Spring Boot + Kafka — Build a Real-Time Messaging System](/blog/spring-boot-kafka-tutorial)
• [Next: Spring Boot + Redis Caching — Make Your API 10× Faster](/blog/spring-boot-redis-caching)