Kafka & ZooKeeper Docker Setup — Quick Deploy Guide

The Composite Local Stack

Local development with Apache Kafka is a rite of passage that usually starts with a docker-compose.yml and ends with a ConnectionRefusedException. The difficulty lies in the networking bridge between the Docker daemon's internal network and your host machine where the Spring Boot application or CLI consumer actually lives.

We use the Confluent distribution image because it packages the necessary helper scripts to manage the KAFKA_ADVERTISED_LISTENERS environment variable, which is the single most important configuration in the stack.

```yaml
version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

kafka: image: confluentinc/cp-kafka:7.5.0 depends_on: - zookeeper ports: - "9092:9092" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 ```

The split between 29092 and 9092 is non-negotiable. If you try to use localhost:9092 inside the Docker network for inter-broker communication, the broker will attempt to connect to itself via the loopback interface of the container, while your host-based Spring Boot app will see localhost:9092 as the actual Docker port. This setup ensures that both internal Docker services and external host processes can resolve the broker metadata correctly.

Topology of the Local Cluster

+---------------------+          +--------------------------------+
|    Host Machine     |          |       Docker Desktop/VM        |
| (Spring Boot / CLI) |          |                                |
|                     |  :9092   |  +--------------------------+  |
|  [Producer/Cons] ---|----------|->|      Kafka Broker        |  |
|                     |          |  +------------^-------------+  |
|                     |          |               |                |
|                     |          |               | :2181          |
|                     |          |  +------------v-------------+  |
|                     |          |  |       ZooKeeper          |  |
|                     |          |  +--------------------------+  |
+---------------------+          +--------------------------------+

Failure Mode: The Fatal Metadata Ghost

Symptom: Your application logs show Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. or, more frustratingly, the app hangs indefinitely during the first producer.send() call.

Root Cause: This is almost always a listener mismatch. When a Kafka client (your Spring app) first connects to localhost:9092, it asks the broker for metadata: "Who are the leaders for topic X?" The broker responds with its own address. If the broker is configured with KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092, it tells your Spring app: "I am at kafka:29092." Since your host machine doesn't know what kafka is (it's a Docker-internal DNS name), the client fails to connect to the second hop.

Fix: Ensure your KAFKA_ADVERTISED_LISTENERS contains the PLAINTEXT_HOST://localhost:9092 entry. This forces the broker to broadcast localhost as its address to any client connecting via the 9092 port. Use kcat -b localhost:9092 -L to verify. The output should explicitly list the broker's hostname as localhost.

Failure Mode: Zombie Consumer Group Lock

Symptom: You restart your Spring Boot application, but it refuses to process any new messages. Looking at the logs, you see Generation 265... Preparing to rebalance group. Meanwhile, the consumer lag on your topic starts climbing rapidly.

Root Cause: In a local Docker environment, we often hard-kill containers or CTRL+C our IDEs. If the session.timeout.ms (defaulting to 45 seconds in many client libs) hasn't expired, Kafka thinks your previous process is still alive. The new instance tries to join the group, triggering a rebalance. If the old instance didn't send a LeaveGroup request, Kafka waits for the full timeout before kicking it out and allowing the new instance to take over the partitions.

Fix: For local development, slash your timeouts in your application.yml: ``yaml spring: kafka: consumer: properties: session.timeout.ms: 6000 heartbeat.interval.ms: 2000 max.poll.interval.ms: 300000 ` This narrows the "zombie window" from 45 seconds to 6 seconds. You can also force a faster rebalance by setting KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0` in your Docker environment variables, which skips the coordinator's wait time for other members to join during the first rebalance.

Failure Mode: The ISR Shrinkage Death Spiral

Symptom: You successfully produce a message, but you get a NotEnoughReplicasException or NotEnoughReplicasAfterAppendException.

Root Cause: This happens when you try to run a "production-like" configuration on a single-node local cluster. If your Spring Boot app specifies acks: all and your topic was created (or the broker is configured) with min.insync.replicas: 2, the write will fail. Kafka cannot guarantee that the message has been replicated because there is only one broker in your docker-compose.yml.

Fix: On single-node dev stacks, you must override the replication defaults. ``yaml # In docker-compose.yml KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 ` If you encounter this on an existing topic, you must manually alter the topic configuration via the container shell: docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --alter --add-config min.insync.replicas=1`

Failure Mode: The ZooKeeper Ephemeral Node Leak

Symptom: The Kafka container starts, but immediately exits with Fatal error during KafkaServer startup. Prepare to shutdown. Digging deeper, the logs show ZkException: org.apache.zookeeper.KeeperException$NodeExistsException: /brokers/ids/1.

Root Cause: You are likely using Docker volumes to persist Kafka data, but you deleted the Kafka container while ZooKeeper kept its state, or vice versa. ZooKeeper still believes Broker ID 1 is active because the session hasn't timed out. When the Kafka container restarts, it tries to register as ID 1 and fails because the node already exists in ZooKeeper. This is common when you pause a laptop (sleep mode) which freezes the Docker VM clock, causing ZooKeeper heartbeats to fail in weird, non-linear ways.

Fix: Wipe the ZooKeeper data volume or restart the ZooKeeper container entirely. In local dev, I strongly advise *against* mounting host volumes for /var/lib/zookeeper/data and /var/lib/kafka/data unless you are specifically testing persistence. Keeping them in-container (volatile) makes docker-compose down -v a reliable "reset to zero" button.

Failure Mode: Large Message Buffer Overflow

Symptom: You attempt to send a 2MB JSON payload, and the producer throws RecordTooLargeException. You "fix" it by increasing the producer's max.request.size, but then the broker logs show MessageSizeTooLargeException and rejects the write. You fix *that* on the broker, and then the consumer starts throwing RecordTooLargeException.

Root Cause: This is the "Trifecta of Max Bytes." Kafka has three distinct gates that a large message must pass through, and they are usually inconsistent in local dev images. 1. max.request.size (Producer side) 2. message.max.bytes (Broker side) 3. max.partition.fetch.bytes (Consumer side)

Fix: You must sync all three. If you're pushing large batches or big blobs: - Broker: Set KAFKA_MESSAGE_MAX_BYTES: 10485760 (10MB) - Producer: properties["max.request.size"] = 10485760 - Consumer: properties["max.partition.fetch.bytes"] = 10485760 Note: Avoid large messages in Kafka whenever possible. I once saw a team's p99 latency drop from 480ms to 90ms simply by moving 5MB image payloads to S3 and only passing the URI through Kafka. Kafka's memory-mapped files and zero-copy mechanism are optimized for small, sequential segments, not huge chunks of binary data.

Failure Mode: The Silent Poison Pill

Symptom: Your Spring Boot consumer is stuck. No errors are being thrown, no CPU is being used, but it's just not processing. You see a message in the logs about DeserializationException.

Root Cause: A "poison pill" is a message that was successfully produced to a topic but cannot be deserialized by the consumer (e.g., a producer sent a string where the consumer expected an Avro record or JSON). By default, Spring's DefaultErrorHandler will attempt to retrying the same failing partition/offset indefinitely, effectively halting the consumer.

Fix: Use a ErrorHandlingDeserializer and a DeadLetterPublishingRecoverer. This allows the consumer to skip the bad message, send its headers and payload to a .DLT topic for later inspection, and move the offset forward to the next valid message. ``java @Bean public ConcurrentKafkaListenerContainerFactory<String, Object> kafkaListenerContainerFactory( ConsumerFactory<String, Object> consumerFactory) { ConcurrentKafkaListenerContainerFactory<String, Object> factory = new ConcurrentKafkaListenerContainerFactory<>(); factory.setConsumerFactory(consumerFactory); factory.setCommonErrorHandler(new DefaultErrorHandler( new DeadLetterPublishingRecoverer(kafkaTemplate), new FixedBackOff(1000L, 3L) // 3 retries, 1s apart )); return factory; } ``

The Single-Node Performance Trap

In local Docker setups, engineers often assume that what they see reflects production behavior. It doesn't. A single-node Kafka setup running on a shared Docker bridge network has zero network hop latency between partitions and zero replication overhead.

When moving from this Docker setup to a real 3-node cluster, you will likely see a significant spike in produce latency. This is why you should always test your Spring Boot batch.size and linger.ms settings. On a local machine, linger.ms=0 feels fast. In production, setting linger.ms=5 or linger.ms=10 can increase throughput by 5-10x by allowing the producer to buffer more records into a single TCP request, counteracting the network round-trip time between brokers that doesn't exist in your local Docker container.

The specific takeaway: Your local Kafka-in-Docker is a functional testbed, not a performance simulator. If your app requires sub-10ms end-to-end latency locally, expect 30ms-50ms in a real distributed environment. Configure your local docker-compose to be volatile and disposable; once you stop trying to persist state across reboots, 90% of your "Kafka is broken" issues will vanish.