Kubernetes Basics for Java Developers

The Wednesday Afternoon Cascade

At 14:12 UTC, the orders-service—a standard Spring Boot 3.2 application running on JDK 21—started throwing 503 Service Unavailable errors. It wasn’t a slow degradation; it was a cliff. By 14:15, the p99 latency for the checkout flow spiked from 180ms to 25s, eventually hitting the gateway timeout.

The trigger appeared to be a routine horizontal scale event. Kubernetes observed CPU utilization exceeding 70% and attempted to spin up four new replicas. Instead of absorbing the load, the new Pods entered a CrashLoopBackOff cycle, while the existing "healthy" Pods began failing liveness checks.

When Java developers treat Kubernetes as a "black box" deployment target, they ignore the contract between the JVM and the Container Runtime Interface (CRI). This incident was a classic breakdown of that contract, specifically regarding how Spring Boot interacts with the Kubernetes Control Plane during the startup and shutdown lifecycles.

Anatomy of a Failed Scheduling Event

Standard Java deployments often fail because dev teams treat the YAML as "boilerplate" rather than infrastructure code. In this case, the Deployment manifest was missing critical resource coordination.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-service
spec:
  replicas: 4
  template:
    spec:
      containers:
      - name: orders-service
        image: internal-registry.io/orders:v2.1.4
        ports:
        - containerPort: 8080
        # The omission of resources and probes caused the cascade
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-creds
              key: password

The sequence of failure was predictable: 1. CPU Throttling: The JVM didn't have resources.limits.cpu defined. On a 64-core node, the JVM saw 64 available processors and sized its ForkJoinPool and Garbage Collector threads accordingly. However, the Kubelet enforced a CFS (Completely Fair Scheduler) quota, starving the process. 2. The OOM Killer: Without resources.limits.memory, the JVM default MaxRAMPercentage (usually 25% on modern versions) calculated a heap based on total node RAM (128GB) rather than the intended partition. The kernel killed the process once it breached the node's actual available slice. 3. Deadlock on Startup: Because no startupProbe was defined, Kubernetes started hitting the livenessProbe while the Spring context was still refreshing. Kubernetes restarted the "unhealthy" container, creating a loop of half-initialized JVMs.

Signal Flow and the Service Mesh

To understand why the traffic didn't reroute, we have to map the networking primitive. In Kubernetes, a Service is not a process; it is an abstraction over iptables or IPVS rules.

[ Internet ] 
     |
[ Ingress Controller (Nginx/Envoy) ] 
     | 
     |-- [ Service: orders-service (ClusterIP) ]
               |
               |--> [ Pod 1: 10.42.1.12 ] (Terminating)
               |--> [ Pod 2: 10.42.2.45 ] (Running - Overloaded)
               |--> [ Pod 3: 10.42.1.13 ] (CrashLoopBackOff)

The Service selects Pods based on the readinessProbe. Our developer team had pointed the readiness probe to /actuator/health, which returned UP as soon as the web server started, but *before* the application had warmed up its connection pool to the PostgreSQL database.

When the new Pods reported "Ready", the Service immediately began routing 25% of the traffic to them. The first request triggered the lazy initialization of the HikariCP pool, which timed out under the surge of concurrent requests. This caused the readiness probe to fail, removing the Pod from the Service rotation, and putting 100% of the load back onto the remaining overwhelmed Pods.

Root Cause: The Resource-Signal Mismatch

We identified two primary root causes that are ubiquitous in Java-on-K8s environments.

1. The CFS Quota vs. JVM Threading By default, the JVM is container-aware, but it only respects the limits you provide. If you set `cpu: "2"`, the JVM sees two processors. But if you provide no limit, and use a high-concurrency framework like Netty (inside Spring Boot's default Tomcat or WebFlux), it spawns threads based on the host core count.

We found that the orders-service was spawning 128 threads for the common pool on a node with 64 physical cores, even though we only wanted it to use 4 cores worth of time. The context switching overhead alone increased p99 latency from 180ms to 900ms before we even hit peak load.

2. Missing Graceful Shutdown The SIGTERM signal sent by Kubernetes to stop a Pod was being ignored. Java processes wrapped in a shell script (e.g., `ENTRYPOINT ["sh", "-c", "java -jar app.jar"]`) do not receive signals sent to the shell.

When Kubernetes tried to roll out the fix, it sent a SIGTERM. The shell died, but the JVM kept running until Kubernetes sent a SIGKILL 30 seconds later. In those 30 seconds, the Pod was still "Running" but the underlying database connections were being severed by the network overlay, leading to thousands of ConnectionResetException errors for users mid-checkout.

Remediating the Manifest

To fix this, we moved away from "bare" YAML and implemented a strict template for all Java microservices. The core change was implementing a tiered probing strategy and explicit memory management.

spec:
  containers:
  - name: orders-service
    image: internal-registry.io/orders:v2.1.5
    resources:
      requests:
        memory: "1Gi"
        cpu: "1000m"
      limits:
        memory: "1Gi"  # Keep requests == limits to prevent swapping/eviction
        cpu: "2000m"
    env:
    - name: JAVA_OPTS
      value: "-XX:MaxRAMPercentage=75.0 -XX:ActiveProcessorCount=2"
    ports:
    - containerPort: 8080
    startupProbe:
      httpGet:
        path: /actuator/health/liveness
        port: 8080
      failureThreshold: 30
      periodSeconds: 10 # Gives the JVM 300s to warm up before killing it
    livenessProbe:
      httpGet:
        path: /actuator/health/liveness
        port: 8080
    readinessProbe:
      httpGet:
        path: /actuator/health/readiness
        port: 8080

Notice the use of ActiveProcessorCount. Overriding the JVM's detection ensures that the JIT compiler and GC threads don't over-subscribe the allocated CPU quota, which prevents the CFS throttling that kills Java performance in multi-tenant clusters.

ConfigMaps and the Spring Refresh Problem

Another contributing factor was the hard-coded database URL. During the incident, we tried to point the service to a read-replica to alleviate load. However, the service was reading its configuration from a ConfigMap mounted as an environment variable.

In Kubernetes, if you update a ConfigMap, environment variables in a running Pod *do not update*. You must restart the Pod. We lost 10 minutes manually killing Pods to force them to pick up the new connection string.

The superior way to handle this for Java developers is mounting the ConfigMap as a volume. Spring Cloud Kubernetes can watch the file system for changes to application.yaml and trigger a context refresh without a Pod restart. This allows for dynamic log-level changes or circuit-breaker threshold adjustments in real-time.

The Shutdown Contract

The final piece of the remediation involved the preStop hook. When a Pod is marked for termination: 1. It is removed from the Service's Endpoints list. 2. The preStop hook executes. 3. The SIGTERM is sent.

If you don't wait between step 1 and step 3, there's a race condition. The Ingress controller might still have the Pod's IP in its local cache and send one last request just as the JVM is shutting down.

We added a 15-second sleep to the preStop hook. This feels "hacky" to some, but it is the industry standard for ensuring zero-downtime Java deployments. It gives the network layer enough time to propagate the "this Pod is leaving" message before the JVM stops listening on 8080.

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 15"]

Hard-Won Lessons in JVM Orchestration

The most significant metric we tracked after these changes wasn't just uptime; it was the stability of the JIT compiler. Before the fix, the JIT compiler was constantly interrupted by CPU throttling, leading to erratic performance "waves." After setting explicit ActiveProcessorCount and matching it to our Kubernetes CPU limits, our p99 dropped from 480ms to 90ms under load, simply because the JVM was no longer fighting the orchestrator for resources.

If you are a Java developer, your responsibility doesn't end at the JAR file. You must understand how the kube-scheduler decides where your app lives and how the kube-proxy directs traffic to it. The JVM is a magnificent, greedy beast; if you don't build a cage of YAML around it, it will consume your entire cluster and ask for more.

The takeaway is simple: Kubernetes is a distributed operating system. You wouldn't write production Java without tuning your OS threads and memory; don't try to run it on Kubernetes without defining the lifecycle hooks and resource boundaries that the platform expects. The cost of ignoring this is a 2:00 AM page that could have been a non-event.