Scaling Java Microservices on AWS EKS with Terraform and Horizontal Pod Autoscaling

Starvation through JVM Heap Inefficiency In the context of AWS EKS, the most common "adversary" isn't a hacker, but the Java Virtual Machine's own memory management strategy. In a default configuration, a Spring Boot application will attempt to claim a percentage of the host's physical RAM. If your EKS worker nodes have 32GB of RAM but your pod limit is 2GB, an unprofiled JVM might attempt to allocate 8GB (25% of host memory) for its heap, leading to an immediate `OOMKilled` status from the Kubernetes kubelet.

To defeat resource starvation, you must synchronize the JVM's internal world-view with the cgroup limits defined in your Terraform-managed manifests. Using -XX:MaxRAMPercentage=75.0 is the standard mitigation, but it is insufficient without strict HPA thresholds. If your HPA triggers at 80% CPU but your JVM spends 40% of its cycles in Garbage Collection (GC) due to memory pressure, the scaler will add more pods that immediately enter a GC loop, creating a "death spiral" where cost increases but throughput remains flat.

```hcl
# terraform/modules/microservice/main.tf
resource "kubernetes_horizontal_pod_autoscaler_v2" "spring_app_hpa" {
  metadata {
    name      = var.service_name
    namespace = var.namespace
  }

spec { max_replicas = 20 min_replicas = 3

scale_target_ref { api_version = "apps/v1" kind = "Deployment" name = var.service_name }

metric { type = "resource" resource { name = "cpu" target { type = "Utilization" average_utilization = 65 } } } } } ```

The specific mitigation here is setting average_utilization to 65% for Java workloads. While Go or Node.js apps can comfortably scale at 80%, a Spring Boot app needs "headroom" for the JIT compiler and GC cycles during a scaling event. If you push Java pods to 80% before scaling, the overhead of the new JVM starting up (which is CPU-intensive) often starves the existing pods of the resources they need to handle the current traffic surge.

Cascading Failure via "Cold Start" Thundering Herds When a traffic spike occurs, the HPA triggers the creation of new pods. In a Java environment, a large Spring Boot application can take 30 to 60 seconds to reach a "ready" state. During this period, the pod is consuming CPU for class loading and JIT compilation but isn't yet accepting traffic. If your liveness and readiness probes are misconfigured, the Kubernetes Service may route traffic to a pod that is technically "up" but computationally bogged down by its own startup routine.

This results in a thundering herd where the newly spawned pods fail to alleviate the load, causing the existing pods to crash under the pressure, which in turn triggers more HPA events. We defeat this by implementing startupProbes that are decoupled from readinessProbes.

# k8s/deployment.yaml excerpt
spec:
  containers:
  - name: spring-svc
    image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/api:v1.2.0
    ports:
    - containerPort: 8080
    startupProbe:
      httpGet:
        path: /actuator/health/liveness
        port: 8080
      failureThreshold: 30
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /actuator/health/readiness
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5

By using a startupProbe with a high failureThreshold (30 attempts * 10 seconds = 5 minutes), you allow the JVM the necessary time to warm up without the livenessProbe prematurely killing the container. This prevents the "reboot loop" that occurs when a container doesn't start fast enough for a strict liveness check.

Node Over-Provisioning and Cost Exfiltration An improperly configured EKS cluster with HPA can become a financial liability. If your Terraform code defines `m5.large` instances but your HPA is scaling out pods that only require 0.5 CPU and 1GB RAM, you will end up with fragmented nodes where 50% of the billable hardware is sitting idle. This is "cost exfiltration"—where your budget escapes through the gaps in your scheduling logic.

To defeat this, we use the Karpenter provider instead of the legacy Cluster Autoscaler. Karpenter allows for "bin-packing" and "under-provisioning protection." It looks at the pending pods and provisions the exact instance type needed, rather than blindly expanding a pre-defined Managed Node Group.

EKS SCALING TOPOLOGY
[ Traffic Spike ] -> [ HPA ] -> [ Deployment: Replicas 5 -> 15 ]
                                    |
                                    v
[ Pending Pods ] <--- [ Karpenter Controller ]
      |                         |
      |          /--------------+--------------\
      v          v                             v
[ Node A ]  [ Node B ]                  [ New Node C ] (Provisioned JIT)
(m5.large)  (m5.large)                  (c5.xlarge - optimized for Java)
[ Pod 1-4 ] [ Pod 5-8 ]                 [ Pod 9-15 ]

A hard-won lesson from production: we shifted from generic m5.large nodes to a mix of c6i (compute-optimized) for our Java services. By switching to Karpenter and allowing it to select c6i instances, our p99 latency dropped from 480ms to 90ms during peak scaling events. Why? Because the newer Intel/Graviton processors handled the concurrent JIT compilation of multiple starting pods significantly better than the general-purpose instances, which suffered from "steal time" under heavy context switching.

Credential Leakage through IRSA Misconfiguration A common vulnerability in EKS environments is the use of broad IAM policies attached to the worker nodes themselves. If a Java microservice is compromised (e.g., via a Log4j-style vulnerability), the adversary can query the EC2 Instance Metadata Service (IMDS) at `http://169.254.169.254/latest/meta-data/iam/security-credentials/` to steal the node's role. If that node role has S3 or DynamoDB permissions, the attacker has access to everything.

We defeat this by implementing IAM Roles for Service Accounts (IRSA) via Terraform. This ensures that the Java pod only has the specific permissions it needs, and we further mitigate the threat by enforcing IMDSv2 with a hop limit of 1.

```hcl
# terraform/iam_irsa.tf
module "iam_eks_role" {
  source    = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  role_name = "${var.service_name}-irsa"

role_policy_arns = { policy = aws_iam_policy.microservice_s3_access.arn }

oidc_providers = { main = { provider_arn = data.aws_iam_openid_connect_provider.eks.arn namespace_service_accounts = ["${var.namespace}:${var.service_name}"] } } }

resource "kubernetes_service_account" "java_sa" { metadata { name = var.service_name namespace = var.namespace annotations = { "eks.amazonaws.com/role-arn" = module.iam_eks_role.iam_role_arn } } } ```

By pinning the IAM role to the Kubernetes Service Account, the Java process receives a temporary OIDC token mounted at /var/run/secrets/eks.amazonaws.com/serviceaccount/token. The AWS SDKs (S3, DynamoDB, etc.) are smart enough to look for this token before trying the IMDS, effectively isolating the pod's identity from the underlying host.

Metric Poisoning and HPA Flapping Adversaries or even misbehaving external clients can trigger an HPA flap by hitting expensive endpoints (like `/api/v1/export-pdf`) in a rhythmic pattern. If the HPA's `scaleDown` behavior is too aggressive, the cluster will enter a state of "flapping"—where pods are constantly being created and destroyed. This is disastrous for Java, as the high cost of startup (CPU/time) means the pod is deleted just as it finishes warming up.

To defeat HPA flapping, we must tune the behavior section of the horizontal_pod_autoscaler_v2. We want to "scale up fast, scale down slow."

# terraform/modules/microservice/hpa.tf (Extended)
spec {
  behavior {
    scale_down {
      stabilization_window_seconds = 300 # Wait 5 minutes before cooling down
      policy {
        type           = "Percent"
        value          = 10
        period_seconds = 60
      }
    }
    scale_up {
      stabilization_window_seconds = 0 # Scale up immediately
      policy {
        type           = "Pods"
        value          = 4
        period_seconds = 15
      }
    }
  }
}

By setting a stabilization_window_seconds of 300 for scale_down, we ensure that the pods remain active for at least 5 minutes after a peak. This "burns" some extra cash in the short term but prevents the massive CPU overhead of constantly re-initializing JVMs, which actually saves cost in the long run by maintaining a stable, warmed-up fleet.

Observability Blindness via Metric Lag A major threat to scaling is relying on metrics that are too old. By default, the Kubernetes Metrics Server scrapes every 60 seconds. In a high-traffic Java microservice, a 60-second lag can be the difference between a successful scale-out and a total outage. If the HPA is making decisions based on 1-minute-old data, it will always be "fighting the last war."

We defeat metric lag by moving away from the default CPU-based HPA and introducing the Prometheus Adapter or AWS Load Balancer Controller metrics (via the External Metrics API). Instead of scaling on CPU, we scale on Request Count Per Target (RPCPT).

# terraform/external_metrics.tf
resource "kubernetes_horizontal_pod_autoscaler_v2" "custom_metrics_hpa" {
  # ... metadata ...
  spec {
    metric {
      type = "external"
      external {
        metric {
          name = "alb_request_count_per_target"
          selector {
            match_labels = {
              "ingress" = "api-ingress"
            }
          }
        }
        target {
          type  = "AverageValue"
          value = "100" # Scale when a pod exceeds 100 requests/sec
        }
      }
    }
  }
}

Scaling on network requests is a "leading indicator" for Java services. CPU utilization is a "lagging indicator"—by the time the CPU hits 70%, the thread pool is likely already saturated and the application's latency is climbing. By scaling on raw request volume, you can provision capacity *before* the JVM begins to struggle with thread context switching and GC pauses.

The sharp takeaway for operating Java on EKS: your infrastructure must ignore the JVM's "internal" health and focus on the external indicators of throughput. A JVM is most efficient when it is hot and heavily utilized; your scaling policy must reflect this by providing a long stabilization tail and aggressive, request-based lead times. If you treat a Java pod like a lightweight Go binary, your cluster will spend more time class-loading than serving traffic.