CI/CD Pipeline with GitHub Actions and Docker

The 02:45 AM Container Registry Death Spiral

The incident began not with a failed test, but with a series of HTTP 429 Too Many Requests errors from our container registry. Within six minutes, our entire CI/CD pipeline—the lifeblood of our Spring Boot microservices ecosystem—ground to a halt. Developers couldn't merge, the staging environment began showing stale artifacts, and the auto-scaler couldn't pull the latest image to replace a failing pod in production.

We weren't victims of a DDoS attack. We were victims of our own naive GitHub Actions workflow that treated Docker layers like disposable toys.

When you scale a Spring Boot application across 40+ repositories, each triggering a multi-stage Docker build on every branch push, the overhead isn't just time; it's the sheer volume of redundant IO. Our legacy pipeline was pulling a 450MB base OpenJDK image 200 times a day and pushing 15-layer images where only the top 2MB actually changed. This incident forced us to rebuild our pipeline from the ground up, moving away from "it works on my machine" scripts toward a production-grade, layer-aware CI/CD architecture.

Timeline of the Collapse

* 02:40 UTC: A developer in the EMEA region pushes a batch of commits to a monorepo-adjacent service. This triggers 12 concurrent GitHub Actions runs. * 02:42 UTC: The "Push to Docker Hub" step begins failing across all active runs. * 02:43 UTC: Internal Prometheus alerts fire: kubelet_image_pull_failures_total spikes. Our EKS cluster can't scale out because it’s hitting the same registry rate limits. * 02:48 UTC: Total lockout. Even manual docker pull commands from local machines are rejected. * 03:05 UTC: Mitigation starts: We implement a global caching strategy and switch to an authenticated, tiered registry structure.

The Root Cause: Layer Bloat and Registry Abuse

The post-mortem revealed that our Dockerfile was a performance nightmare disguised as "simple." We were using a single-stage build that included the entire Gradle wrapper, source code, and build-time dependencies in the final image.

Every time a developer changed one line of Java code, the Docker engine invalidated the cache for the entire RUN ./gradlew build layer. This meant we were pushing 300MB of "new" layers for every tiny commit. Multiply that by 10 developers and 20 commits a day, and you're saturating your bandwidth and registry quotas for no reason.

LEGACY BUILD TOPOLOGY (The "Registry Killer")
+-------------------------------------------------------+
| GH Runner: Checkout -> Setup Java -> Build Jar        |
+-------------------------------------------------------+
            |
            v
+-------------------------------------------------------+
| Docker Build: COPY . . -> RUN ./gradlew build         |
| (Everything is re-run because source changed)         |
+-------------------------------------------------------+
            |
            v
+-------------------------------------------------------+
| Docker Push: 350MB of un-cached layers to Registry    |
+-------------------------------------------------------+

We realized we were ignoring the two most powerful features of GitHub Actions and Docker: type=gha caching and multi-stage builds.

Engineering the Remediation: Layer-Aware Pipelines

To fix this, we redesigned the workflow to be "layer-conservative." We shifted the heavy lifting of dependency management into the Docker build process itself, but we used GitHub’s internal cache as a backend for Docker’s buildx.

Here is the hardened .github/workflows/pipeline.yml that we implemented to resolve the incident. Pay attention to the cache-from and cache-to parameters—this is where the p99 build time dropped from 7 minutes to under 90 seconds.

```yaml
name: Production Grade Spring CI

on: push: branches: [ main ] paths: - 'src/**' - 'build.gradle' - 'Dockerfile'

jobs: build-and-push: runs-on: ubuntu-latest steps: - name: Checkout Code uses: actions/checkout@v4

- name: Set up JDK 17 uses: actions/setup-java@v3 with: java-version: '17' distribution: 'temurin' cache: 'gradle'

- name: Set up Docker Buildx uses: docker/setup-buildx-action@v3

- name: Login to GHCR uses: docker/login-action@v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }}

- name: Build and Push uses: docker/build-push-action@v5 with: context: . push: true tags: ghcr.io/${{ github.repository }}:latest,ghcr.io/${{ github.repository }}:${{ github.sha }} cache-from: type=gha cache-to: type=gha,mode=max ```

The "Secret Sauce" of `mode=max`

Most engineers use cache-to: type=gha, but they omit mode=max. By default, Docker only caches the layers of the final stage. In a multi-stage Spring Boot build, that’s useless because the heavy lifting happens in the "builder" stage. mode=max forces the exporter to save the cache for every single stage in the Dockerfile.

Refactoring the Dockerfile for Spring Boot Optimization

The GitHub Action is only half the battle. If the Dockerfile isn't structured to maximize layer reuse, no amount of YAML magic will save you. We moved away from the "fat jar" approach and embraced Spring Boot’s layer indexing.

Standard Spring Boot jars are essentially a zip of three things: your code (small), your dependencies (huge), and the loader (static). By exploding the JAR during the build, we can cache the dependencies/ layer separately from the application/ layer.

```dockerfile
# Stage 1: Dependency Extraction
FROM eclipse-temurin:17-jre-alpine as builder
WORKDIR application
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} application.jar
RUN java -Djarmode=layertools -jar application.jar extract

# Stage 2: Final Runtime Image FROM eclipse-temurin:17-jre-alpine WORKDIR application # These layers rarely change (p99 cache hit rate) COPY --from=builder application/dependencies/ ./ COPY --from=builder application/spring-boot-loader/ ./ COPY --from=builder application/snapshot-dependencies/ ./ # This layer changes on every commit COPY --from=builder application/application/ ./

ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"] ```

By using JarLauncher, we ensured that if only a controller changed, Docker would only need to upload the application/ layer—usually around 200KB. The 150MB of Hibernate, Spring Core, and Jackson libraries remained untouched in the registry.

Security Debt: Why Root Containers Failed Us

While fixing the pipeline, we found a secondary contributing factor to our production instability: the Docker images were running as root. During the incident, a developer had tried to hot-fix a permissions issue by shell-execing into a container and changing /app ownership.

A high-performance CI/CD pipeline must enforce security boundaries during the build. We updated the final stage of our Dockerfile to use a non-privileged user:

RUN addgroup -S spring && adduser -S spring -G spring
USER spring:spring

This prevents the container from having write access to system directories, reducing the attack surface if the Java process is compromised. It also prevents the "dirty overlay" problem where a running container can modify its own image layers, causing drift between what is in Git and what is running in the pod.

Contributing Factors to the Registry Outage

1. Tag Overwriting: We were pushing latest for every commit. This caused a massive number of "untagged" or "orphaned" blobs in our registry, which the registry's garbage collector struggled to process in real-time. 2. Pull-Through Caching: Our EKS nodes were pulling directly from the public internet without a local pull-through cache (like Harbor or AWS ECR Public Gallery). 3. Parallelism Overload: GitHub Actions allows 20+ concurrent builds by default. Without concurrency groups in our YAML, we were bombarding the registry with simultaneous uploads of the same base layers.

We added the following to our GitHub Action to ensure we don't waste resources on stale builds:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Hard-Won Lessons from the Frontline

The transition from a "scripts that run Docker" mindset to a "Docker-native CI/CD" mindset provided immediate, measurable results:

* Registry IO: We reduced outbound data transfer from 850GB/month to 42GB/month. * Build Speed: Cold start builds remained at ~5 minutes, but the 95th percentile (incremental changes) dropped from 440 seconds to 82 seconds. * Stability: We haven't hit a 429 Too Many Requests error in the 14 months since this change.

One specific number that shocked the team: we found that type=gha caching, while slightly slower than a local SSD cache, was still saving us roughly $400/month in compute time because the runners finished their jobs so much faster.

If you are running a Spring Boot application with GitHub Actions, the default behavior is your enemy. The framework is heavy, the JARs are fat, and the defaults are designed for "Hello World," not a high-velocity engineering team. By decoupling your layers and using a deep caching strategy (mode=max), you move the bottleneck from the network back to where it should be: your CPU.

The sharp takeaway here is that CI/CD is not just about automation; it is about resource efficiency. If your pipeline treats the registry as an infinite bucket, the registry will eventually treat your requests as an attack. Build for layer reuse, or prepare for the 3:00 AM page.