Circuit Breaker in Spring Boot with Resilience4j — Protect Your System from Overload

A circuit breaker is not a retry — and treating them as the same will hurt you

The first time I deployed Resilience4j I had circuit breakers, retries, timeouts, and bulkheads all wrapping the same outbound HTTP call. It looked thorough. Under load, the system behaved bizarrely — calls would retry, fail the breaker, then retry again because the retry was outside the breaker. I had built a self-amplifying failure mode.

Resilience patterns compose, but only if you understand the layer each one operates at. This article is the working mental model and the Spring Boot configuration that I actually ship.

What each pattern is for

• Timeout — bound the wall-clock time of a single call. Without a timeout, a hung downstream takes your threads with it.
• Retry — handle *transient* failures (network blip, brief 503). Cheap to do, dangerous if the failure is not actually transient.
• Circuit breaker — stop calling a downstream that is unambiguously broken. Fail fast so callers can degrade gracefully instead of piling up waiting.
• Bulkhead — cap concurrency to a downstream so it cannot consume all your threads.
• Rate limiter — protect *yourself or the downstream* from issuing too many requests per second.

The order matters: timeout is the foundation, the rest sit on top. Without a timeout, every other pattern is unreliable.

The composition that actually works

For an outbound call, the right order from outside in is:

Bulkhead -> CircuitBreaker -> Retry -> Timeout -> the actual call

Why this order:

• The bulkhead limits how many threads can enter the protected region at all.
• The breaker decides whether to try at all.
• The retry handles a single transient failure within the budget the breaker allows.
• The timeout bounds each individual attempt.

If you put retry *outside* the breaker, the retry will keep hammering even after the breaker has opened, which is the exact failure mode I shipped the first time.

The Spring Boot configuration

resilience4j:
  circuitbreaker:
    instances:
      payments:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 20
        failureRateThreshold: 50
        waitDurationInOpenState: 10s
        permittedNumberOfCallsInHalfOpenState: 5
        minimumNumberOfCalls: 10
  retry:
    instances:
      payments:
        maxAttempts: 3
        waitDuration: 200ms
        exponentialBackoffMultiplier: 2
        retryExceptions:
          - java.io.IOException
          - org.springframework.web.client.HttpServerErrorException
  timelimiter:
    instances:
      payments:
        timeoutDuration: 2s
        cancelRunningFuture: true

And the call site:

```java
@CircuitBreaker(name = "payments", fallbackMethod = "fallback")
@Retry(name = "payments")
@TimeLimiter(name = "payments")
public CompletableFuture<PaymentResponse> charge(ChargeCommand cmd) {
  return CompletableFuture.supplyAsync(() -> client.charge(cmd));
}

private CompletableFuture<PaymentResponse> fallback(ChargeCommand cmd, Throwable ex) { return CompletableFuture.completedFuture(PaymentResponse.deferred(cmd)); } ```

Why the specific numbers:

• minimumNumberOfCalls: 10 — the breaker will not open on the first failure. Without this, a single bad request can trip it.
• failureRateThreshold: 50 — sounds high. It is. A 30% threshold opens on minor blips; 50% is "the downstream is genuinely sick".
• waitDurationInOpenState: 10s — short enough that you recover quickly, long enough that you do not pound a recovering service.
• maxAttempts: 3 with exponential backoff — three attempts total, ~600ms of waiting in the worst case. Beyond that the user is waiting too long and the retry is providing negative value.
• timeoutDuration: 2s — sized to the downstream's p99 plus headroom. If p99 is 800ms and your timeout is 800ms, you will fail the slow tail of legitimate calls.

What goes in the fallback

The fallback method runs when the breaker is open or the retry budget exhausts. It must not call the same downstream — that defeats the entire purpose. Useful fallback strategies:

1. Return a cached previous result. Works for read-heavy paths. 2. Defer the action. For writes, persist a "pending" record and have a background job retry once the breaker closes. 3. Return a degraded response. "We can't show your recommendations right now" beats a 500. 4. Re-throw a typed exception. Sometimes the right answer is "tell the caller this is unavailable" and let them decide.

What the fallback must never do: silently swallow the error. The user gets a response; your monitoring still sees the underlying failure rate.

Retries on non-idempotent operations are bugs

If charge(cmd) is not safe to call twice, retrying it is wrong. The retry will turn an intermittent timeout into a duplicate charge. The fixes:

• Make the operation idempotent at the protocol level (Stripe's Idempotency-Key header).
• Only retry on errors that prove the operation did not start (connection refused, DNS failure). Do not retry on read timeouts unless you can prove idempotency.

The retryExceptions config in the example above lists IOException (connection-class failures) and HttpServerErrorException (5xx). It does *not* list HttpClientErrorException — retrying a 4xx is always wrong.

What to monitor

The Resilience4j Actuator integration exposes per-instance state. The three metrics that matter:

• resilience4j_circuitbreaker_state — gauge, 0 closed / 1 open / 2 half-open. Alert when this stays open for more than a minute.
• resilience4j_circuitbreaker_calls with kind=failed. The rate tells you whether the breaker is doing useful work.
• resilience4j_retry_calls with kind=successful_with_retry. A non-zero rate is fine; a rising rate means the downstream is getting flakier.

Without these dashboards, you have no idea whether the breaker config is right. With them, you can tune the thresholds against real traffic.

Bulkheads — when you actually need them

If you have one downstream that is slow and others that are fast, and they share a thread pool, the slow one will starve the fast ones. A bulkhead reserves a bounded concurrency for the slow one:

resilience4j:
  bulkhead:
    instances:
      payments:
        maxConcurrentCalls: 20
        maxWaitDuration: 50ms

Twenty concurrent calls into payments, anything beyond that fails fast. The rest of the application keeps its threads free for other work.

The non-obvious win: bulkheads make capacity planning legible. You can write down "payments can use at most 20 threads" and the system enforces it, instead of relying on hope.

The closing rule

Resilience patterns are not "best practices to bolt on". They are a coordinated set of decisions about *how each kind of failure should behave*. Pick the patterns that match your actual failure modes, configure them with numbers you can defend, and put dashboards on every one of them. The default Resilience4j config is reasonable; the configuration above is the one I would defend in a post-mortem.

What this guide consolidates

The separate 'unreachable services' explainer and the 'retry with Resilience4j' page both covered aspects of the same resilience story. They have been folded into this canonical Resilience4j guide so circuit breaker, retry, bulkhead and time-limiter patterns are explained together — the way they are actually combined in production.

Circuit Breaker: Handling Unreachable Services in Spring Boot

The Cost of the "Connection Refused" Death Spiral

When a downstream service becomes unreachable—not just slow, but physically down or network-isolated—the default behavior of a Spring Boot RestTemplate or WebClient is a catastrophic waste of resources. We tested a baseline scenario where a Payment-Service went offline (TCP port closed), and an Order-Service attempted to process 100 concurrent requests without a circuit breaker.

The result: The Order-Service thread pool (Tomcat default 200) was exhausted in less than 2.4 seconds. Even though the TCP ConnectException happens relatively quickly, the constant retry-and-fail cycle creates a "livelock" state. The CPU spikes to 85% just handling exception stack traces and thread context switching.

Downstream Down -> Threads Blocked -> Throughput Drops to Zero
[User] -> [Order-Service (100% Threads Busy)] -> [Payment-Service (OFFLINE)]
                                  |
                                  +--> [Inventory-Service (SITS IDLE)]

If you don't cut the line immediately, your upstream service dies trying to talk to a corpse. The goal isn't just to "handle" the error; it's to stop making the call entirely once we know the downstream is dead.

Can Generic Timeouts Prevent Thread Pool Exhaustion?

Many engineers assume a tight 500ms timeout is a sufficient circuit breaker. We measured this by setting client-connector timeouts on a WebClient bean and hammering it with 500 requests per second (RPS) while the target was unreachable.

* Metric: Active Thread Count * Result: 188/200 threads active. * P99 Latency: 540ms (despite the service being down).

The problem with relying solely on timeouts is that you still incur the cost of the connection attempt, the handshake wait, and the overhead of the reactive/blocking machinery for *every single request*. In a high-traffic system, 188 blocked threads still represent a massive degradation in the ability of the Order-Service to serve other, unrelated endpoints (like /health or /metrics).

A timeout is a reactive measure; it requires the failure to occur before it acts. A circuit breaker is proactive. Once it transitions to OPEN, the cost of a failed call drops from ~500ms (timeout) to ~1ms (instant rejection).

At What Threshold Should the Circuit Flip?

Setting the failureRateThreshold is the most debated configuration in Resilience4j. If you set it too low (e.g., 10%), a single network blip trips the circuit. If you set it too high (e.g., 80%), you’ve already saturated your thread pool before protection kicks in.

We simulated a "flaky network" scenario with a 15% packet loss.

1. Metric: Error Rate at 50% threshold. 2. Observation: The circuit stayed CLOSED, but P99s fluctuated wildly between 40ms and 2000ms. 3. Refined Metric: Error Rate at 25% threshold with a minimumNumberOfCalls of 10. 4. Observation: The circuit flipped to OPEN within 4 seconds of the degradation, stabilizing the P99 at 5ms for the fallback response.

The lesson here is that minimumNumberOfCalls is more important than the percentage. If you set it to 100, you have to suffer 100 failures before the breaker protects you. In a production environment with a P99 of 200ms, that means your service is degraded for at least 20 seconds before the breaker triggers. We found that a window of 10 to 20 calls provides the best balance between stability and sensitivity.

Implementing the Resilient Call Wrapper

To handle an unreachable service gracefully, we use the io.github.resilience4j:resilience4j-spring-boot3 starter. The configuration must be explicit about what constitutes a failure. For an unreachable service, java.net.ConnectException and java.net.UnknownHostException are the primary culprits.

```java
@Service
public class PaymentClient {

private final WebClient webClient;

public PaymentClient(WebClient.Builder builder) { this.webClient = builder.baseUrl("http://payment-service:8080").build(); }

@CircuitBreaker(name = "paymentService", fallbackMethod = "handlePaymentFailure") @Bulkhead(name = "paymentService", type = Bulkhead.Type.THREADPOOL) public CompletableFuture<PaymentResponse> processPayment(PaymentRequest request) { return webClient.post() .uri("/v1/payments") .bodyValue(request) .retrieve() .toEntity(PaymentResponse.class) .toFuture() .thenApply(ResponseEntity::getBody); }

// Fallback must have the same signature + the Exception parameter public CompletableFuture<PaymentResponse> handlePaymentFailure(PaymentRequest request, Throwable t) { log.error("Payment Service unreachable or failing: {}", t.getMessage()); // Return a safe, degraded response or a cached value return CompletableFuture.completedFuture(new PaymentResponse("PENDING_OFFLINE", 0.0)); } } ```

By adding a Bulkhead, we isolated the payment-service calls to a dedicated thread pool. When the circuit opened, the handlePaymentFailure was invoked immediately without even attempting to grab a thread from the global pool. In our testing, this dropped the "Unreachable Service" impact on neighboring components to near zero.

How Long Should the Service Stay in the OPEN State?

When a service is unreachable due to a pod restart or a network partition, it rarely recovers in 5 seconds. However, the default waitDurationInOpenState in many tutorials is a mere 10 seconds.

We monitored a Kubernetes cluster during a rolling update gone wrong. The Order-Service circuit breaker was set to 10 seconds. * Observed Behavior: The circuit would open, wait 10 seconds, enter HALF_OPEN, allow 3 requests through, fail (because the service was still down), and go back to OPEN. * Result: This "pulsing" effect caused spikes in the error logs and triggered unnecessary alerts every 10 seconds.

We increased the waitDurationInOpenState to 60 seconds and implemented an exponential backoff for the wait duration. This reduced the "pulsing" noise by 80%.

resilience4j.circuitbreaker:
  instances:
    paymentService:
      registerHealthIndicator: true
      slidingWindowSize: 20
      permittedNumberOfCallsInHalfOpenState: 5
      waitDurationInOpenState: 60s
      failureRateThreshold: 50
      eventConsumerBufferSize: 10
      recordExceptions:
        - org.springframework.web.reactive.function.client.WebClientRequestException
        - java.io.IOException
        - java.net.ConnectException
      ignoreExceptions:
        - com.org.exceptions.BusinessValidationException

Crucially, we added BusinessValidationException to the ignoreExceptions list. If a downstream service returns a 400 Bad Request, that is a successful communication from a network perspective. If you count 400s as failures, your circuit will trip because of invalid user input, not because the service is unreachable.

The Half-Open State: Precision vs. Speed

The HALF_OPEN state is the most critical phase of recovery. It’s where the circuit breaker tests the waters. If you allow too many calls in (permittedNumberOfCallsInHalfOpenState), you risk overwhelming a service that is just barely starting back up.

We tested a recovery scenario where the Payment-Service came back online but was under heavy load during its warm-up phase. * Trial A: 20 calls in HALF_OPEN. The service crashed again immediately due to the sudden burst. * Trial B: 3 calls in HALF_OPEN. The circuit breaker correctly identified the service was still struggling, flipped back to OPEN, and gave it another 60 seconds to stabilize.

The takeaway? Keep permittedNumberOfCallsInHalfOpenState low (between 3 and 5). You want a representative sample, not a stress test.

Impact on P99 Latency and System Stability

After implementing these patterns—specifically moving from simple timeouts to a Reslience4j circuit breaker with a dedicated Bulkhead—we saw a dramatic shift in the "unreachable service" profile.

In a controlled failure injection (cutting the network route to the payment database), we compared the two approaches under a load of 400 RPS.

| Metric | Timeout Only (Blocking) | Circuit Breaker + Bulkhead | | :--- | :--- | :--- | | P99 Latency (Global) | 2,800 ms | 110 ms | | Success Rate (Other APIs) | 62% | 99.8% | | Memory Footprint | 1.2 GB (Thread overhead) | 640 MB | | CPU Usage | 78% | 12% |

The most significant number here is the P99 dropping from 2800ms to 110ms. The "Global" P99 includes calls to other services that were completely healthy but were starving for threads because the Order-Service was "waiting to fail" on the payment calls.

Final Measurement: The Shutdown Phase

An often overlooked aspect of handling unreachable services is how the circuit breaker behaves during application shutdown. If the Order-Service is shutting down, it shouldn't be attempting to transition states or fire off fallbacks that require external resources (like logging to a remote DB).

We observed that without proper lifecycle management, the CircuitBreakerRegistry would throw RejectedExecutionException during JVM SIGTERM because the fallback logic was trying to execute on a thread pool that had already been destroyed.

The fix was ensuring that the Bulkhead and Circuit Breaker were ordered correctly in the Spring lifecycle, allowing the "protective shell" to stay active until the last request was drained.

To solve the "unreachable service" problem, you must stop viewing the failure as an exception to be caught and start viewing it as a state to be managed. The circuit breaker is not an error handler; it is a state machine that guards your system’s finite resources against the infinite emptiness of a downed network. The most resilient call you can make is the one you decide not to make at all.

Spring Boot Retry + Circuit Breaker with Resilience4j

The Cascade Failure Trap: Why Naive Retries Kill Services

Most engineers treat Retry as a silver bullet for transient 503s. They annotate a method, set maxAttempts: 3, and walk away. In a high-throughput Spring Boot environment, this is a recipe for a self-inflicted Distributed Denial of Service (DDoS). If your downstream service latency spikes from 50ms to 2s, and you blindly retry three times, you have just tripled the connection hold-time on your upstream thread pool.

The marriage of Resilience4j’s Retry and CircuitBreaker isn't just about "making it work"; it’s about math. You must ensure the Retry logic sits *inside* the Circuit Breaker’s scope. If the Retry is outside, the Circuit Breaker sees three failures for every one user request, tripping the breaker prematurely and preventing the very recovery you're trying to achieve.

Request Flow Topology:
[ Inbound Request ] 
       |
[ Resilience4j Aspect Handler ]
       |
       +-- [ Circuit Breaker (Outer Layer) ]
              |
              +-- [ Retry (Inner Layer) ]
                     |
                     +-- [ HTTP Client / Feign / WebClient ]
                            |
                            V
                     [ Downstream Service ]

When configured correctly, the Retry handles the "glitch" (a single dropped packet or a TCP reset), while the Circuit Breaker handles the "catastrophe" (the downstream service is OOMing or under heavy load).

Sizing Thread Pools and Connection Backlogs

When you introduce retries, your application's concurrency model changes fundamentally. If your service handles 500 Requests Per Second (RPS) and you allow 2 retries (3 total attempts), your potential outbound RPS is 1,500.

To calculate the necessary max-connections for your ConnectionPool, use the Little’s Law derivative for retries: Total Connections = Inbound RPS * (1 + Retry Rate) * Average Latency.

If your p99 latency is 200ms and you expect a 5% failure rate (triggering retries), your sizing looks like this: 500 * (1 + 0.05) * 0.2s = 105 concurrent connections.

However, during a partial outage where latency climbs to your 2-second timeout, that number explodes: 500 * (1 + 0.05) * 2.0s = 1,050 concurrent connections.

If your HikariCP or HttpClient pool is capped at 50, your threads will block waiting for a connection, leading to ConnectionPoolTimeoutException. You must size your IOPS and connection limits based on the *timeout* value, not the *average* latency. I've seen p99 response times drop from 850ms to 120ms simply by increasing the max-connections to 2x the calculated saturation point, preventing thread contention during minor blips.

Memory Overhead of Resilience4j Event Buffers

Resilience4j isn't free. Every Circuit Breaker maintains a SlidingWindow—either count-based or time-based. A count-based window of 1,000 entries stores the outcome (Success/Failure) and the duration of the last 1,000 calls.

For a high-volume service with 50 different downstream integrations, the memory footprint adds up: Memory = Number of Instances * Window Size * (Event Object Size).

An ArrayList backed sliding window with 1,000 entries consumes roughly 24KB to 40KB of heap depending on the metadata captured. While negligible for one instance, if you are dynamic-scaling and creating "per-user" or "per-tenant" breakers, you can easily leak hundreds of megabytes.

Always explicitly set eventConsumerBufferSize. If you leave it to default in a service emitting 10k events/sec, the internal RingBuffer used for monitoring (to feed Prometheus/Grafana) can cause significant GC pressure. Limit your eventConsumerBufferSize to 100 or less for production environments where you only sample metrics.

CPU Cycles and the Cost of Exponential Backoff

The CPU cost of Resilience4j itself is low, but the cost of the *waiting* strategy is not. If you use Thread.sleep (common in custom retry logic) or poorly configured non-blocking retries, you waste cycles. Resilience4j’s IntervalFunction with exponential backoff is the industry standard, but it requires careful tuning of the multiplier.

Formula for total wait time: Total Backoff = Initial Interval * (Multiplier ^ (Attempts - 1))

If you use an initialInterval of 100ms and a multiplier of 2.0 with 5 retries, your last retry waits 1.6s. Total time spent for a failing request is ~3.1s. In a CPU-bound Spring Boot application (like one doing heavy JSON parsing or crypto), holding that thread open for 3 seconds reduces your effective throughput.

resilience4j:
  retry:
    instances:
      inventoryService:
        maxAttempts: 3
        waitDuration: 100ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        retryExceptions:
          - feign.RetryableException
          - java.io.IOException
  circuitbreaker:
    instances:
      inventoryService:
        slidingWindowSize: 100
        permittedNumberOfCallsInHalfOpenState: 10
        waitDurationInOpenState: 10s
        failureRateThreshold: 50
        registerHealthIndicator: true

The critical lesson: Never retry on 4xx errors. Retrying a 403 Forbidden or 400 Bad Request is a waste of CPU and IOPS. Filter your retryExceptions to only include network-level issues or specific 5xx codes that are actually transient.

IOPS Saturation and the "Storming Herd"

When a circuit breaker transitions from OPEN to HALF_OPEN, it allows a limited number of "test" calls to pass through. If these calls succeed, the circuit closes. If you have 10 instances of a microservice all transitioning to HALF_OPEN at the same time, you hit the downstream service with a "storming herd."

If your downstream database is limited to 5,000 IOPS and it's already struggling, 100 concurrent "test" calls from 10 upstream instances can push it back over the edge.

To mitigate this, use zitter (randomized delay) in your backoff strategy. Resilience4j provides IntervalFunction.ofExponentialRandomBackoff(). By adding 10-20% jitter, you spread the IOPS load across a wider time window, preventing the synchronized retry spikes that often keep a struggling service in a permanent state of failure.

In one production incident, we found that synchronized retries were creating 40,000 IOPS bursts every 5 seconds on a legacy Oracle DB. By simply introducing a 250ms random jitter, those bursts flattened to a manageable 8,000 IOPS, allowing the DB buffer cache to recover.

Determining the Breaking Point with First Principles

To properly size the Circuit Breaker's failureRateThreshold, you must understand your baseline "background noise" of errors. In any distributed system, there is a non-zero failure rate due to GC pauses, network blips, or deployment restarts.

If your steady-state error rate is 0.1%, setting a failureRateThreshold of 1% is too sensitive. If it’s 50%, you might as well not have a breaker. The sweet spot is usually 3x your peak "normal" error rate.

The Math for Transition: If WindowSize = 100 and Threshold = 50, the breaker trips when 50 requests fail. If those 100 requests happen over 1 tick of your monitoring window (say, 1 second), you are protecting the system. If they happen over 1 hour, your window is too large, and you're reacting to stale data.

Use slidingWindowType: TIME_BASED for low-traffic endpoints to ensure the breaker trips based on recent reality, not failures that happened three hours ago. For high-traffic routes (>100 RPS), COUNT_BASED is more predictable for capacity planning.

The Sharp Edge of the Implementation

The most dangerous configuration in a Spring Boot environment is a Retry that wraps a CircuitBreaker. When the breaker is OPEN, it throws a CallNotPermittedException. If your Retry logic is configured to retry on "all Exceptions," it will catch this, wait, and try again—only to hit the open breaker again.

You have effectively created a busy-wait loop that consumes your own application threads while doing zero useful work.

Final Engineering Requirement: Always configure your Retry to ignore io.github.resilience4j.circuitbreaker.CallNotPermittedException. This ensures that when the breaker is open, the failure propagates immediately to the caller, allowing your system to shed load instantly.

The goal of resilience is not to hide errors at any cost; it is to fail fast and recover gracefully. By sizing your connection pools for the "timeout case" rather than the "average case," and by ensuring your retries don't trigger on circuit-breaker exceptions, you move from a fragile system to one that can survive a 50% packet loss scenario without dropping a single thread due to pool exhaustion.