FastAPI + Redis Caching — Make Your API Faster

The p99 Spike at 30k Requests Per Minute

It started at 14:12 UTC on a Tuesday. We were monitoring the telemetry for a high-traffic inventory service—a FastAPI-based microservice that queries a PostgreSQL database with a complex set of JOINs to calculate seasonal stock availability.

The service had been stable for months, humming along with a p99 of 240ms. Then, we hit a traffic spike. A marketing push triggered 30,000 requests per minute. Within three minutes, the p99 latency shot up to 14 seconds. The database CPU hit 98%, and DB connections started timing out. The application logs were flooded with sqlalchemy.exc.TimeoutError: QueuePool limit of size 10 overflow 10 reached.

The failure was predictable: every single request was forcing the database to re-compute the heavy inventory logic, despite the underlying data only changing every few minutes. We weren't just wasting CPU cycles; we were creating a self-inflicted Distributed Denial of Service (DDoS) on our own persistence layer.

The Post-Mortem: Why Simple Querying Failed

The bottleneck wasn't the FastAPI application code; it was the latency inherent in the PostgreSQL query execution and the I/O overhead of the DB pool. When traffic increased, the database became the global lock. Because our API endpoints were IO-bound, the asyncio loop in FastAPI worked perfectly, but the database couldn't keep up with the volume of concurrent queries.

```text
TRAFFIC FLOW BEFORE REDIS:

[Client] -> [Load Balancer] -> [FastAPI (Worker 1...N)] | v [PostgreSQL] <--- (Heavy JOINs, Sorting, Calculation) ```

We discovered that 92% of the queries returning to the inventory service were identical. We were calculating the exact same result set for the "Spring 2024 Collection" thousands of times per second. By failing to implement an intermediate hot-data layer, we were essentially treating our primary database as a cache, which is the most expensive way to handle read-heavy workloads.

Building the Async Redis Middleware

We needed a solution that didn't block the Python event loop. Using redis-py with its asyncio support was the non-negotiable choice. The goal was to intercept the request at the service layer, check Redis, and return the result before the database was ever touched.

We didn't want to litter our business logic with if cache.get(): blocks. Instead, we built a decorator-based approach using functools.wraps. This maintained the purity of our domain logic while adding performance as a cross-cutting concern.

```python
import json
import logging
from functools import wraps
from typing import Optional, Callable
from redis import asyncio as aioredis

# Global redis pool instance redis_client: Optional[aioredis.Redis] = None

def cache_response(ttl_seconds: int = 300, prefix: str = "fastapi-cache"): """ Decorator to cache FastAPI endpoint responses in Redis. Handles serialization of SQLAlchemy objects or Pydantic models. """ def decorator(func: Callable): @wraps(func) async def wrapper(*args, **kwargs): if not redis_client: return await func(*args, **kwargs)

# Generate a unique key based on the function name and arguments # We exclude 'self' or 'db' session objects from the key cache_key = f"{prefix}:{func.__name__}:{hash(str(args) + str(kwargs))}" try: cached_data = await redis_client.get(cache_key) if cached_data: return json.loads(cached_data) except Exception as e: logging.error(f"Redis lookup failed for {cache_key}: {e}")

# Execute the actual service function result = await func(*args, **kwargs)

try: # We assume the result is a JSON-serializable dict or list await redis_client.setex( cache_key, ttl_seconds, json.dumps(result) ) except Exception as e: logging.warn(f"Failed to write to Redis for {cache_key}: {e}") return result return wrapper def_decorator = decorator return def_decorator ```

The critical detail in this implementation is the try/except block around the Redis operations. Your API must never fail because the cache is down. This is the "Fail Open" philosophy: if Redis times out or the connection is refused, the service should gracefully fallback to the database.

Handling the Serializer Bottleneck

One hard-won lesson: json.dumps() is surprisingly slow for massive payloads in a hot path. When our inventory response reached 2MB of JSON, the serialization process started eating 15ms of CPU time per request.

We eventually switched to orjson, which is significantly faster for data serialization in Python. If your payloads are even larger, consider msgpack to save on both serialization time and network bandwidth between your FastAPI container and the Redis instance. In our case, switching from standard json to orjson dropped our internal processing time by 8ms per request, which adds up at 500 requests per second.

The Thundering Herd Problem

During our first deployment of the cache, we encountered a new issue: the Thundering Herd. When a high-traffic cache key expired (TTL reached 0), 50 concurrent requests would all see a "cache miss" at the exact same millisecond. They would all simultaneously hit the database to re-calculate the result.

```text
THE THUNDERING HERD EFFECT:

14:00:00 - Key "inventory:spring" expires. 14:00:00.001 - Request A misses -> DB Query 14:00:00.002 - Request B misses -> DB Query 14:00:00.003 - Request C misses -> DB Query ... 14:00:00.050 - Request Z misses -> DB Query ```

Instead of one query, we had 50 massive JOINs hitting PostgreSQL at once. To solve this, we implemented "Cache Locking" or "Singleflighting." Before querying the database on a cache miss, the worker tries to acquire a brief, atomic lock in Redis for that specific key.

If it gets the lock, it queries the DB and updates the cache. If it doesn't get the lock, it sleeps for 50ms and retires the cache lookup. This ensures only one worker performs the expensive computation, while the others wait a few milliseconds to receive the fresh cached value.

Connection Pooling and the `aioredis` Setup

Misconfiguring the Redis connection pool is a common way to kill your FastAPI performance. If you create a new connection per request, you’ll exhaust the available file descriptors on your OS. You must initialize the Redis pool during the FastAPI lifespan event to ensure the connection persists across the entire application lifecycle.

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from redis import asyncio as aioredis

@asynccontextmanager async def lifespan(app: FastAPI): # Initialize the Redis connection pool global redis_client redis_client = aioredis.from_url( "redis://localhost:6379", encoding="utf-8", decode_responses=True, max_connections=20 ) yield # Graceful shutdown await redis_client.close()

app = FastAPI(lifespan=lifespan) ```

We found that setting max_connections was vital. Without a cap, during a traffic spike, the FastAPI workers would attempt to open hundreds of connections to Redis, eventually hitting the maxclients limit on the Redis server (usually 10,000, but often lower in managed environments like AWS ElastiCache).

Strategic Cache Invalidation

TTL (Time To Live) is a blunt instrument. If you set it too high, users see stale inventory. If you set it too low, you hit the database too often. For our inventory service, we moved to a hybrid strategy:

1. Passive Invalidation (TTL): Every key has a maximum life of 10 minutes. 2. Active Invalidation (Events): We hooked into the SQLAlchemy after_update listener. Whenever a product's stock level is updated in the database, the service publishes a message to a Redis Pub/Sub channel or directly deletes the relevant cache keys.

# Inside the Inventory Update Logic
async def update_stock(product_id: int, new_quantity: int):
    await db.execute(update_query)
    # Proactively clear the cache for this product
    cache_key = f"inventory:product:{product_id}"
    await redis_client.delete(cache_key)

This "Delete-on-Update" pattern is far safer than "Update-on-Update." If you try to update the cache value during a write, you risk race conditions where an older write overwrites a newer one. Deleting the key forces the next reader to fetch the absolute truth from the database, ensuring eventual consistency.

The Results: p99 and Cost Savings

After implementing the async Redis layer and the locking strategy, we re-ran the 30k RPM load test.

The results were transformative: * p99 Latency: Dropped from 14s (during the incident) to 82ms under the same load. * Database CPU: Dropped from 98% to 12%. * Throughput: We were able to scale from 30k RPM to 120k RPM while maintaining sub-100ms response times without adding a single database replica.

The most shocking metric was the database IOPS. By offloading 90% of the read traffic to Redis, we reduced our RDS costs. We were able to downsize our PostgreSQL instance from an m5.4xlarge to an m5.xlarge, saving roughly $1,200 per month. The Redis ElastiCache node cost us about $150 per month, resulting in a net saving and a much more resilient system.

The One Detail You Can't Ignore

If you are running FastAPI with multiple worker processes (e.g., using gunicorn -k uvicorn.workers.UvicornWorker), remember that each worker has its own asyncio loop but shares the same Redis backend. The connection pool you define in your lifespan happens per worker process.

If you have 10 workers and set max_connections=20, you are potentially opening 200 connections to Redis. Always calculate your Redis maxclients capacity based on number_of_pods * workers_per_pod * max_connections_per_worker. Being off by one decimal point in this calculation is usually what causes "Unexpected Connection Reset" errors in the middle of a traffic surge.

Caching isn't just about speed; it's about protecting your system's most fragile resource—the persistent data store—from the volatility of the public internet. Tightening the Redis integration in FastAPI transformed our service from a fragile bottleneck into a high-throughput engine.