Observability in Microservices — Prometheus, Grafana and OpenTelemetry

The Myth of Isomorphic Latency: Why Sidecar Overhead Costs 4ms Standard microservices architecture assumes that adding a service mesh or an observability sidecar is "negligible." To verify this, we benchmarked a Spring Boot 3.3.x service running on a 2-node EKS cluster (m5.large). We measured the round-trip latency of a simple `/health` endpoint under two conditions: a direct connection to the container and a connection routed through an OpenTelemetry (OTel) Collector sidecar.

| Metric | Direct Pod Access | OTel Sidecar Choke | | :--- | :--- | :--- | | Mean Latency | 2.1ms | 5.8ms | | p99 Latency | 8.4ms | 13.9ms | | CPU Usage (mCore) | 12m | 45m |

The data shows a consistent 3.7ms entry tax just for hitting the sidecar's telemetry pipeline before the application code even executes. This is often caused by greedy batch processor settings in the otel-collector-config.yaml. If your timeout is set to 200ms and your send_batch_size is too high, the collector holds spans in memory, introducing jitter.

Conclusion: In high-throughput systems, do not use sidecars for every pod. Instead, deploy the OTel Collector as a Kubernetes DaemonSet. It reduces the per-pod footprint and allows for better resource pooling, bringing that 4ms tax down to under 1ms.

Span Bloat: Does High-Cardinality Attribution Kill Throughput? Engineers love to add custom tags to every OpenTelemetry span. We tested the performance impact of adding 20 high-cardinality attributes (e.g., `user_id`, `session_uuid`, `request_id`) per span versus the standard 4 attributes (e.g., `service.name`, `http.method`).

# Load testing the OTLP exporter with K6
k6 run --vus 100 --duration 30s script.js

Measurement: - Baseline (4 tags): 4,200 Requests Per Second (RPS) before CPU saturation. - High-Cardinality (20 tags): 3,100 Requests Per Second (RPS). - Memory Pressure: The heap usage of the javaagent rose from 120MB to 310MB.

The bottleneck isn't the network; it's the serialization overhead of the OTLP (OpenTelemetry Protocol) protobuf generator within the JVM. Every dynamic string you add to a span must be encoded. When we moved user_id from a span attribute to a logged metadata field (MDC), throughput recovered by 18%.

Conclusion: Keep spans lean. If you need high-cardinality data for debugging, push it to logs or use Exemplars in Prometheus. Do not poison your tracing pipeline with values that change on every single request.

The Prometheus Scrape Gap: 15s vs 60s How much resolution do you actually need? We ran an experiment where we simulated a "micro-burst" failure: a downstream database connection pool saturation that lasted exactly 22 seconds. We scraped the Prometheus metrics at two different intervals: 15 seconds and 60 seconds.

[ Traffic Spikes ] ---> [ DB Pool Exhausted ] ---> [ Recovery ]
|--- 0s --- 10s --- 20s --- 30s --- 40s --- 50s --- 60s ---|
            ^ Burst Starts        ^ Burst Ends

Measurement: - 60s Scrape: The dashboard showed a slight bump in latency but never crossed the critical 95% threshold. No alert was fired. - 15s Scrape: Two data points captured the 100% saturation. The alert manager fired a P1 incident within 45 seconds of the event start.

The cost of this resolution is storage. A 15s interval generates 4x the data of a 60s interval. For a cluster with 50,000 active series, this meant an increase from 12GB to 48GB of TSDB data over a 15-day retention period.

Conclusion: Use 15s scrape intervals for service_level_indicator (SLI) metrics like http_server_requests_seconds_bucket, and use 60s or even 120s for "background" metrics like jvm_memory_used_bytes.

Distributed Tracing: The Ratio of Sampling to Sanity Tracing every request (100% sampling) in a microservices environment doing 10,000 RPS is a ticket to a massive cloud bill and a slow UI. We measured the UI responsiveness of Grafana Tempo when querying traces from a 24-hour window at different sampling rates.

| Sampling Rate | Index Size | Trace Query Time (Avg) | | :--- | :--- | :--- | | 100% | 1.2 TB | 42.0s | | 10% | 120 GB | 4.1s | | 1% | 12 GB | 0.8s |

Measurement: At 100% sampling, the tempo-querier pods frequently hit OOM (Out of Memory) kills when users tried to find "slow traces" because the result set was too large to sort in memory.

Hard-won lesson: We initially thought 1% sampling would miss critical errors. We were wrong. By using a tail-based sampler in the OTel Collector, we configured a policy to keep 100% of traces that resulted in an HTTP 5xx or took longer than 500ms, while keeping only 0.1% of "happy path" HTTP 200s.

# otel-collector-config.yaml excerpt
processors:
  tail_sampling:
    policies:
      - name: errors-policy
        type: status_code
        status_code: {values: [ERROR]}
      - name: slow-traces-policy
        type: latency
        latency: {threshold_ms: 500}
      - name: probabilistic-policy
        type: probabilistic
        sampling_percentage: 0.1

Conclusion: Head-based 100% sampling is for development. Production requires tail-based sampling to balance visibility with storage costs.

Micrometer Observation API: Standardizing the Stack Spring Boot 3 replaced the old `Timed` annotation with the `Observation` API. We compared the development effort and the resulting data quality of using the manual `OpenTelemetry SDK` versus the `Micrometer Observation` wrapper.

Observation: 1. Manual SDK: Requires 15 lines of boilerplate per method to start a span, set the scope, and handle errors. Missing a .close() on a scope leads to thread-local memory leaks. 2. Micrometer API: One @Observed annotation or a Observation.observe() lambda.

```java
// Modern Spring Boot 3 Observation Logic
private final ObservationRegistry observationRegistry;

public UserProfile getProfile(String id) { return Observation.createNotStarted("user.profile.lookup", observationRegistry) .lowCardinalityKeyValue("user.type", "premium") .highCardinalityKeyValue("user.id", id) .observe(() -> userRepository.findById(id)); } ```

Result: Using the Micrometer abstraction reduced our "observability boilerplate" code by 70%. More importantly, it automatically links your Prometheus metrics to your OpenTelemetry traces using Exemplars. In the Grafana dashboard, we could click a spike in the Prometheus graph and jump directly to the specific trace ID that caused it.

Conclusion: Stop using the OpenTelemetry API directly in your business logic. Use Micrometer Observation as a facade. It’s cleaner, safer, and provides the context-linking that makes Prometheus-to-Trace navigation actually work.

Grafana Dashboard Load Times: The Variable Trap If your Grafana dashboard takes 20 seconds to load, your engineers won't use it. We investigated why a "Standard Microservice Dashboard" was lagging. The culprit: Template Variables.

The dashboard had a variable $pod defined by the query: label_values(container_cpu_usage_seconds_total, pod). On a cluster with 4,000 pods, this single query took 8 seconds to run every time the dashboard refreshed.

Measurement: - Dynamic Variable: 8.2s load time. - Chained Variable (Namespace -> Service -> Pod): 1.4s load time.

By filtering the $pod query by a previously selected $namespace and $service, we reduced the data set Prometheus had to scan by 99%.

Conclusion: Never use global label_values queries without filters. Always chain your variables. Start with a Namespace selector, then a Service selector, then a Pod selector.

The Cost of Excessive Logging: `INFO` is the New `DEBUG` Most teams log the request and response body at `INFO` level. We benchmarked the throughput of a Spring Boot service with `ConsoleAppender` sending logs to `stdout` (which FluentBit then collects) under heavy load.

• Log Level WARN: 8,500 RPS | 12% CPU.
• Log Level INFO (with Request Tracing): 5,200 RPS | 34% CPU.
• Log Level DEBUG: 1,800 RPS | 88% CPU.

The "hidden" cost of logging isn't just the disk space; it's the context switching and the lock contention on the ConsoleAppender. Even with asynchronous logging (Logback AsyncAppender), the overhead of string formatting for high-volume INFO logs caused a p99 spike of 40ms.

One specific win: We moved our p99 from 480ms to 90ms simply by switching from standard-out logging to Mapped Diagnostic Context (MDC) logging in a JSON format with a non-blocking disruptor-based appender.

Conclusion: If you can't justify an alert for a log line, it's not an INFO. It's a DEBUG. Turn it off in production.

Operational Takeaway: The 5% Observability Budget Observability is not free. Between Prometheus storage, Tempo trace indexing, and the CPU cycles spent in the OTel Collector, you are paying a performance tax.

Our final measurement showed that a fully observable system (Metrics + Traces + Logs) consumes roughly 12% of the total cluster compute and adds an average of 3-5ms to the request lifecycle. By optimizing the collector to run as a DaemonSet, moving cardinality to logs, and enforcing tail-based sampling, we brought that overhead down from 25% to 7% while maintaining 100% visibility into system failures.

The sharp lesson: Optimize your telemetry pipeline with the same rigor you optimize your database queries. A neglected observability stack will eventually become the loudest source of latency in your cluster.