Skip to content

Day 38 of 100

The SLO With the Wrong SLI

SREAdvanced20–25 min

🔥 Problem Statement

The reliability report looks excellent — 99.99% availability, comfortably meeting the SLO, month after month. It's the kind of number a team would proudly put in front of leadership.

Customer support tells a different story: recurring complaints, specifically about the product feeling slow, not about it being down. Nobody using the product is reporting outages. They're reporting frustration with how long things take.

🏗️ Environment

  • SLO: 99.99% availability, tracked and reported monthly
  • SLI: HTTP success rate across all endpoints, aggregated
  • Support: recurring customer complaints about slowness
  • No latency-based SLO currently defined

🤔 Your Challenge

Is the current SLI actually measuring the thing customers care about — and if not, what is it missing?

  • Does 'the request eventually returned a 200' capture whether it returned fast enough to be a good experience?
  • What's the difference between availability and latency as reliability dimensions, and does this SLO cover both?
  • Could an aggregate success-rate metric across all endpoints hide a specific slow, high-traffic endpoint?
  • What would a customer actually mean by 'the site is broken' versus what the current SLI measures as 'broken'?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

An HTTP success-rate SLI answers one specific question: did the request eventually return a successful status code. It says nothing about how long that took. A request that takes eight seconds to return a 200 counts identically, in this SLI, to one that returns in eighty milliseconds — both are "available." Customers experiencing slowness aren't necessarily experiencing unavailability by this metric's definition, which means the SLO can stay green indefinitely while the exact thing users are complaining about goes completely unmeasured.

Step 2 — Identify the likely bottleneck

This isn't a data quality problem — the availability number is probably accurate for what it measures. It's a scope problem: the SLI was chosen to represent "is the system reliable," but it only actually represents one dimension of reliability (does it eventually respond successfully) while leaving out another dimension entirely (does it respond fast enough to be a good experience). Latency degradation, on a specific endpoint or across the board, can persist for a long time without ever registering as an availability problem.

Step 3 — Investigation

  • Pull latency percentiles (P50, P95, P99) for the product's key user flows, not just an aggregate success rate — this directly tests whether latency has actually degraded, independent of availability.
  • Correlate latency trends against the timing of customer complaints — if P95 or P99 latency has crept up over the same period complaints increased, that's a strong, direct link.
  • Check whether latency degradation is evenly spread or concentrated on specific endpoints or user flows — an aggregate view can easily hide a single slow, high-traffic endpoint behind an otherwise-healthy average.
  • Talk to support directly about the specifics of complaints — which actions, which pages, how slow, how often — to identify the actual user-facing symptom precisely enough to know what to measure.

Define a latency-based SLO alongside the existing availability SLO — for example, a target percentage of requests (or of key user-journey completions) served within an acceptable latency threshold, based on what's actually tolerable for this product's users. Scope it to the specific flows that matter most, not just a blanket aggregate across every endpoint, since a critical, frequently used flow deserves its own visibility rather than being averaged in with far less important ones.

Once in place, this new SLO becomes the thing actually tracked against the complaints support has been receiving — closing the loop between what's measured and what users are experiencing, instead of the two continuing to diverge silently.

Step 5 — Engineering lesson

An SLO is only as good as the SLI underneath it, and a green SLO measuring the wrong thing isn't lying — it's answering a real question, just not the one that matters to the people actually using the product. Customer complaints that contradict a green dashboard aren't noise to explain away; they're a direct signal that the metric and the actual user experience have drifted apart, and the fix is usually adding or correcting what's being measured, not dismissing what customers are telling you.

🧠 Today’s Takeaway

You cannot improve reliability if you measure the wrong thing.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.