SLIs, SLOs and Error Budgets: How to Turn Reliable Enough Into a Number
One person says ship it, another says the system is not stable enough, and neither of them has a number — so the loudest voice wins. Here is the arithmetic that ends that argument: what an SLI actually measures and where to measure it, why an SLO is meaningless without a time window, how an error budget turns a target into an allowance you are meant to spend, and why burn rate is the only one of the four that tells you how much time you have left.
There is an argument that happens in every team, roughly once a week.
One person says: ship it, the change is ready. Another says: no, the system is not stable enough right now. Both of them are completely sure. Neither of them has a number. So the loudest voice wins, or the most senior one does, and a week later the same argument happens again about a different change — and nothing was learned, because nothing was measured.
The problem is not that two people disagree. The problem is that the word reliable does not mean anything yet. It is a feeling. And you cannot plan a release around a feeling.
SLIs, SLOs and error budgets are the machinery that turns that feeling into arithmetic. Four terms, and they stack:
- SLI — the number you measure.
- SLO — the target you set on that number, over a stated window.
- Error budget — what the target buys you: the failure you are allowed.
- Burn rate — how fast you are spending it.
First, a picture: two jugs on a shelf
Before any definitions, hold this image.
The first jug is locked — a chain and a padlock hold the lid shut. Not one drop is allowed to leave it. That sounds like the safest rule you could possibly write. But look at what it costs: nobody may open it. Nobody may fit a new tap, clean it, or repair the crack near the base. Perfect means frozen.
The second jug has measuring marks ruled up the side, a small tap at the bottom, and a chalk slate leaning against it. You are allowed to let out this much this month. You can see how much has already gone. And you can decide, today, whether to slow down or keep pouring.
That second jug is an error budget. Everything below is just how you rule the marks.
The SLI: the number you measure
The SLI — service level indicator — is nothing more mysterious than the measured number. Take your requests, decide what counts as a good one, divide the good by the total.
def sli(requests: list[Request]) -> float:
"""Fraction of requests that were good."""
if not requests:
return 1.0
good = sum(1 for r in requests if is_good(r))
return good / len(requests)
def is_good(r: Request) -> bool:
# Speed is part of correctness.
return r.status < 500 and r.latency_ms <= 300
Look closely at is_good. Latency is part of the definition of correct. A request that returns a perfectly valid 200 after nine seconds did not serve that customer — they had already closed the tab. If your SLI only counts status codes, you will report a healthy service on the exact day your users give up on you.
Where you measure decides what you are allowed to conclude
This is the mistake that ruins the number before you ever get to use it.
It is tempting to count inside the application, because that is where your code runs and that is where the instrumentation is easy to add. But think about what the application is able to see: it can only count requests that actually reached it.
Take 100,000 requests over some window:
| Measured at | Requests seen | Good | Reported SLI |
|---|---|---|---|
| Inside the app | 92,000 | 91,954 | 99.95% — everybody relaxes |
| At the edge (nginx) | 100,000 | 91,954 | 91.95% — the truth |
Eight thousand requests never got that far. The edge had no healthy application server to hand them to, so it returned 502 on its own and the app never heard about it. Your customers were living in the second number the whole time.
Measure at the edge, where the customer is standing. In practice that means the access log of whatever terminates the connection — nginx, your load balancer, your CDN. That log is not a nice-to-have; it is the only place the honest number can come from.
SLI vs SLO vs SLA
Three words that get mixed up constantly, separated once:
| What it is | Who it binds | What breaking it costs | |
|---|---|---|---|
| SLI | The measured number | Nobody — it is an observation | Nothing; it is just a fact |
| SLO | The target you set on the SLI | You, internally | You slow down and fix things |
| SLA | A promise in a customer contract | Your company, legally | Money. Credits, refunds, penalties |
The operational rule that falls out of this: always set your SLO stricter than your SLA. The gap between the two is your warning room — the space where you find out you are in trouble before a customer can send you an invoice about it. An SLA of 99.5% with an SLO of 99.9% means you start reacting while you still have four times the room you legally need.
The error budget: where it becomes arithmetic
Say the service takes 10 requests per second, steadily, and you measure over a rolling window of 30 days.
- 10 × 60 × 60 × 24 × 30 = 25,920,000 requests in the window.
- Set the SLO at 99.9% good.
- Invert it: 0.1% are allowed to be bad.
- 0.1% of 25,920,000 = 25,920 requests.
That is your error budget. It is not a failure. It is an allowance, and it is yours to spend.
You can read the same target as time instead of requests: 30 days is 43,200 minutes, and 0.1% of that is 43 minutes and 12 seconds of complete outage.
Say the window out loud, every single time. The common recommendation is actually a rolling four weeks, not 30 days. On 28 days the very same 99.9% target allows 40.3 minutes instead of 43.2. A percentage with no window attached to it means nothing — 99.9% measured over a year hides a four-hour outage completely.
Spending it: one bad Tuesday
Abstract allowances do not change behaviour. Spending one does.
On a Tuesday you push a release and it is broken. For three minutes, every single request fails.
- 180 seconds × 10 req/s = 1,800 failed requests.
- 1,800 / 25,920 = 6.94% of the month's budget.
One bad deploy, three minutes long, spent almost seven percent of the entire month. Not seven percent of that Tuesday — seven percent of the month. That single sentence changes how a team argues about releases.
Put it in code, not in someone's head
def error_budget(objective: float, total_requests: int) -> float:
"""How many bad requests the objective allows."""
return (1.0 - objective) * total_requests
def budget_spent(requests: list[Request]) -> int:
"""Bad requests inside the window."""
return sum(1 for r in requests if not is_good(r))
def budget_remaining(objective: float, requests: list[Request]) -> float:
"""Fraction of the budget left, 1.0 = untouched, 0.0 = gone."""
allowed = error_budget(objective, len(requests))
if allowed <= 0:
return 0.0
return max(0.0, 1.0 - budget_spent(requests) / allowed)
budget_remaining is the number you put on a wall where the whole team can see it. Everything else here is built on top of it.
The trap: comparing to the target instead of the budget
Here is the mistake that catches careful teams, the ones who are actually looking at their dashboards.
You check yesterday. Yesterday was 99.5%. Your target is 99.9%. That is only 0.4 percentage points below — it feels close.
It is not close at all.
- Yesterday had 864,000 requests. 0.5% of them failed = 4,320 bad requests.
- Against a monthly allowance of 25,920, that is 16.67% of the month.
One day ate one sixth of the budget — five days' worth of the daily allowance, spent inside a single day. The daily number looked almost fine because it was being compared to the wrong thing.
Compare to the budget, never to the target. A percentage next to a percentage always looks reasonable. A percentage against a finite allowance tells you the truth.
Burn rate: the only number with time in it
Which brings us to the most useful quantity in the whole subject. Burn rate is one line of arithmetic:
def burn_rate(bad_fraction: float, objective: float) -> float:
"""How many times faster than 'exactly on target' you are spending."""
allowed_fraction = 1.0 - objective
return bad_fraction / allowed_fraction
Take the fraction of requests that are bad right now and divide it by the fraction you are allowed to have.
| Burn rate | Meaning | 30-day budget gone in |
|---|---|---|
| 1 | Exactly on target | Exactly 30 days — you finish as the window ends |
| 2 | Twice the allowed failure rate | 15 days |
| 14.4 | The classic fast-burn threshold | ~2 days |
A burn rate of 1 is not a problem — it means you spend all of the budget and run out of nothing early. That is the system working as designed.
Burn rate turns a percentage into a speed, and a speed is the only thing that tells you how much time you have left.
Alerting on speed instead of thresholds
Now you can page people for something meaningful — not a single failed request, and not a threshold somebody picked because it felt about right. Two rules cover most of real life:
| Rule | Condition | Budget consumed | Response |
|---|---|---|---|
| Fast burn | Burn rate > 14.4 for 1 hour | 2% of the month, in an hour | Page someone. Wake them up. |
| Slow burn | Burn rate > 6 for 6 hours | 5% of the month, quietly | File a ticket. No phone call. |
The second half of each rule is what makes it usable in production: pair the long window with a shorter window that also has to agree before the alert fires. That is what stops you being paged for a blip that already fixed itself.
And one warning about 14.4: it is not a magic constant. It is 2% × hours_in_window — 0.02 × 720 = 14.4 for a 30-day window. On a 7-day window the same policy gives you 0.02 × 168 = 3.36. Work it out from your own window rather than copying the number off a slide.
One honest month
Run a real month, with real incidents in it, against that 25,920-request allowance:
| Incident | Duration | Failure rate | Bad requests |
|---|---|---|---|
| Bad config shipped | 8 min | 100% | 4,800 |
| Database failover | 20 min | 50% of reads | 6,000 |
| Cache stampede after a purge | 100 min | 10% timing out | 6,000 |
| Noisy neighbour on one box | 4 min | 2% | 48 |
| Total spent | 16,848 / 25,920 = 65% | ||
Sixty-five percent of the budget gone with eleven days still to go — 9,072 requests left, about fifteen minutes of outage.
Now look at the shape of that list. The short sharp blip, the noisy neighbour, cost 48 requests: essentially nothing. It was the long, partial failures — the 100-minute cache stampede where only one request in ten timed out — that emptied the jug. Nobody paged anybody for that one. It never looked like an outage.
That is the opposite of where most teams spend their attention.
What the number actually decides
This is the part that matters, and it is a policy you agree on once, in advance, while nobody is angry yet:
| Budget remaining | Policy |
|---|---|
| > 50% | Ship freely. Take the risky refactor. Break something on purpose to see what happens. A budget you never touch is a target that was set too loosely. |
| < 33% | Slow down. Smaller releases. Deploy in the morning when people are awake, not at 2am. |
| 0% | Feature freeze. The whole team moves onto reliability work until the window rolls forward. |
And you can make it automatic, so it stops being a conversation at all:
import sys
MIN_BUDGET_TO_DEPLOY = 0.35
def deploy_gate(objective: float, requests: list[Request]) -> None:
remaining = budget_remaining(objective, requests)
if remaining >= MIN_BUDGET_TO_DEPLOY:
print(f"OK: {remaining:.1%} of the error budget left. Deploying.")
return
print(
f"BLOCKED: only {remaining:.1%} of the error budget left "
f"(need {MIN_BUDGET_TO_DEPLOY:.0%}). Reliability work first.",
file=sys.stderr,
)
sys.exit(1)
The argument this post opened with — ship it versus no, it is not stable enough — is now a script that returns 0 or 1. And it says the same thing to everybody, including the most senior person in the room.
The same machinery for LLM and AI services
If you are running an inference API, a RAG pipeline or an agent, you have this problem worse than a CRUD backend does, because "did it work?" is genuinely harder to define. The structure carries over unchanged — only is_good gets more interesting.
| Layer | Classic web service | LLM / AI service |
|---|---|---|
| Latency SLI | Response under 300ms | TTFT (time to first token) under ~500ms for streaming, plus TPOT (time per output token) — a single end-to-end number is useless when responses stream |
| Correctness SLI | status < 500 | Valid schema on structured output, tool call parsed and executed, no refusal on an in-policy request, groundedness score above threshold on a sampled eval set |
| Capacity SLI | Not shed by the load balancer | Not rejected by the admission queue — GPU batching means the failure mode is queue wait, not CPU saturation |
| What the budget buys | Risky deploys | Model swaps, prompt changes, quantization, a bigger batch size — every one of which is a live experiment on quality and latency at the same time |
Two things bite specifically here. First, TTFT and total latency need separate SLOs; a model that starts streaming in 300ms and finishes in 20 seconds is a good experience, and one number cannot express that. Second, an error budget is exactly the right instrument for prompt and model changes, because those are the deploys nobody thinks of as deploys. Swapping a model version is a production change with no code diff attached — and the budget is the thing that makes it cost something.
Measure at the edge here too. The gateway in front of your model servers sees the requests that timed out in the queue and never reached a GPU. Your inference server does not.
Four honest catches
1. A target of 100% is a bug, not an ambition. It gives you a budget of zero, so every deploy is forbidden and the very first failed request has already broken it. That is the padlocked jug.
2. Do not set the target from what your graph happened to do last month. Set it from what your users actually need, then measure the distance to it. Reverse-engineering an SLO from current performance just enshrines the status quo.
3. A percentage with no window means nothing. Say the window out loud, every single time.
4. An error budget only works if the freeze is real. If leadership can overrule the freeze, you do not have an error budget — you have a chart.
The verdict
Four lines is the whole subject:
- The SLI is the number you measure — and you measure it at the edge, where the customer is standing, not inside the app that never saw the failed requests.
- The SLO is the target you set on that number, and it always comes with a window attached.
- The error budget is what falls out of the target:
1 − objective, expressed as requests or minutes. It is an allowance you are meant to spend, not a failure you are meant to avoid. - The burn rate is how fast you are spending it — the only one of the four that tells you how much time you have left.
Or, the version worth carrying into your next release meeting:
"Reliable enough" is a feeling until somebody puts a number on it. The number is not there to make you safer — it is there to tell you how much risk you have already paid for.
Your turn: your team has a 99.9% SLO over 30 days and is 22 days into the window with 8% of the budget left. Somebody wants to ship a database migration. What does the policy say — and, more importantly, did you agree on that policy before today?
Prefer the 2-minute version? Watch the Short — or the full 13-minute episode. Follow along on Telegram for more CS fundamentals and system design.