Error Budgets: The Contract That Funds Reliability Work
Last updated . Sources are named and dated inline - how we source claims.
An error budget is the one mechanism in common use that converts an argument about technical debt into an arithmetic fact. Either the budget is there to spend, or it is not.
Most debt conversations fail because they are conversations, and the loudest person wins. An error budget replaces the negotiation with a number both sides agreed to before anything broke, plus a policy saying what happens when the number runs out.
What an Error Budget Actually Is
Three terms get used interchangeably and should not be. A service level indicator (SLI) is, in the words of Google's Site Reliability Engineering book, "a carefully defined quantitative measure of some aspect of the level of service that is provided." It is a measurement, not a goal. A service level objective (SLO) is "a target value or range of values for a service level that is measured by an SLI." That is the goal. A service level agreement (SLA) is something else entirely: "an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain." The SLA has money attached. Your internal SLO should always be stricter.
The error budget falls out of the SLO by subtraction. The Site Reliability Workbook puts it in one line: "the SLO is a target percentage and the error budget is 100% minus the SLO." Commit to serving 99.9 percent of requests successfully over 30 days and you have committed to a 0.1 percent allowance for failure. That allowance is not a regrettable leftover. It is a resource with a size, a refill schedule, and an owner.
Do the arithmetic out loud, because that is what makes it real. The Workbook's own burn-rate table works this exact case: at a 100 percent error rate, a service with a 99.9 percent SLO exhausts its entire monthly budget in about 43 minutes. A bad deploy taking 20 minutes to roll back has just spent half the month. Say that in a planning meeting and the deploy pipeline stops being a preference and starts being a line item.
Reliability was a quality adjective. Now it is a quantity with a balance, and anything draining that balance faster than planned is a cost you can name, measure, and schedule work against. That is the whole trick.
An SLO Is a Product Decision, Not an Engineering One
Teams get this wrong constantly, and getting it wrong quietly destroys the mechanism. In Google's model the objective is set by the people who own the product, not the people who operate it: "Product Management defines an SLO, which sets an expectation of how much uptime the service should have per quarter. The actual uptime is measured by a neutral third party: our monitoring system."
Two reasons this matters. First, legitimacy: if engineering sets the target, engineering is grading its own homework, and every freeze looks like engineers inventing a reason to stop shipping. If product sets it, the freeze is product enforcing its own commitment. Second, it is genuinely a product question. Is this a checkout flow where a failure costs a sale, or a reporting job where a failure costs a retry? Engineers can say what a target costs to hold. Only the person accountable for the outcome can say what it is worth.
The measurement, though, must belong to neither party. When the team judged by the number also computes it, the number drifts. Measure the SLI in the monitoring system, as close to the user as you can get, and keep the arithmetic boring and public. See observability debt for what to do when the telemetry needed for an honest SLI does not exist yet, which for most teams is the real first task.
The Budget Is a Currency
Once reliability has a balance, the two things engineering organizations fight about stop being opponents and become purchases from one account.
Spending It Buys Velocity
Every risky thing you want to do -- a fast release cadence, a canary that skips its soak, a migration cut over on a Thursday -- consumes budget in expectation. That is not an abuse of the mechanism, that is the mechanism. A budget you never spend was set too loose, or your organization is too timid to use what it paid for.
Exhausting It Buys a Freeze
The SRE book states the rule plainly: "As long as the system's SLOs are met, releases can continue. If SLO violations occur frequently enough to expend the error budget, releases are temporarily halted while additional resources are invested in system testing and development." The freeze is not a punishment. It is the price agreed in advance for velocity already taken.
The Incentives Converge
The most useful line in the chapter is about behavior, not arithmetic: "When the budget is large, the product developers can take more risks. When the budget is nearly drained, the product developers themselves will push for more testing or slower push velocity." Reliability stops being the operations team's hobby and becomes a shared balance the feature team manages.
The Policy: A Contract Signed Before the Incident
An error budget without a policy is a dashboard. It informs, and then nothing happens. The policy does the work, because it is written and approved while everyone is calm, and it therefore removes the decision from the moment when nobody is.
Consider the alternative. The budget is blown and someone now has to propose halting the release train. That proposal costs them politically, gets litigated in a room where the loudest stakeholder has a launch date, and gets relitigated next quarter. A pre-agreed policy converts that recurring negotiation into a lookup: we are past the threshold, so the agreed action applies. The argument was had once, in advance, with nobody's launch on the line.
What the Policy Must Contain
The Site Reliability Workbook is specific, and every item on its list exists because leaving it out caused a problem somewhere. The policy documents "the policy authors, reviewers, and approvers"; "the date on which it was approved, and the date when it should next be reviewed"; "the actions to be taken in response to budget exhaustion"; and "a clear escalation path to follow if there is disagreement."
It also names who signs: the product manager, the development team, and the SREs responsible for defending the SLO. Three signatures, because the policy only has force if the person who will later want to override it already put their name on it. On exhaustion, the Workbook describes actions such as prioritizing reliability bugs, focusing exclusively on reliability issues, or instituting "a production freeze [that] halts certain changes."
The escalation path is the clause people skip and the clause that gets tested first. Disagreement is not a failure of the policy, it is a normal event. What matters is that the route is known in advance, goes to a named person, and is visible when used. An override requested in writing from a specific executive is a fundamentally different thing from one that happens because a director said "not this sprint" in a stand-up.
How a Policy Turns a Complaint Into Funded Work
Take the most common unfunded piece of debt on earth: the flaky deploy pipeline. Here is the same problem run through both systems.
Without a Budget: An Opinion
An engineer says the pipeline is flaky and rollbacks are slow. True, and indistinguishable from a preference: no size, no deadline, no owner outside the team already annoyed by it. It competes against a feature with a launch date and a revenue projection, and loses. It loses again next quarter, by which point the engineer has stopped raising it -- which leadership reads as the problem having gone away.
With a Budget: An Attributed Cost
Every incident consumes budget, and every incident review records which incident consumed how much. After two months the ledger says bad deploys and slow rollbacks account for most of the burn on this service. That is no longer an engineer's opinion. It is the answer to "where did our reliability allowance go", and the product manager who set the objective now has a direct interest in the answer.
The Policy Makes It Automatic
Now the policy fires. The budget is exhausted, the agreed action is that the team works reliability items until the balance recovers, and the top item is the thing draining the balance. Nobody had to sell the pipeline work, build a business case, or find a sympathetic executive. It was funded by a rule the organization wrote for itself, before it knew which piece of debt the rule would end up paying for.
And Its Honest Limit
The pipeline gets fixed, burn drops, and the freeze lifts on its own -- the mechanism needs nobody to keep believing in it, which is what separates it from a quality initiative. But it only funds debt that shows up in the SLI. A hostile build or an untested module nobody touches burns no budget and must be argued for on other grounds. See measuring tech debt for the metrics covering that half.
Burn Rate Alerting Versus Threshold Alerting
Threshold alerting asks a local question: is the error rate above some line right now? It is easy to implement and it is why so many teams drown in pages. Set the line low and you page for every transient blip. Set it high and you miss the slow bleed that costs you the whole month.
Burn rate asks a budget question instead. The Workbook defines it as "how fast, relative to the SLO, the service consumes the error budget." A burn rate of 1 means the service is on track to consume exactly its whole budget by the end of the window and no more. A rate of 2 means it will be gone in half the time. The threshold is no longer an error rate you guessed at; it is a statement about how much of a finite resource you will lose before waking someone up.
| Burn rate | Error rate | Time to exhaustion |
|---|---|---|
| 1 | 0.1 percent | 30 days |
| 2 | 0.2 percent | 15 days |
| 10 | 1 percent | 3 days |
| 1,000 | 100 percent | 43 minutes |
The failure burn-rate alerting exists to catch is the sustained moderate error. The Workbook describes simple threshold approaches as having low recall precisely because that case slips past them, and works an example where a sustained elevated error rate consumes an entire monthly budget in roughly 20.5 hours without ever tripping a simpler alert. Nobody is paged. The month's allowance is simply gone, and the team finds out at the review.
The practical implementation is a pair of windows. A short window at a high burn rate catches the outage that will destroy the budget within the hour and pages a human. A long window at a low burn rate catches the slow leak and opens a ticket instead. Together, every page corresponds to something genuinely threatening a commitment you made -- the only sustainable definition of an actionable alert, and the fastest route out of the alert fatigue described in observability debt.
Why Perfect Reliability Is the Wrong Target
The SRE book does not hedge: "100% is probably never the right reliability target: not only is it impossible to achieve, it's typically more reliability than a service's users want or notice." The second half is the interesting part. Users on mobile networks, behind corporate proxies, on aging devices, already experience failures your service did not cause. Past a point, additional nines are invisible to the person you built them for.
The cost of those invisible nines is not invisible at all: "Extreme reliability comes at a cost: maximizing stability limits how fast new features can be developed and how quickly products can be delivered to users, and dramatically increases their cost." Each additional nine tends to demand a qualitative change rather than more of the same effort -- redundancy across failure domains, failover you actually test, change control that slows every release, and staffing to defend it around the clock. So "good enough" is not a compromise, it is the goal. Find it by asking what the next nine would cost and what it would buy, then declining when no user would notice.
One more consequence. Chronically overachieving your SLO is not free either, because users calibrate to what they observe rather than what you promised. The SRE book notes that "users build on the reality of what you offer, rather than what you say you'll supply, particularly for infrastructure services." A service that has run at five nines for a year has quietly made a five-nines promise, whatever the document says. Some organizations deliberately take a service down inside the budget to keep that expectation honest.
Choose Indicators Users Feel, Not Ones That Are Easy to Measure
The most common way an error budget program dies quietly is that the SLI measures something nobody cares about. The instruction in the SRE book is one sentence long: "Start by thinking about (or finding out!) what your users care about, not what you can measure." Easy to say, hard to do, because the metrics already lying around are the ones that were cheap to collect.
The Easy Metric Trap
Host uptime, CPU headroom, and "the load balancer says the pool is healthy" are all trivially available and all compatible with a completely broken product. A service can be up on every host while returning an empty cart to every customer. If your SLI can be green during an outage users would call total, it is measuring the infrastructure, not the service.
Measure the Journey
Pick the two or three journeys that define the product -- sign in, search, check out -- and measure their success and speed as close to the client as your telemetry allows. A good test of a candidate SLI: would a support agent agree the service was broken during every window the SLI says it was broken? If not, you have the wrong indicator.
Latency Is Availability
A response arriving after the user gave up is a failure your success-rate SLI will happily count as a success. Fold the latency threshold into the indicator: the proportion of requests that succeeded and returned within target. Measure at a high percentile, because averages hide exactly the tail users remember and tell other people about.
Keep the Set Small
A service with a dozen SLOs has none, because something is always red and the policy therefore never fires with credibility. Two or three indicators per service, each tied to a journey that matters, is enough.
The Failure Modes
Error budget programs rarely die from a bad formula. They die from one of these, and every one is a governance problem wearing an engineering costume.
Set So Loose It Never Fires
Someone picks a target the service already comfortably beats, usually to avoid an uncomfortable conversation about the current state. The budget is never exhausted, the policy is never invoked, and within a year the apparatus is dropped as overhead. Diagnose it with one question: has this policy ever changed what anyone did? If not, the target is theatre.
Nobody Enforces It
The dashboard is red and has been red for six weeks. Everyone can see it. Nothing happens, because no action was ever attached to the number. This is the usual outcome for teams that adopted SLOs from a conference talk without adopting the policy that gives them teeth. A budget with no consequence is a metric, and metrics with no consequence get ignored within a quarter.
The First Freeze Gets Overridden
The fatal one. The policy fires for the first time, lands in a week with a launch, and leadership waves it through. Everyone now knows the freeze is advisory and it will never bind again. If an override is genuinely necessary, take the documented escalation path, record it, and pair it with a dated commitment to the deferred work -- otherwise it is not an exception to the policy, it is the repeal of it.
The Indicator Misses the Outage
Customers are complaining loudly and the SLO is green, because the indicator is measured server-side, excludes the failing path, or is computed over a window long enough to smooth away a two-hour disaster. Every time reality and the dashboard disagree, the dashboard loses credibility, and credibility is the only thing making a freeze enforceable.
The Freeze Blocks the Fix
A freeze written as "no deploys" stops the reliability work it was meant to fund, and the team routes around it within days. Write the exemptions in from the start: reliability fixes, security patches, and rollbacks ship during a freeze. What stops is feature work.
Introducing This to a Team That Has Never Had One
Do not start with the freeze. Leading with the enforcement clause guarantees that the first serious conversation about error budgets is adversarial, and the idea dies before anyone has seen it work. Start with the measurement and let the number do the persuading.
Pick One Service and One Journey
Choose a service that matters commercially and that the team genuinely owns end to end. Define a single SLI on the journey a customer would name if asked what the service does. A pilot producing one credible number beats a program producing forty disputed ones.
Measure for a Quarter With Nothing Attached
Publish the SLI, watch it, change nothing else. You are calibrating, and you are building the credibility of the measurement before anyone has a reason to attack it. Set the first objective near current performance rather than at an aspiration -- an objective missed on day one teaches everyone the number is unreachable and therefore ignorable.
Attribute the Burn Before Attaching a Policy
In every incident review, record how much budget the incident consumed and what caused it. After a quarter you have a ranked list of what is actually costing you reliability -- the most persuasive artifact you will ever bring to a debt conversation, and you have it before asking anyone to change their behavior.
Write the Policy With Product in the Room
Draft it jointly. Name the actions, exemptions, escalation path, review date, and three signatures. Start enforcement soft: a first breach means the next planning cycle prioritizes reliability items, not an immediate freeze. Escalating consequences are easier to agree to, and a policy that gets signed beats a stricter one that does not. Review it on the date you wrote down, amend it with the same three signatures, and keep it alongside your other tech debt governance policy rather than in a monitoring tool nobody opens.
Related Resources
Observability Debt
You cannot budget what you cannot measure. Fix the telemetry gaps that make an honest service level indicator impossible to compute.
DevOps Debt
Slow pipelines and manual rollbacks are usually the largest single consumer of an error budget. This is the work the policy ends up funding.
Governance Policy
An error budget policy is one clause in a wider governance model. See how pre-agreed rules beat case-by-case debt negotiation.
Measuring Tech Debt
The budget only prices debt that reaches users. These metrics cover the debt that slows developers down without showing up in an SLI.
Frequently Asked Questions
An SLI is the measurement, an SLO is the target for that measurement, and an SLA is a contract with consequences attached if the target is missed. The error budget is what is left over: the SLO is a target percentage and the error budget is one hundred percent minus the SLO. Your internal SLO should always be stricter than any SLA you have signed, so you notice and react to degradation well before it becomes a contractual event with money attached.
The Site Reliability Workbook names three parties who must approve it: the product manager, the development team, and the reliability engineers responsible for defending the objective. Product ownership of the target is the critical piece, because it makes the resulting freeze an act of product discipline rather than an engineering veto. The policy should also record its authors, reviewers, approvers, approval date, next review date, and the escalation path to use when someone disagrees.
The first override teaches everyone whether the policy is real, so handle it carefully. Route it through the documented escalation path rather than a hallway decision, write down who approved it and why, and pair it with a dated commitment to the reliability work that was deferred. An override that is visible, recorded, and paid back later is a legitimate exception. An override that happens informally and repeatedly is not an exception at all, it is the quiet repeal of the policy.
Two or three, each tied to a user journey someone outside engineering could name. A service with a dozen objectives effectively has none, because something is always breaching and the policy therefore never fires with credibility. Start with one indicator on the most important journey, prove the measurement is trusted, and add a second only when there is a real user experience the first cannot see. Adding an objective later is easy; removing one a team has organized around is not.
Yes, and in a smaller organization the mechanism is arguably more valuable, because there is no separate function whose job it is to argue for reliability. What you cannot skip is the separation of roles: the objective is set by whoever owns the product outcome, the indicator is computed by the monitoring system rather than by hand, and the policy is written down and signed. One team can hold all those roles as long as it holds them explicitly and does not quietly rewrite the target after a bad month.
It converts a preference into an attributed cost and attaches a pre-agreed consequence to it. Once every incident review records how much budget was consumed and why, the debt draining the balance becomes a ranked list rather than a complaint, and when the policy fires that list is the work. The honest limit is that this only funds debt users feel: a hostile build or an untested module burns no budget and still has to be argued for on other grounds.
Stop Negotiating. Start Budgeting.
Write the policy while everyone is calm, sign it with product in the room, and let the arithmetic decide what gets funded.