{
  "uid": "cogitave.learn.reliability-and-sre.operating-for-reliability",
  "kind": "moduleUnit",
  "href": "/modules/reliability-and-sre/operating-for-reliability/",
  "title": "Operating for reliability",
  "summary": "",
  "type": null,
  "products": [],
  "roles": [],
  "levels": [],
  "subjects": [],
  "headings": [
    "The error-budget policy: the rule with teeth",
    "Sustainable on-call and the toil cap",
    "Capacity, chaos, and the SLA it backs"
  ],
  "source": "An SLO with no consequence is a dashboard. The **error-budget policy** - section 3\nof the [reliability standard](../../../../standards/docs/standards/reliability.md),\nrecorded as a governance decision in\n[ADR-0020](../../../../standards/docs/decisions/0020-reliability-and-secure-sdlc.md)\n- is what gives the number teeth, agreed **in advance, in calm**, so the decision\nat the moment of breach is mechanical, not political.\n\n## The error-budget policy: the rule with teeth\n\nEvery Tier-1 and Tier-2 service must have a written policy, `slo.yaml` plus a\npolicy note, encoding this governing rule:\n\n> **Budget healthy** -> normal velocity, ship freely behind the deployment gates.\n> **Budget exhausted** (or projected to exhaust before the window closes) ->\n> **per-service feature freeze**: only reliability fixes, rollbacks, and\n> security/critical patches deploy; reliability work goes to the top of the\n> backlog until the budget recovers. **Repeated breach** -> escalate: re-tune the\n> SLO, or fund a reliability epic.\n\nThe freeze is **enforced in the pipeline** - the same protected-environment gate\nthat governs deploys, reused as the budget freeze - not by good intentions, and it\nis **per-service**: one service's blown budget never freezes an unrelated one.\nOne named **reliability champion** owns the weekly freeze/thaw call; product owns\nthe SLA but cannot override a freeze by fiat, and a genuine emergency uses the\nexisting logged, time-boxed break-glass path, never a silent bypass. Security and\ncritical-severity fixes are always exempt.\n\n## Sustainable on-call and the toil cap\n\nWhat keeps a service inside its budget between incidents is the operating model\naround it. On-call runs **primary and secondary** on a bounded **1-week**\nrotation with at least two people in the pool, escalating on a timed no-ack to the\nsecondary and then to the incident-commander path. Pages are scoped to\n**user-facing SLO symptoms only**, time off is owed after a night page, and a\nrotation that exceeds a sustainable page count is itself a **reliability defect**\nto fix, not a person to toughen up.\n\n**Toil** - manual, repetitive, automatable operational work - is capped at\n**~50%** of reliability time; the rule-of-three (a task done manually three times)\nproduces an automation ticket, not a fourth manual repeat.\n\n## Capacity, chaos, and the SLA it backs\n\nTier-1 services carry a load model with **20-30% headroom** and degrade\ngracefully under overload rather than collapse. Resilience is **tested, not\nassumed**: staging-first, hypothesis-driven chaos experiments and quarterly\ngame-days validate that a failure really does degrade gracefully and that the\nrunbook actually works - reusing the same DR game-days\n[business continuity](../../../../ops/business-continuity/README.md) already runs.\n\nFinally, a customer-facing **SLA must be strictly looser than the SLO** that backs\nit. That gap is the operating margin that lets the team react to a burn before\nthe customer-facing promise breaks - one of exactly two rules in the standard\nthat may only be strengthened, never relaxed.\n",
  "partOf": "cogitave.learn.reliability-and-sre",
  "durationInMinutes": 9,
  "quiz": null
}