Lesson 0049 · Initiative Lead · Module 3
Test, Observe, and Roll Out Safely
Done is not “merged,” and launch is not one event. A responsible design includes the evidence, controls, and ownership needed to learn in production without betting every user.
Mission tie-in: testability and operability are architecture qualities. If a team cannot see, limit, or reverse behavior, the system is not finished enough to release.
Knowledge: build a release as a controlled experiment
Test according to risk, not a universal pyramid quota. For returns:
- Domain examples prove policy rules, boundaries, refusal reasons, and state transitions quickly.
- Integration and contract tests prove persistence, order and payment contracts, carrier translation, timeout, and idempotency.
- A few end-to-end journeys prove wiring, accessibility, authorization, analytics, and deployment.
- Quality tests exercise the named drivers: p95 latency, carrier recovery, access isolation, data deletion, and alert response.
- Exploratory testing looks for surprising behavior the examples did not imagine.
Map telemetry to both outcome and system health. A green error dashboard does not prove returns resolve faster. Track resolution time and support-contact rate by cohort alongside request rate, decision/refusal distribution, carrier latency, label retries, stuck state age, queue depth, and manual workload. Segment enough to expose who is harmed, while protecting privacy and sample size.
| Stage | Entry | Watch / duration | Advance or reverse |
|---|---|---|---|
| team only | tests pass; support path exists | all flows; two days | no critical security or accessibility defect |
| staff accounts | runbook exercised; alerts route | errors, retries, comprehension; 50 cases | zero duplicates; p95 within target |
| 1% eligible | support briefed; rollback tested | outcome and guardrails; three days | improvement visible; fraud not worse |
| 10% then 50% | prior gate signed off | capacity, segments, manual queue | all thresholds hold at each stage |
| 100% | operational owner accepts service | SLO and outcome continuously | flag removal has owner and date |
Staged rollout is usually explained as a smaller blast radius, and that is the least interesting half. The real mechanism is that exposure makes the population a variable you control while everything else is held constant, so a difference in the measure can be attributed rather than argued about. Every release is a hypothesis test with an unknown effect size: the exposure percentage buys statistical power, and the duration buys the slow failure modes—memory growth, queue drift, nightly batches, weekly cycles—that an instant jump to 100% cannot reveal at any percentage. That is why a gate needs a minimum observation window as well as a threshold. A stage that advances in ten minutes because “it looks fine” has measured nothing but the deploy.
Separating deployment from release works on a second mechanism: it decouples code risk from behavior risk. The code goes out cold on the release train; the behavior is enabled by configuration. Reversal then costs seconds instead of a deployment, which matters concretely here—the order platform's train runs twice a week, so a code rollback was never a real recovery plan.
Canaries stop earning their cost when the population is too small to move the measure. At the pilot's 160 requests a day, a 1% stage is under two requests a day. That is why the staff stage above is sized in cases rather than percent, and why the 1% stage carries a three-day floor and gates on things a handful of cases can still falsify—a duplicate label, a fraud signal, a refusal nobody understands—rather than on a median that needs volume to mean anything. They also stop applying when the change has no user-visible surface and no runtime cost, and when partial exposure is itself the hazard—a data migration, a price change that must be simultaneous, a legal cutover with a fixed date.
Separate deployment from release. Deploy dormant behavior behind a short-lived release flag, validate it internally, then expose cohorts gradually. The flag needs an owner, removal date, safe off-state, and tests for the live and fallback configurations. Flags reduce release risk while adding state-space and cleanup cost.
Define rollback before rollout: who decides, what signal triggers it, how long it takes, what happens to data already written, and how customers in flight recover. A code rollback cannot undo an email sent, refund issued, or schema contracted. Some effects need roll-forward or compensation.
Operational readiness is part of “done”: dashboards, actionable alerts, structured events with correlation ids, runbook, support script, ownership/on-call, capacity assumption, data recovery, known failure states, and an exercised disable path. Run a 30-minute failure drill before the pilot: carrier returns timeouts, worker backlog grows, one request is duplicated. Can the team detect, diagnose, limit, recover, and explain?
How the lead actually ran the drill
Week 11, thirty minutes, whole team, one scenario: at 14:05 the carrier starts returning timeouts. The lead's opening question is not about dashboards. It is who turns this off, and how long is it between noticing and off? Everything else in the drill is downstream of that sentence.
The alert fires at 14:11—label failures above 2% for five minutes, as designed. Dana finds it. The disable is a configuration flag, ninety seconds, no deploy. So far the plan holds. Then the drill produces its actual finding, which nobody had anticipated on paper: sixty-one requests have already been accepted with labels pending, and turning off label creation does not un-accept them. The customers who are mid-journey are now in a state the “off” switch does not address.
Karl proposes the simplest fix and is nearly right: a boolean kill switch, off means off. The lead rejects it because the boolean answers the wrong question—it protects the system and abandons the sixty-one people. The flag gets three positions instead of two: on, accept but hold (keep taking eligible requests, queue the label work, tell the customer honestly), and refuse new (stop accepting, drain what exists). Karl's objection to that is legitimate and is answered rather than overruled: three states do not triple the test matrix if you test each state's contract rather than every combination of state and path.
The consequence lands in week 12, during the real twenty-two-minute carrier outage. The team used accept but hold. No requests were lost, the outbox drained after recovery, and the honest cost was four support contacts from customers who did not understand the pending state. A boolean switch would have refused sixty-one eligible customers to protect a metric—and the dashboard would have looked better for it, which is exactly the trap a drill exists to find.
| Claim | What is absent | Required evidence |
|---|---|---|
| “Tests are green.” | Production behavior and outcome. | Telemetry plus staged exposure. |
| “We can roll back.” | Data and external side effects. | Rehearsed reversal or compensation. |
| “There is a dashboard.” | An owner and decision threshold. | Actionable alert and runbook. |
| “The flag makes it safe.” | Fallback tests and deletion. | Owner, off-state, removal date. |
- A canary means one percent.
- A canary is a population small enough to limit harm and large enough to measure. At low volume that is a case count and a duration; a percentage chosen because it is conventional will either expose nobody or teach nothing.
- Rollback undoes the release.
- It undoes the code. Emails sent, refunds issued, rows written, schemas contracted and vendor calls made are not undone. The reversal plan has to name each irreversible effect and say whether it is compensated, rolled forward, or accepted.
- Observability means dashboards.
- A dashboard nobody is required to look at is decoration. The unit that changes behavior is an alert with a threshold, an owner, and a documented response—plus the ability to answer a question you had not thought of before the incident.
- More tests make a release safer.
- Coverage of the wrong risk buys expensive confidence. The only useful question about a test is which failure it would have caught; a suite that cannot answer that for its top three risks is long, not strong.
When this is the wrong move
A staged rollout is overkill for an internal tool with twelve users: tell the twelve, ship to all of them, watch, and revert—the coordination cost of stages exceeds the harm they would prevent. It is impossible, not merely expensive, for a simultaneous cutover where two populations on different behavior would be the incident. A failure drill for a change with no external dependency and no persistent side effect is theatre; run one when there is a dependency that can fail, a state that survives the failure, or an authority question nobody has answered. And a feature flag on a change that will be fully rolled out within a day is usually a net cost—you pay for the state space, the tests, and the removal, and buy back very little.
Skill: design the evidence and control
A healthy error rate proves:
A rollout stage needs which four things?
Why rehearse rollback before release?
Practice: write the rollout contract
Choose one upcoming release. Map tests to its top risks and architecture drivers. Define outcome, guardrail, and health telemetry with owners. Write exposure stages with entry, minimum observation, advance threshold, stop threshold, and reversal. Rehearse one dependency failure and one data-side-effect rollback.
In the lab and the capstone. The lab's phase-4 checkpoint asks you to add outcome telemetry to the lab system and then express one rollout gate as an automated test, so “are we allowed to advance?” has an answer that does not depend on who is in the room. The worked artifact is the rollout contract in the strong capstone packet—stages in case counts, a stop threshold beside every advance threshold, and a named reversal for each irreversible effect. Compare with the weak version: a launch date, a percentage ramp with no observation window, and “roll back if there are problems”.
Reveal: the “done” test
The slice is done when it can be deployed, observed, supported, limited, and recovered—not only used on the happy path. The team can name who watches it, which threshold changes exposure, and what happens to every in-flight state if behavior is disabled.
- In two sentences, explain to a new lead why a rollout stage needs a minimum duration and not only a threshold.
- Take your next release. Name its irreversible effects, and say for each one whether the plan compensates, rolls forward, or simply accepts it.
- Predict the failure: if a team ships to 100% behind green dashboards, what is the first thing a customer notices that no alert covers, and how long until anyone connects the two?
Your win
You can build risk-shaped tests, pair product outcomes with operational telemetry, and release through explicit gates with a rehearsed response to failure.
Read and watch deeper
- Canarying Releases, Google SRE Workbook — partial, time-limited exposure evaluated against control.
- Feature Toggles, Pete Hodgson — release versus deploy, flag categories, testing, and carrying cost.
- Release It!, Nygard — stability, production readiness, and designing for operational reality.
- Service Level Objectives, Google SRE Book — choosing indicators and targets that a gate can actually be written against.
- Reference: Initiative Playbook · readiness and rollout contract.
Ask your agent-teacher to run a tabletop incident against the plan; make it track customer impact, data side effects, authority, and recovery rather than only server errors.