# Weekly review and closeout checklist

Leadership is keeping the truth cheap to see: review evidence while options remain, change
the plan without disguising the change, and finish the operational work after launch.

Source: lesson 0050 · Steer, Replan, and Close the Initiative.

---

## Blank — weekly initiative review, one page

Built from live sources, not from a status performance. Product, design, and engineering
answer these together.

| Field | One-line guidance |
| --- | --- |
| Outcome | Baseline, current, target, cohort, and your confidence. Numbers from the dashboard, not from memory. |
| Guardrails | Each one, with its threshold, and any segment moving the wrong way. |
| Learned | Which assumption became stronger, weaker, or false — and what decision that changes. |
| Delivered | Integrated, demonstrable behavior. Not activity, not percentages. |
| Flow | What is aging or blocked, and where the team should swarm. |
| Risk and dependency | Which signal changed, the owner, the action, whether it escalates. |
| Decisions | Made, waiting, and the dissent carried. Who owns the next evidence and when. |
| Forecast | The current range and confidence, what changed, and the smallest scope trade that preserves the outcome. |
| Ask | One explicit decision or resource, with the options and their consequences. |

### Continue / change / stop — agree these before you need them

| Choice | When |
| --- | --- |
| Continue | Outcome movement and guardrails support the hypothesis. |
| Change | The opportunity remains, but the evidence rejects this mechanism or this slice. |
| Stop | The opportunity is too small, the intervention cannot move it economically, a guardrail cannot be protected, or the strategy changed. |

Stopping after cheap evidence is capital allocation, not team failure.

### Propagating a change

When something changes, change it everywhere it appears: the frame, the story map, the
threat model, the contracts, the telemetry, the rollout plan, and the forecast. "We are still
on plan" after the plan changed is how a team trades trust for temporary comfort.

---

## Blank — closeout checklist

- [ ] Outcome and guardrails reported against baseline, including the segments that did not benefit and any cost that moved elsewhere.
- [ ] Steady-state ownership accepted **by name**: service, on-call, vendor relationship, data, support knowledge, roadmap.
- [ ] Release flags, temporary adapters, manual pilot paths, elevated access, pilot dashboards, and test data removed — or each recorded as debt with its interest and a repayment trigger.
- [ ] Architecture views and contracts updated; superseded decisions superseded, not rewritten; fitness functions still running in CI.
- [ ] Learning review completed: expected, observed, surprised, next time. At most two improvements, each owned and dated.
- [ ] Continue, change, stop, or hand over decided explicitly and communicated.

---

## Completed — week 3, 11 September

Week 3 is the week the evidence arrived: three results within four days of each other. One
changed an architecture decision, one changed the slice order, and one refused to say anything
at all. The review took thirty minutes and was built from live sources.

| Field | Entry |
| --- | --- |
| Outcome | **Not yet measurable** — no customer is exposed. The first reading is due at the 1% stage, forecast 26 September to 3 October. Reporting "on track" here would be a claim with no source. |
| Guardrails | One moved: the manual-queue projection is now **34 referrals/day** at full exposure — 68% of the 50/day cap — against a week-1 estimate of "about 12". The others are unchanged and unmeasured. |
| Learned | (1) The carrier is far slower and less safe under retry than its documentation says (A3, falsified). (2) The referral volume implied by the A1 sample is nearly three times the week-1 estimate. (3) Customer comprehension is still unknown; the n = 5 test was **inconclusive**, not positive. |
| Delivered | The walking skeleton runs end to end from a clean build, including the worker process and correlation ids across it. The 200-case policy suite runs and 186 of 200 pass; all 14 failures are gift orders, which is a missing rule rather than a defect. Demonstrable Friday. |
| Flow | Two slices in progress, at the WIP limit. The appeal flow has not moved in three days because Marco is on the comprehension retest. It is the item the swarm rule applies to — and the item the scope trade below removes. |
| Risk and dependency | R3 (carrier latency) closes today, replaced by a smaller operational risk: worker backlog age. Payments confirmed `return_id` as the refund idempotency key in writing on 8 September. Privacy has one open finding — log retention is 3 days, not the 30 the review assumed. |
| Decisions | Two made today, below. One waiting: whether the gift-order rule enters slice 3 or is refused for the pilot — Priya, by 18 September, needs the volume from Anya. |
| Forecast | Pilot open to 1% between 26 September and 3 October, about 70% confidence, **down from about 80%** last week because the worker is six days of work that was not in last week's plan. Today's scope trade restores about five of them. |
| Ask | One decision from Ola by Monday on log retention, because the audit trail's design depends on it. No staffing change requested. |

### The three pieces of evidence, and what each one did

**Evidence 1 · the carrier spike — changed an architecture decision.** p50 640 ms, p95 1.9 s,
p99 11.4 s, and two duplicate labels in ten timeout-retries made without a client reference.
This falsified A3 and, with it, ADR 0002, which had said the carrier call belongs inside the
request: the 800 ms budget is unreachable at p99 11.4 s, and a 30-minute outage would have
refused every eligible request. **ADR 0002 is superseded by ADR 0003**, signed the following
day. Sam's dissent is recorded in the ADR with the condition that would prove it right.
Artifacts updated the same day: the ADR log, the container view (worker process), a new
dynamic view for the timeout path, the carrier-failure acceptance example, the delivery map
(six days added), the rollout contract (the disable flag and its rehearsal), and risk R3
closed.

**Evidence 2 · the referral volume model — changed the slice order.** Applying the A1 sample's
63/26/11 decision mix to the demand extract gives 34 referrals/day at full exposure, rising to
about 41/day in the seasonal ramp, against a week-1 assumption of "about 12". It falsified
nothing — the guardrail still holds at 41 against 50 — but the headroom is nine requests a
day, not thirty-eight. Slice 2 becomes the agent decision view and the four-hour referral SLA;
the in-product appeal flow moves behind it. **This is exactly the scope trade the week-1
forecast pre-identified, taken now on evidence rather than later under deadline pressure.**
Marco dissented: "Refusal is where the contact rate lives. Sending a refused customer to an
agent to appeal is the exact behavior the outcome measures." Priya owns scope and took the
trade. The threshold that would prove Marco right — contact rate inside the refused segment
above 55% at the 1% stage — was instrumented before the stage opened. A measure was added to
the telemetry map specifically to adjudicate a dissent.

**Evidence 3 · the comprehension test at n = 5 — changed nothing, on purpose.** Two of five
participants read "your label is being prepared" as a refusal. The honest reading is
*inconclusive*: two of five is a strong-sounding fraction of a sample too small to separate a
wording problem from two participants. The wording changed anyway — "Accepted. We're preparing
your label; we'll email it within the hour" — because it is a one-line, fully reversible change
that costs nothing if the finding was noise, and a retest at n = 12 was commissioned. The
distinction is worth naming: changing something cheap and reversible on weak evidence is fine;
changing something expensive and irreversible on weak evidence is not.

Three results, three different responses: one decision reversed, one re-ordered, one
deliberately not made. A review in which every piece of evidence conveniently supports the
existing plan is a review that is being performed rather than run.

---

## Completed — closeout, after a pilot from 29 September to 24 November

### Outcome and guardrails, with sources

| Measure | Baseline | Target | Actual | Verdict |
| --- | --- | --- | --- | --- |
| Median submission → decision communicated | 72 h | under 4 h | **3.1 h** exposed; 69 h in the holdback control. n = 6,480 exposed, 694 control | Met |
| Support-contact rate within 7 days | 41% | 31% or lower | **31.6%** exposed; 40.2% control — a 23% relative reduction | **Missed by 0.6 points.** Reported as a miss, not rounded into a win |
| Fraud loss per 1,000 eligible returns | 2.4 | 2.4 or lower | 2.2 exposed (14 events in 6,480) | Held on the agreed measure — with the caveat below |
| Manual review queue | — | 50/day or fewer | Mean 26/day; peak 41/day on 11 November | Held, with nine a day of headroom at peak |
| WCAG 2.2 AA on four critical flows | — | No open AA issue | Three found at the staff stage, all fixed before the 1% stage; zero open at the external audit. Two of the three were found by manual keyboard testing, not by the automated check | Held |
| Carrier cost per resolved return | $3.42 | $3.42 or lower | $3.39 — lower because the manual process used to buy and void about 3% of its labels, and the idempotent path bought none twice | Held |
| Personal data retained beyond 30 days | — | Zero records | Zero, verified by an independent query | Held |
| Duplicate carrier labels | — | Zero | Zero in 6,480 returns | Held |

**The weakest number in this packet, named.** The fraud guardrail's control cohort contains
694 requests. At a base rate near 2.4 per 1,000 that is fewer than two expected events, so the
control cannot detect a fraud change of any plausible size. What can honestly be said is the
absolute exposed figure against the baseline. The guardrail held on the measure agreed in
advance, and **the measure itself was under-powered** — a defect in the evidence, not in the
system, and the first thing the steady-state owner should fix.

### Who did not benefit, and what moved elsewhere

| Segment or cost | What happened | Decision |
| --- | --- | --- |
| Guest orders — 29% of eligible demand | Untouched. The headline result covers 71% of the demand it appears to describe, and the packet says so everywhere the outcome is quoted | Sized; decided on 18 November to build in Q1 with a separate fraud model, not folded into this initiative |
| Refused customers — 11% of requests | Contact rate 47%, above the 41% overall baseline. Being refused quickly is still being refused | Below Marco's 55% dissent threshold, so the appeal flow was not pulled forward — but the number is now on slice 3's exit condition |
| Referred customers — 26% of requests | Contact rate 38%; median resolution 3.7 h, inside the SLA but carrying almost all the variance | Kept. The referral state is what made the automatic path safe enough to expose |
| Mobile customers | 11 points below desktop in the first exposed week; 4 points below by week 8 after two rounds of wording and layout work | Closed as a gap, retained as a monitored segment |
| Support operations | Front-line contact volume fell; specialist referral work rose to 26/day mean. **Work moved rather than disappearing** | Two agents rostered permanently for referrals, agreed at handover |

### Ownership transferred, by name

| Asset | To | Evidence of readiness |
| --- | --- | --- |
| Service, on-call, alerts, dashboards | Kim (platform services) | Kim's team ran the carrier-outage drill unaided on 20 November and disabled labels in 80 seconds |
| Referral queue, SLA, support runbook | Nils, with two rostered agents | Roster published; the SLA met on 97% of referrals in the final fortnight |
| Eligibility policy content and the 200-case regression suite | Anya, with Kim's team executing changes | Anya released one policy change unaided on 17 November; elapsed time four hours — driver D4's claim, measured |
| Carrier vendor relationship and the sandbox key | Asha (platform) | Key rotated into the vault; the rotation runbook tested |
| Outcome dashboard and the holdback experiment | Analytics, with Priya as reader | Queries in the dashboard repository; holdback released 24 November |
| Retention job and log-redaction fitness function | Kim, with Ola auditing quarterly | Independent retention query run at handover |

### Temporary machinery

| Item | Disposition |
| --- | --- |
| The release flag and its fallback branch | **Deleted**, along with both configurations' tests |
| Exposure percentage and holdback machinery | **Deleted.** A holdback nobody is reading is a cohort of customers receiving a worse product for no reason |
| The label-attempt disable flag | **Kept** — it is an operational control, not release machinery. Moved into the runbook, its off-state tested quarterly |
| Elevated database access for three engineers | Revoked |
| Pilot dashboards and the staff test cohort | Deleted; the four steady-state panels moved to Kim's dashboard |
| Manual warehouse receipt confirmation | **Debt, recorded.** Its removal trigger — 14 consecutive days under 0.1% late receipt events — fired on 9 November and was not acted on. Interest: about 40 clerk actions a day, roughly 1.5 hours of someone's day, indefinitely. Trigger to pay it down: the next returns initiative, or the first week the clerk's queue delays a refund past 24 hours. Owner Kim, reviewed quarterly |
| The old email-based return path | **Kept deliberately** as the carrier-outage fallback, documented as a fallback rather than left running as a forgotten alternative |

### Decision hygiene

ADR 0002 remains superseded, not deleted: a newcomer can still read what the team believed on
1 September and why it stopped being true. ADR 0005 (manual receipt) is **not** superseded,
because the decision still stands — the debt entry records that its removal trigger fired and
was not acted on, and superseding it would have hidden the debt. Context and container views
were updated. All four fitness functions remain in the build after handover: the import rules,
log redaction, the "nothing writes refunded" check, and the 200-case policy suite. They are
the part of the packet that keeps working when everyone who wrote it has moved on.

### Learning review

| Expected | Observed | Surprised |
| --- | --- | --- |
| Speed would drive the contact reduction | It did not, alone. The drop tracked proactive status more than raw speed — the referred segment resolves in 3.7 h and still contacts at 38% | Telling people what is happening is worth more than shaving hours. Would have changed the slice order in week 1 |
| Policy would be about 90% codifiable | 63% in pilot scope. The gap was not exceptions but undocumented rules held by two people | The sample cost two analyst-days and reshaped the outcome statement. It should have been the first thing done, not the third |
| The carrier would be the easy integration | It was the decision that reshaped the architecture | Vendor documentation was wrong by an order of magnitude at p99, and unsafe under retry. Measure vendors before designing around them |
| The disable flag would work because it existed | It half-worked, and only a rehearsal revealed it | An untested control is a belief. Thirty minutes found a defect that would otherwise have surfaced at 02:00 |

**Two improvements owned, not twelve.** (1) Every future initiative buys a domain sample
before writing an outcome statement — owner Priya, applied to the Q1 guest-order initiative.
(2) Every guardrail gets a power check at design time: if the agreed measure cannot detect the
change it is meant to catch, it is not a guardrail — owner Risk, due before the Q1
initiative's first exposure.

### The explicit decision

**Hand over and continue in a new initiative.** The returns capability moved to steady-state
ownership on 24 November. Slice 3 and guest orders become a separate Q1 initiative with their
own frame, because they need a different fraud model and a different cohort — not because they
are "phase two". Continuing under this frame would have let the outcome statement drift to
cover work it was never evidence for, which is how initiatives become permanent programs with
unfalsifiable goals. And not stopping, because both measures moved substantially against a
held-back control and every guardrail held: the contact-rate miss is 0.6 points on a measure
that moved 9.4 points.

**What the next lead is told:** read ADR 0003 before touching the label path; the fraud
guardrail is under-powered, so fix the measure before widening the cohort; the manual-receipt
debt has a trigger that has already fired; and the outcome statement covers 71% of demand —
quote it with that clause attached or you will mislead someone.

**Traces to:** the outcome and every guardrail in `initiative-frame.md`, each reported against
its own baseline; assumptions A1, A2, A3, and A6 in `assumption-map.md`, with what each turned
out to be worth; ADR 0003's accepted costs and Sam's dissent, both revisited; the flags and
cohorts from `rollout-contract.md`, now removed.