# Phase 4 — reference solution

Code in this directory: `p4_telemetry.py`, `p4_rollout_gate.py`, and
`ADR-0001-expand-contract.md`. Verified by `verify_phase_4.py` (12 tests).

---

## 4.1 Correlation ids

`new_correlation_id()` is four lines. The work is everywhere else: the id is
generated at the edge if the caller did not supply one, lives on
`RequestContext` (`../phase-2/p2_composition_root.py`), and is written to every
log line, every audit row, every notification record, every domain event
(`../phase-2/p2_events.py` puts `correlation_id` on the `DomainEvent` base) and
the response envelope.

**Forces.** A field in every table, a parameter in every signature, and noise
in every log line — paid at construction. The benefit arrives during an
incident, which is exactly why teams skip it and exactly why they regret it.

**Consequence accepted.** Accepting an inbound id from the caller means a
client can send you a duplicate or a hostile value. Validate the shape, do not
trust it for authorization, and record both the inbound id and your own.

**A second valid solution — when you would choose it.** Full distributed
tracing (OpenTelemetry) instead of a hand-rolled id. Choose it the moment there
is more than one process to correlate across, or when you already run a
collector. For a modular monolith with an in-process worker, a correlation id
on a structured log line answers the same questions for none of the
operational cost.

---

## 4.2 Outcome telemetry → `p4_telemetry.py`

Four outcome numbers, each paired with a system-health number:

| Outcome | Paired health metric | Why the pair |
|---|---|---|
| median and p95 resolution hours | carrier failure rate | a fast median with a broken carrier means you resolved by denying |
| automatic resolution rate | notification duplicate rate | automation that spams is not automation |
| peak review-queue depth | — | it is the promise to the support team |
| **unresolved count** | — | the most important number on the dashboard |

**Definitions, written down first.** A return resolves when it reaches a
terminal state — refunded, denied, or cancelled. A held return does *not*
resolve when an agent opens it; it resolves when the agent's decision lands.
`RESOLVED_STATES` names them, and `hours_to_resolve` returns `None` for
everything else so that unresolved returns are counted rather than averaged
away.

**Percentiles, not averages.** A median of 4 hours with a p95 of 40 is a
different product from a median of 4 with a p95 of 6, and an average hides
both.

**The number the lab's counters cannot produce.** `unresolved_count()` catches
the return the carrier outage stranded in `AWAITING_LABEL`. The lab's seven
counters all go up and none of them can tell you a customer is waiting forever.

**Forces.** Outcome metrics lag and are noisy — you learn the median a week
after the release that moved it. System metrics are immediate and can all be
green while the product fails. You need both, and you must say in advance which
one you would roll back on.

---

## 4.3 The rollout gate → `p4_rollout_gate.py`

A pure function of a snapshot, returning every breach with the distance by
which it was missed, so the failure message *is* the decision.

**The conditions and who each one is a promise to:**

| Condition | Promise to |
|---|---|
| p95 resolution under 8h | the customer |
| automatic resolution over 70% | the business case |
| review queue under 50/day | the support team |
| carrier failure rate under 2% | operations |
| zero duplicate refunds | finance |
| zero unresolved returns | the customer, again |

**Two things the reference does on purpose.**

- An **empty window fails**. "No data" is not a pass. A gate that greenlights a
  release because nothing happened is worse than no gate.
- One guardrail is **not about the feature**: the review-queue cap is a promise
  to another team, and breaching it is a rollback reason even when resolution
  time looks excellent.

**Forces.** A strict gate blocks good releases on noisy data; a loose one lets
a regression through. Calibrate by running the gate against historical windows
before you trust it, and state the blast radius you accept between two
evaluations — staged rollout narrows it, a big-bang release turns the gate into
a post-mortem.

**The reversal.** A gate without one is a wish. Here: a flag on automatic
approval that routes everything to the agent queue instead. Note what it does
to returns already in flight — they keep their current state, and the queue
absorbs the new ones, which is why the 50/day cap is part of the gate.

---

## 4.4 The ADR → `ADR-0001-expand-contract.md`

A worked example for the expand/contract decision.

**What makes it useful rather than ceremonial:**

- The rejected option (a maintenance window) is one a reasonable person would
  choose, and the reason for rejecting it is a *constraint* — finance's release
  calendar — not a preference.
- A third option (patch the conversion, keep the column) is recorded as the
  fallback rather than dismissed, because it is genuinely the right answer under
  time pressure.
- The consequences include the bad ones: four deploys, three weeks of two
  columns that can disagree, a dual-write path someone must remember to delete,
  and a dependency on a team outside our control.
- The revisit trigger is observable and dated: 60 days after step 3, or the
  discovery of a second external consumer.

**Forces.** ADRs cost twenty minutes and are only valuable if someone reads
them later. One per decision produces a graveyard; one per *irreversible or
expensive-to-reverse* decision produces a record. Lesson 0046's rule — scale
review effort to reversibility — applies to the record as much as the decision.