Assessment · Phase III
Phase 3 assessment · Master Designer
Twelve trade-offs, and no best answers. Every question below has at least two defensible responses; what is being assessed is whether you name the forces, show the arithmetic, and say what you are giving up. Lessons 0027–0038.
The scenario questions still have one best answer, because they test whether you can read a force from a situation. The written tasks do not. A reviewer scoring Task 1 should be able to disagree with your choice and still score it Lead-ready — if the alternatives are real, the numbers are shown, and the accepted cost is stated. That is the standard from here on.
Part 1 · Six scenarios
A vendor charges €1,400 a month for address validation. Building the equivalent is estimated at six weeks of one engineer, fully loaded at €9,000 a month, and €600 a month afterwards to run and maintain.
Ignoring risk entirely, building breaks even:
Two teams have near-identical 30-line CSV exporters. They differ in one column header and one
date format. Somebody proposes a shared export_csv(config) with nine options.
The strongest argument against extracting the shared exporter now is:
You add a setting so one customer can change a default. The cost accepted is:
An eligibility decision must be shown to the customer within 800 ms. It needs a fraud score from a provider that answers in 120 ms at p95 and 2.5 s at p99.9.
The design that keeps the 800 ms driver honest is:
Splitting returns into its own service would let the returns team deploy independently. Every eligibility decision would then make a synchronous call to the order service.
The trade this split actually makes is:
In an expand and contract migration, the first irreversible moment is:
Part 2 · Three written tasks
Twenty-five minutes each. These are longer than Phase II’s because a trade-off answer without its arithmetic is an opinion.
Task 1 · The option table with a break-even year
What to write
The returns pilot needs shipping labels. Three genuinely different options are on the table, with the numbers the team has:
- Direct integration with the one contracted carrier. Estimated 3 weeks of one engineer; no licence; roughly 2 days a quarter of maintenance when the carrier changes its API.
- A multi-carrier aggregator at €0.11 per label plus €450 a month. One week to integrate. Pilot volume is 900 labels a month, forecast 6,000 a month at full rollout.
- Manual labels generated by an agent from the carrier’s own web portal. No build. Around 4 minutes per return; the manual queue guardrail is 50 a day.
Assume a fully loaded engineer cost of €9,000 a month. In about twenty-five minutes write the option table (what each buys, what each costs, what evidence would decide it), the break-even arithmetic with its numbers shown, the option you choose for the pilot, the cost you accept, and the measurable trigger that reopens the decision.
Self-review — answer these before you open the reveal
- Show your arithmetic. If the break-even year is a feeling rather than a division, the table has no spine.
- Did you compute break-even at pilot volume and at rollout volume? They point in opposite directions here, and that is the whole decision.
- Which option did you choose for the pilot, and is it the same one you would choose for the rollout? If it is not, say what changes and when.
- Name the cost you accepted, in a sentence a non-engineer would understand and could object to.
- Is your revisit trigger a number with a threshold and an owner, or is it “when we outgrow it”?
- Did the manual option get a fair reading? It has the shortest path to evidence and the lowest reversal cost, and it is the one most often dismissed on aesthetics.
Now score what you actually wrote, not what you meant. Each dimension links to its level descriptions.
Quality of alternatives and trade-off reasoning
Use of measurable architecture drivers
Delivery sequencing and uncertainty reduction
Technical communication and decision clarity
Not scored yet.
Reveal: a strong answer, a weak one, and the difference
Strong
Option table.
| Option | Buys | Costs | Evidence that decides it |
|---|---|---|---|
| Direct carrier integration | lowest marginal cost; no third party in the label path | ~€6,750 to build; ~€900 a quarter maintenance; one carrier only | whether a second carrier is on the roadmap inside 18 months |
| Aggregator | fastest to working labels; second carrier is configuration, not a project | €549 a month at pilot volume, €1,110 at rollout; an availability dependency | the aggregator’s measured p95 and its incident history |
| Manual | zero build; the fastest route to real policy evidence | 60 hours of agent time a month at pilot volume; does not survive rollout | whether daily volume stays under the 50-a-day queue guardrail |
Break-even, shown. Build cost: 3 weeks is 0.75 months at €9,000, so €6,750, plus roughly €900 a quarter (€300 a month) of maintenance. Aggregator at pilot volume: 900 × €0.11 + €450 = €549 a month, so the direct integration saves €249 a month and repays €6,750 in about 27 months. At rollout volume: 6,000 × €0.11 + €450 = €1,110, saving €810 a month, so break-even is about 8 months. The ranking flips with volume, which is the actual finding.
Choice for the pilot: manual, for the first two weeks, then the aggregator. Pilot volume is 900 labels a month, roughly 30 a day, inside the 50-a-day guardrail with headroom. Two weeks of manual labels costs about 30 hours of agent time and buys the thing no option can buy later: real refusal rates, real exception cases, and the actual distribution of reasons customers give. Then the aggregator, because at pilot volume the direct build takes 27 months to repay and the pilot may not survive that long.
Cost accepted. We are paying roughly €250 a month more than the cheapest long-run option, for a year, in exchange for three weeks of engineering time going into the outcome instead of into a carrier API. If the pilot is cancelled, we have spent €6,600 rather than €6,750 plus three weeks.
Revisit trigger. Reopen when monthly label volume exceeds 3,000 for two consecutive months — break-even at that point is under 15 months — or when the aggregator misses its availability scenario twice in a quarter. Owner: the returns tech lead, checked at the monthly review. Both are numbers on a dashboard somebody already looks at.
What the money model leaves out, said out loud: the aggregator adds a third party to the label path, so a carrier outage and an aggregator outage are now two independent ways to fail, and the on-call runbook needs both. That is not in the €249.
Weak — and plausible
I would build the direct integration. Building it costs about €6,750 once, while the aggregator costs €549 every month, so the build pays for itself in a year and after that it is free. We also avoid vendor lock-in and keep full control of the label process, which matters because shipping labels are core to the returns product. The manual option is not a real option because it does not scale.
“Pays for itself in a year” is off by more than a factor of two, because it compares the build cost against the whole licence rather than against the difference, and drops the €300 a month of maintenance the same paragraph had just been told about. That is the ordinary way build-versus-buy goes wrong: the recurring cost of the thing you build is counted at zero.
“After that it is free” is the second error, and the more expensive one. Nothing you own is free. Somebody carries the pager for it, and somebody spends two days a quarter when the carrier changes its API — which the brief stated and the answer ignored.
“Core to the product” is doing work it has not earned. Labels are core to the outcome and not differentiating: no customer chooses this retailer because of who prints the barcode. Lesson 0027’s question is whether the capability is where you compete, not whether it is on the critical path.
Dismissing manual is the biggest miss. At 30 labels a day it fits inside the stated guardrail, and it is the only option that produces policy evidence in week one. “It does not scale” is true and irrelevant for a two-week evidence step.
Task 2 · The operation ledger
What to write
The returns flow has these operations. For each one, write the two cost sentences — a stale answer costs… and a failed or slow answer costs… — then the lean (consistency or availability), and where staleness is served, exactly how the customer is told.
- Read: is this order eligible for return? (needs the order’s date and status)
- Read: the customer’s list of open returns, shown on their account page
- Write: create the return and reserve the refund amount
- Read: the refund status shown after the item is scanned at the warehouse
- Write: request the carrier label
Then answer: which one of these five needs an idempotency key, and what does the key have to be derived from? About twenty-five minutes.
Self-review — answer these before you open the reveal
- Are your two cost sentences actually different for each operation? If three rows have the same pair, you have written a policy, not a ledger.
- Where you leaned availability, what does the customer see? “As of two minutes ago” is an answer; a number with no timestamp is a promise the system cannot keep.
- Where you leaned consistency, what happens during a partition — an error, a queue, or a wait? Say which, because they feel very different to a customer.
- Look at operations 3 and 5. Are they one decision or two? What breaks if the label succeeds and the return record does not?
- Your idempotency key: is it generated by the client and reused across retries, or minted fresh each attempt? A fresh key per attempt buys nothing.
- Did any row lean consistency purely because the word “refund” appears in it? Money is a good reason and not automatically the deciding one.
Now score what you actually wrote, not what you meant. Each dimension links to its level descriptions.
Data ownership, contract evolution, and failure semantics
Quality of alternatives and trade-off reasoning
Use of measurable architecture drivers
Not scored yet.
Reveal: a strong answer, a weak one, and the difference
Strong
1. Eligibility read — consistency. Stale costs: an order returned after the
window because the read saw an older placed_at, or worse, a refused return that was
eligible, which the customer experiences as the product being wrong. Failure costs: the customer
cannot start a return, which is the outcome the whole initiative is measured on. Both are expensive,
so this is where the 800 ms budget and a fallback earn their keep: read from the order
system’s authoritative store, timeout at 400 ms, and on timeout route to manual review
rather than guessing. Leaning consistency here is a choice paid for with a manual queue.
2. Open-returns list — availability. Stale costs: a return the customer created 20 seconds ago is missing from the list, and they create it again. Failure costs: the account page is blank. The list is a summary, not a decision, so serve it from a replica and label it — but the “they create it again” consequence is why operation 3 needs the idempotency key, and the two decisions are connected.
3. Create return and reserve refund — consistency. Stale is not a concept for a write; the risk is double application. Failure costs: the customer retries, which is exactly what produces the duplicate. Strong within the aggregate, with the refund reservation in the same transaction as the return record, and an idempotency key so a retry returns the first result.
4. Refund status after scanning — availability, visibly. Stale costs: the customer sees “received” a minute later than the warehouse knew it. Nobody is harmed and nobody spends money on the difference. Failure costs: a support contact, which is a guardrail metric the initiative is measured on. Serve from a read model, show “updated 2 minutes ago”, and refresh. This is the clearest availability lean of the five.
5. Carrier label — availability, with a pending state. Stale is meaningless; the question is what happens when the carrier is down. Failure costs: if the label blocks the return, a carrier outage becomes an outage of the outcome. Accept the return, persist the label intent, produce the label from a worker, and tell the customer it is coming. The accepted cost is a pending state in the interface and a queue somebody owns.
The idempotency key belongs to operation 3, and operation 5 inherits it. It must be minted by the client when the customer presses the button — not per HTTP attempt — and reused across every retry of that intent, so the key identifies this customer’s decision to return this order. A key derived server-side from a request timestamp, or regenerated on each retry, produces two returns and two refunds. The carrier call reuses the same key so a retried label request does not print two labels, which is the failure the carrier itself charges for.
Weak — and plausible
Reads 1, 2 and 4 can be eventually consistent because they are just reads and a small delay is acceptable to users. Writes 3 and 5 need strong consistency because they change data and we cannot afford to lose a refund or a label. I would use the database’s default isolation level for the writes and read replicas for the reads, and add caching on the read path to improve performance. Idempotency keys should be added to both writes to be safe.
Splitting on reads-versus-writes is the mistake the ledger exists to prevent. Two of the three reads here have wildly different costs of staleness: a stale refund status costs a support contact, a stale eligibility read costs a wrongly refused return — which is the initiative’s own outcome metric moving the wrong way. Deciding by the shape of the operation instead of by its cost is the per-database thinking lesson 0032 argues against, wearing a per-operation label.
Nothing in the weak answer tells the customer anything. “A small delay is acceptable” becomes, in the interface, a number with no timestamp, which is a promise the system cannot keep. The visible “as of” is not decoration; it is the difference between an honest stale answer and a wrong one.
“Idempotency keys on both writes to be safe” skips the only hard part: where the key comes from. A key the server generates, or one the client regenerates per attempt, is ceremony. The design lives in the sentence “minted once when the customer presses the button, and reused for every retry of that intent”.
Task 3 · Slicing an irreversible change
What to write
The returns table has a status column holding one of six strings. It is written from
four places, read by the customer-facing API, by a nightly finance export, and by a dashboard another
team owns. It must become a guarded state machine with a transition history, and the pilot must keep
shipping throughout.
In about twenty-five minutes write the slice list. For each slice: what it changes, how it is reversed, and what evidence must exist before the next slice starts. Then mark the first slice that cannot be reversed by a deploy, and say what makes it safe to take.
Self-review — answer these before you open the reveal
- For every slice, finish the sentence “to undo this we …”. A slice whose reversal is “restore from backup” is not reversible, it is survivable.
- Count the slices that are independently deployable. If the answer is one, you have written a plan with a big-bang in the middle of it.
- The dashboard belongs to another team. Which of your slices needs them, how much notice do they get, and what happens if they are late?
- Where in your list does the pilot stop shipping features? If nowhere, check again — and if truly nowhere, say what makes that possible.
- The finance export runs nightly. What does it read during the slices where both the old and the new representation exist?
- Did you name the evidence that makes the irreversible slice safe, or only the date it is scheduled for?
Now score what you actually wrote, not what you meant. Each dimension links to its level descriptions.
Delivery sequencing and uncertainty reduction
Data ownership, contract evolution, and failure semantics
Testability, observability, security, and reversibility
Product, design, and engineering collaboration
Not scored yet.
Reveal: a strong answer, a weak one, and the difference
Strong
Slice 1 — characterize the current behavior. Write tests that pin what the six strings actually mean today, including the two transitions the code allows that nobody intended. Changes no production behavior. Reversal: delete the tests. Evidence to proceed: every one of the four writers is covered, and the transition graph is drawn from the tests rather than from memory.
Slice 2 — one door in. Introduce ReturnStatus as an enum and a
single transition_to() that all four writers call. It still writes the same string to
the same column; the guards log rather than raise. Reversal: revert the deploy. Evidence: a week of
logs with zero guard violations — or, more usefully, a handful, each explained. Illegal
transitions in the log are the finding, and they are cheaper now than after the guards are armed.
Slice 3 — arm the guards. The same code, refusing instead of logging, behind a flag defaulting to off, then on. Reversal: flip the flag. Evidence: the refusal rate stays at zero for three days at full traffic.
Slice 4 — expand. Add a return_transitions table; write to it
inside the same transaction as the status update; keep the column authoritative. Nothing reads the
new table yet. Reversal: stop writing it and drop it. Evidence: row counts match status changes for
a week.
Slice 5 — backfill. Rebuild what history can be recovered from the audit log and the finance export, in idempotent batches, marked as reconstructed rather than observed. Reversal: truncate and re-run. Evidence: reconstructed history reconciles with the nightly export for a sample of 200 returns. This slice is where the honest limit shows up: transitions that happened before slice 2 are partly unknowable, and the answer is to label them, not to invent them.
Slice 6 — switch the reads. The customer API reads current state from the transition table’s latest row; the column is still written. Reversal: revert the deploy. Evidence: identical responses for a day of shadow comparison.
Slice 7 — consumers. The finance export and the other team’s dashboard move to the new source. This is the slice that needs a date from somebody else: four weeks of notice, both representations valid throughout, and a compatibility window that ends only when both have confirmed. If they are late, slices 1–6 have already delivered the guards and the history, and slice 8 simply waits.
Slice 8 — contract. This is the first irreversible slice. Drop the
status column. Everything before it is undone by a deploy or a flag; this one is undone
by a restore, which is not a reversal. What makes it safe is evidence, not a date: 30 days of query
logs showing zero reads of the column, written confirmation from both consumers, and a verified
restore of the pre-drop snapshot into a staging database. Take it in a low-traffic window with the
snapshot fresh.
Where the pilot keeps shipping: slices 1–4 are invisible to the product; 5–6 are behind read-path parity checks. The only slice that competes with feature work is 7, and it competes for a conversation rather than for engineering days.
Weak — and plausible
I would create the new return_transitions table and a ReturnStatus enum
with the allowed transitions, migrate the existing data across in a migration script, update all four
writers and the three readers to use the new model, and deploy it together so the system is never in
an inconsistent state. Then drop the old column in the same release. The migration would be tested
thoroughly on a copy of production data first, and we would take a backup before running it.
“Deploy it together so the system is never inconsistent” is the exact instinct that makes the change unreversible. One deploy touching four writers, three readers, a new table and a dropped column has one rollback: restore the database. And one of the three readers belongs to another team, whose deploy schedule this plan has silently assumed.
A backup is not a reversal. Restoring a database means losing every write since the snapshot, which for a live returns flow means losing customers’ returns. “We took a backup” describes how bad the worst case is, not how to avoid it.
Testing on a copy of production finds data problems, not behavior problems. The interesting defect here is the transitions the current code allows that nobody intended — and you find those by logging guard violations under real traffic (slice 2), not by running a migration script against a snapshot.
What the strong answer gives up: eight deploys instead of one, several weeks instead of a sprint, and history before slice 2 that is honestly labeled as reconstructed. It says so. An answer that claims a migration with no losses has not looked hard enough at where the old data was.
Part 3 · What to practice next
| If you missed | What it points at | What to practice next |
|---|---|---|
| p3q1 | Build-versus-buy arithmetic — recurring costs counted at zero. | 0027 and 0026, then redo Task 1 with your own numbers. |
| p3q2 or p3q3 | Complexity trade-offs — abstractions and knobs bought before their cost was named. | 0028, 0029, 0030. |
| p3q4 | Deciding synchronicity from a latency distribution rather than from a habit. | 0031, then 0034. |
| p3q5 | Seeing that a service boundary trades code coupling for runtime and human coordination. | 0033 and 0036. |
| p3q6 | Reversibility — knowing which step is the one you cannot take back. | 0035 and 0037. |
| Task 1 scored below Practicing | Arithmetic missing, or only one volume point considered. | 0027, then the architecture-economics supplement. |
| Task 2 scored below Practicing | One policy applied to five operations, or staleness served without being shown. | 0032 and 0045. |
| Task 3 scored below Practicing | A big-bang hiding in the middle of a slice list, or a backup standing in for a reversal. | 0035, then the brownfield case. |
Exit capability for Phase III. Given a consequential choice, you can produce at least two genuinely different options, show the arithmetic that separates them, choose one, state the cost you accept in a sentence a non-engineer can object to, and set a measurable trigger that reopens it. The decision-change simulations are the place to practice the part this assessment cannot reach: what you do when the evidence moves after you have committed.
Next: the Phase 4 assessment, after lessons 0039–0050, where the problem stops being a system and starts being an initiative.