Software Design Reference Master Designer · System Trade-Offs

Lesson 0031 · Master Designer · Module 2

Synchronous vs Asynchronous Systems

Sync is simple until it isn't fast; async is fast until it isn't simple. The Master's move: choose per workflow, not per system — and know the failure modes you're buying.

Reading ≈ 8 minPractice ≈ 25 min

Mission tie-in: the first System Trade-Off — where lesson 0025's event-driven machinery becomes a measured decision instead of a default.

Knowledge: the ledger of the two modes

SynchronousAsynchronous
CallerGets the result nowGets an acknowledgment; result later
FlowVisible in one call stackDistributed across handlers (lesson 0025's honest cost)
FailureFails in your face, retryable by the callerOrdering, retries, dead letters, visibility — a new failure zoo
WinsSimplicity, debuggability, consistencyDecoupling, resilience, latency hiding, scale
The principle behind the principle

The two forces are immediacy — the caller holds the answer inside one call stack, so the flow is visible, debuggable and consistent — and decoupling — the caller stops inheriting the callee's latency and downtime. The unit of decision is the workflow step: one caller, one obligation. Never the system.

The mechanism is arithmetic. A synchronous call multiplies availabilities. Chain five dependencies at 99.9% each and the workflow is 0.999⁵ ≈ 99.5% available: on 40,000 checkouts a day that is roughly 200 failures a day that no single component caused, and every team can honestly say their service was up. Async replaces the multiplication with durable storage. The queue holds the obligation while the callee is down, so the caller's success no longer depends on the callee being up at that instant. That is the entire benefit — and it is paid for in state: a message that exists, is not yet applied, and may be applied twice or out of order.

So the rule stops applying at two boundaries. Where the caller needs the result to choose its next step, async does not remove the wait; it relocates the wait into a state machine you now own, debug and staff. And where volume is low, the coupling you removed was cheap, while the queue, its dead letters and its correlation ids are a permanent operational cost that arrives every week regardless of traffic.

The deciding question is about the workflow: does the caller need the result to continue? Confirming an order needs the payment result now — sync. Sending the confirmation email doesn't gate anything — async. Most systems are a mix:

def place_order(cart: Cart, gateway: PaymentGateway) -> Order:
    charge = gateway.charge(cart.total, cart.token)      # sync: caller needs it
    order = order_service.confirm(cart, charge.id)
    bus.publish(OrderConfirmed(order))                   # async: reactions don't gate
    return order

The async failure modes you must be able to name before you adopt it: ordering (handlers may see events out of order), at-least-once delivery (idempotency needed), dead letters (what happens when a handler keeps failing), and observability (the flow isn't in one stack trace). If you cannot answer "what happens when the handler dies mid-way?" — the workflow isn't ready for async.

Worked: one step, reasoned out loud

A returns flow: the customer files a return, the service decides eligibility, then a carrier API mints a shipping label. Volume is 3,000 requests a day. The carrier's label endpoint answers in 240 ms at p50 and 2.4 s at p99, and fails about 0.3% of the time. Where does the sync/async boundary go?

The first thing a lead looks at is not the latency graph. It is the customer's next action. Eligibility decides what the customer does next — pack the item, or call support — so it gates. The label does not: the customer's next action is to wait for an email, and a label that arrives ninety seconds later changes nothing they do. That one question, what does the caller do differently with this result?, settles more sync/async arguments than any percentile.

# the shape that follows from the question
def request_return(req: ReturnRequest, bus: Bus) -> ReturnCase:
    case = policy.decide(req)             # sync: the customer acts on this now
    bus.publish(ReturnApproved(case.id))  # async: label, restock, email react
    return case

# the rejected shape: tidy, straight-line, and on the customer's path
def request_return_inline(req: ReturnRequest, carrier: CarrierApi) -> ReturnCase:
    case = policy.decide(req)
    label = carrier.create_label(case)    # p99 2.4 s, 0.3% failure, not gating
    return case.with_label(label)

The alternative the lead rejects is the tidy one: both calls inline, so the response carries the label and the code reads as a straight line. It is rejected on arithmetic. 0.3% of 3,000 is nine customers a day told their return failed when the return had in fact succeeded — eligibility was already decided; only the label step broke. Their recovery is to refile, which produces exactly the support contact the work was meant to remove.

The mirror-image move is rejected too: making eligibility async as well, "so everything is uniform". Uniformity is not a force. Async eligibility turns a two-second answer into a case the customer must come back for, raises the contact rate rather than lowering it, and buys the team a status-polling screen nobody asked for.

The consequence shows up over the following year, not on launch day. In the event-shaped version, every new reaction — restock, fraud scoring, warehouse pre-notice — is one more subscriber to a fact that has already happened, and the customer's path stays one call long. In the inline version, each new reaction adds a call to the critical path: after four features the p99 is the sum of five p99s, and any one of those five dependencies can now fail a return that the business had already approved.

† The asymmetric default: sync unless a concrete reason demands async. Async-everywhere is the pattern that turns every feature into a distributed-systems problem (lesson 0025's cost, now at system scale).
Field notes · the async risk inventory, answered
Failure modeThe question you must answer firstA concrete answer
out-of-order deliveryWhat if OrderShipped arrives before OrderConfirmed?Key handlers by order id; ignore events older than the state.
at-least-once deliveryWhat happens when this handler runs twice?An idempotency key, or INSERT … ON CONFLICT DO NOTHING.
a handler that keeps failingWhere does message five go after three attempts?A dead-letter queue that a human actually watches.
no single stack traceHow do you follow one order across six handlers?A correlation id on every event and every log line.
What people get wrong after reading this
"Async is faster."
End-to-end it is usually slower: you added a broker hop and a scheduling delay. What async changes is who waits — the caller is released, the work still takes as long. Say "the customer stops waiting", not "the system got faster", or you will be measured on a number you did not improve.
"A queue makes the system more reliable."
A queue makes failure quiet. Reliability arrives only with the thing nobody budgets for: an owner who reads the dead-letter queue on a schedule. An unmonitored DLQ is silent data loss with extra infrastructure.
"We publish events, so the services are decoupled."
You swapped compile-time coupling for schema coupling. Consumers still depend on the fields you emit; the difference is that breaking them now fails in production at 3 a.m. instead of in CI. Decoupling is what the compatibility rules buy (lesson 0033), not what the transport buys.
"Sync means HTTP and async means a message broker."
The mode is about gating, not transport. An HTTP call that returns 202 Accepted with a case id is async; a broker request/reply where the caller blocks on the response is sync with more moving parts.

When this is the wrong move

The sync-first default is the wrong starting point in three recognisable situations. When a step fans out to several downstream systems, sync makes you inherit the union of their downtime — five 99.9% dependencies is a worse promise than any of them made. When the callee is a third party whose latency you do not control and whose contract gives you no percentile, a synchronous call exports their bad day onto your customers. And when the work is inherently bulk — reprocessing 40,000 rows, regenerating every label — a synchronous request times out before correctness is even in question.

There is also a team-shaped exception. If the organisation already runs a bus with dead-letter monitoring, replay tooling and correlation ids in every log line, the marginal cost of one more consumer is near zero, and the "async is complex" argument is quoting a price the team has already paid. Judge the machinery you actually have, not the machinery a blog post assumes.

In the lab: the companion Phase 3 exercise hands you a returns service that awaits a carrier call inside the customer's request. You classify each step by whether the caller gates on it, move the non-gating work behind an event, and answer the four failure questions before the change counts as done.

Skill: which mode for this workflow?

What decides whether a workflow step may run asynchronously?

The confirmation email doesn't gate anything. It should be:

Before adopting async, you must be able to answer:

Practice on your own code

List the workflows in your system. For each, mark: does the caller gate on the result? Where failure modes exist (ordering, duplicates, dead handlers), write the answer to "what happens if the handler dies mid-way?" — on paper, before you build more async.

Reveal: a workflow-by-workflow decision

An e-commerce system: place order — sync (payment gates confirmation). Send receipt — async (event, idempotent by order id). Update inventory — async but ordered per SKU, with a dead-letter queue for the warehouse sync. Analytics — async, best-effort, no dead-letter (loss is acceptable). Four workflows, three modes, each decision written down with its failure answer.

Teach it back
  1. In two sentences, explain to a junior why "make it async" is not a performance fix.
  2. Find one step in your own system that sits on a caller's path without gating anything. Name its cost in the units your team already tracks — p99 milliseconds, failed requests per day, or pages per month.
  3. Predict the one-year state if nobody moves it: how long is the critical path after four more features, and what is the first incident that comes from it?

Your win

You can classify each workflow by its gating requirement, and you carry the async failure inventory — ordering, at-least-once, dead letters, observability — as a checklist before adopting it.

Read and watch deeper

Walk your workflow list past your agent-teacher and classify each one together.