Software Design Reference System Designer · Flexible & Composable Code

Lesson 0018 · System Designer · Module 1

Memory-Efficient Pipelines with Generators

Processing a terabyte of logs doesn't require a terabyte of RAM. Generators turn "load everything, then process" into "process as it flows" — with the same readable shape.

Reading ≈ 8 minPractice ≈ 25 min

Mission tie-in: "build memory-efficient data pipelines with generators and streaming" — the composability skill that keeps large-data code honest and small.

Knowledge: lazy stages, chained

A generator produces values one at a time, on demand. Chain generators and each stage processes a single item as it flows past — memory stays flat no matter how much data streams through:

def read_lines(path: str):
    with open(path) as f:
        for line in f:                # the file itself streams
            yield line

def parse(line: str) -> LogEntry:
    ...

def error_entries(lines):
    for line in lines:
        entry = parse(line)
        if entry.level == "ERROR":
            yield entry

def summarize(entries) -> Counter[str]:
    counts = Counter()
    for entry in entries:
        counts[entry.service] += 1
    return counts

total = summarize(error_entries(read_lines("app.log")))

At any moment, exactly one line is in memory: the file streams, each stage transforms one item, the counter aggregates. The pipeline reads like a sentence — summarize(error_entries(read_lines(...))) — and each stage is independently testable.

The contrast is the list-based version: read all lines into a list, build a filtered list, build a parsed list… each intermediate list is the whole dataset again. That's the memory failure mode of naive code — and of AI code, which defaults to lists.

The principle behind the principle

Laziness does not remove work; it changes when the work happens and who is holding things while it happens. Peak memory falls from O(n) per stage to O(1) per stage because no stage ever owns a whole collection. What you hand over in exchange is everything that depends on the collection existing: random access, length, a second pass — and, less obviously, the execution context. The body of a generator runs inside the consumer's call frame, at the consumer's moment, which means errors, resource lifetimes and transaction boundaries migrate to the consumer too. That single fact explains almost every generator bug worth knowing about.

So the rule stops being true in four places. When you need more than one pass, laziness costs you the data. When the generator owns a resource — an open file, a database cursor, a transaction — the resource now closes whenever the consumer finishes, which may be never. When an argument is invalid, the exception is raised not where the generator was called but where someone iterated it, sometimes in another module, which is the exact inversion of lesson 0014. And when the sink wants batches, per-item streaming turns one round trip into a million.

How a senior designer works the case

An export endpoint in a long-running web worker. The pipeline is the textbook one above: read_lines opens the file and yields inside the with. A preview feature wraps it in islice(stream, 100). Two weeks later, the box serving the most previews dies with OSError: too many open files.

What they look at first. Not the memory profile — the with statement's position relative to the yield:

def read_lines(path: str):
    with open(path) as f:          # the context manager's exit is inside the generator
        for line in f:
            yield line             # abandoned here, the file stays open

The question they ask. "Who owns this resource, and at what moment does it close?" The answer is uncomfortable: it closes when the generator is exhausted, or when the generator object is collected and CPython throws GeneratorExit into the suspended frame. Refcounting usually makes that prompt — and "usually" is not a design. Hold a reference in a cache, a partially consumed result, or an exception traceback, and the descriptor lives as long as the reference does.

The alternative they reject. Going back to return f.readlines(). It restores deterministic cleanup and reinstates the original problem: this endpoint exports files of several gigabytes, which is why the pipeline was written lazily in the first place. Trading a descriptor leak for an out-of-memory kill is not progress.

def parse_lines(lines: Iterable[str]) -> Iterator[LogEntry]:
    for line in lines:             # owns no resource; owns no lifetime
        yield parse(line)

def export(path: str) -> Response:
    with open(path) as f:          # the request owns the file, and closes it
        return stream_response(parse_lines(f))

The second half of the same case. The fixed pipeline writes its results to Postgres, one row at a time, because streaming is the habit now. Two million rows at half a millisecond per round trip is roughly seventeen minutes of network latency doing nothing else. The fix is the other half of the rule — stream between stages, batch at the sink:

def batched(iterable, size):                    # itertools.batched, on Python 3.12+
    iterator = iter(iterable)
    while chunk := list(islice(iterator, size)):
        yield chunk

for chunk in batched(entries, 1000):            # flat memory, 2,000 round trips
    db.insert_many(chunk)

The consequence over time. Both fixes are small, and both come from the same idea rather than from generator trivia: laziness moves ownership to the consumer, so decide deliberately what the consumer owns. Teams that skip this end up with the two classic outcomes — a service that leaks descriptors under partial consumption, and a nightly job that is memory-perfect and four hours long.

† Composability rule: a generator stage should take an iterable and yield items — never materialize a collection in between. itertools (chain, islice, groupby, takewhile) is the standard library's pipeline toolbox.
Field notes · what it looks like in real code
You see thisWhat it costsThe move
rows = f.readlines()The whole file lands in memory before any work starts.for line in f: — the file is already a stream.
[x for x in …] between two loopsOne extra full copy of the dataset per bracket pair.Drop the brackets: a generator expression.
len(stream) or stream[0]Something materialized the stream just to index it.itertools.islice, or count as you go.
a generator consumed twiceThe second pass sees nothing — a silent empty result.itertools.tee, or materialize once, deliberately.
What people get wrong after reading this
"Generators are faster."
They lower peak memory, which is often why a job gets faster — it stops swapping or stops being killed. Per item they add overhead, so over a few thousand elements a list comprehension usually wins. Streaming is a memory and latency decision, not a speed trick.
"A generator holds no memory."
It holds its suspended frame: every local, everything it closed over, and every resource it opened. A generator iterating a list keeps the whole list alive. What it avoids is materialising new collections between stages.
"Nothing runs until you iterate, so mistakes show up early."
The opposite. Calling a generator function runs none of its body, so a bad argument raises nothing at the call — the error appears at the first next(), often in a different module, with a traceback pointing at the consumer. Validate arguments in a plain wrapper function that returns the generator.
"I can iterate it twice."
The second pass yields nothing, silently — no exception, just an empty result and a wrong total. If two passes are needed, materialise once on purpose, or build the source twice.

When this is the wrong move

Skill: list or stream?

A pipeline over a huge file should be built with:

A generator stage should accept and return:

The right time to reach for generators is:

Practice on your own code

Find a function that builds a list, transforms it into another list, then another (three or more materializations). Convert it to chained generator stages with an iterable contract, and measure the peak memory if you can.

In the lab: the phase-2 checkpoint has a returns export that builds three full lists in a row and then writes rows one at a time. You convert the middle to streaming stages, batch the write, and decide where the file handle should be opened once you have.

Reveal: list pipeline → stream 
# before: three full copies of the dataset
rows = fetch_all_rows()
filtered = [r for r in rows if r.active]
parsed = [parse(r) for r in filtered]
total = sum(r.amount for r in parsed)

# after: one pass, flat memory
def active(rows):
    for r in rows:
        if r.active:
            yield r

total = sum(parse(r).amount for r in active(fetch_rows_stream()))
Teach it back
  1. In two sentences, explain to a junior why "stream between stages, batch at the sink" is one rule rather than two.
  2. Find one generator in your code this week that opens a resource. Say what happens to that resource if a caller stops iterating halfway.
  3. Assume your team converts every pipeline to generators over the next year. Name the failure you expect first, and the metric that would reveal it.

Your win

You can spot the list-chained pipelines in your code (and AI output), convert them to flat-memory streams, and keep each stage composable and testable.

Read and watch deeper

Bring a list-chained function to your agent-teacher and convert it to a stream together.