Lesson 0018 · System Designer · Module 1

Memory-Efficient Pipelines with Generators

Processing a terabyte of logs doesn't require a terabyte of RAM. Generators turn "load everything, then process" into "process as it flows" — with the same readable shape.

Mission tie-in: "build memory-efficient data pipelines with generators and streaming" — the composability skill that keeps large-data code honest and small.

Knowledge: lazy stages, chained

A generator produces values one at a time, on demand. Chain generators and each stage processes a single item as it flows past — memory stays flat no matter how much data streams through:

def read_lines(path: str):
    with open(path) as f:
        for line in f:                # the file itself streams
            yield line

def parse(line: str) -> LogEntry:
    ...

def error_entries(lines):
    for line in lines:
        entry = parse(line)
        if entry.level == "ERROR":
            yield entry

def summarize(entries) -> Counter[str]:
    counts = Counter()
    for entry in entries:
        counts[entry.service] += 1
    return counts

total = summarize(error_entries(read_lines("app.log")))

At any moment, exactly one line is in memory: the file streams, each stage transforms one item, the counter aggregates. The pipeline reads like a sentence — summarize(error_entries(read_lines(...))) — and each stage is independently testable.

The contrast is the list-based version: read all lines into a list, build a filtered list, build a parsed list… each intermediate list is the whole dataset again. That's the memory failure mode of naive code — and of AI code, which defaults to lists.

† Composability rule: a generator stage should take an iterable and yield items — never materialize a collection in between. itertools (chain, islice, groupby, takewhile) is the standard library's pipeline toolbox.

Field notes · what it looks like in real code

You see this	What it costs	The move
`rows = f.readlines()`	The whole file lands in memory before any work starts.	`for line in f:` — the file is already a stream.
[x for x in …] between two loops	One extra full copy of the dataset per bracket pair.	Drop the brackets: a generator expression.
len(stream) or stream[0]	Something materialized the stream just to index it.	`itertools.islice`, or count as you go.
a generator consumed twice	The second pass sees nothing — a silent empty result.	`itertools.tee`, or materialize once, deliberately.

Skill: list or stream?

A pipeline over a huge file should be built with:

A generator stage should accept and return:

The right time to reach for generators is:

Practice on your own code

Find a function that builds a list, transforms it into another list, then another (three or more materializations). Convert it to chained generator stages with an iterable contract, and measure the peak memory if you can.

Reveal: list pipeline → stream

# before: three full copies of the dataset
rows = fetch_all_rows()
filtered = [r for r in rows if r.active]
parsed = [parse(r) for r in filtered]
total = sum(r.amount for r in parsed)

# after: one pass, flat memory
def active(rows):
    for r in rows:
        if r.active:
            yield r

total = sum(parse(r).amount for r in active(fetch_rows_stream()))

Your win

You can spot the list-chained pipelines in your code (and AI output), convert them to flat-memory streams, and keep each stage composable and testable.

Read and watch deeper

Fluent Python, Ramalho — ch. 17 "Iterators, Generators, and Classic Coroutines": generator fundamentals and pipeline composition.
Python docs — itertools reference: the streaming toolbox; and the Functional Programming HOWTO.
Effective Python, Slatkin — the items on generators: when lazy evaluation wins and when it doesn't.
Watch: ArjanCodes YouTube — search "generators python".
Next module: lesson 0019, managing complexity in growing applications.
Reference: Patterns — generator pipeline, with the iterable-in/items-out contract.

Bring a list-chained function to your agent-teacher and convert it to a stream together.

Mark lesson complete