Generators & yield
Overview
A generator function is a function that contains the `yield` keyword. Unlike a normal function that computes a result and returns it all at once, destroying its local state, a generator function pauses its execution at each `yield`, returning a value to the caller but keeping its local variables and execution pointer intact. When `next()` is called on it again, it resumes exactly where it left off. You can think of a generator function as a resumable machine: instead of building the entire product and handing it over at the end, it spits out one piece on demand, pauses, and waits for you to ask for the next piece.
Generators allow you to produce sequences of values over time without having to load all of them into memory at once, enabling efficient processing of large datasets, infinite sequences, or streams.
Where used: FastAPI dependency injection (yield for cleanup), pytest fixtures (yield for teardown), Processing large CSV files or database queries
Why learn this
- Processing files larger than your available RAM
- Writing FastAPI dependencies with setup and teardown steps
- Creating efficient data pipelines
Code walkthrough
def one_two_three():
yield 1
yield 2
yield 3
gen = one_two_three()
print(next(gen))
print(next(gen))Focus: print(next(gen))
Aha moment
def tricky_gen():
print('Starting')
yield 1
print('Continuing')
yield 2
g = tricky_gen()
print('Created generator')Prediction: What does this print when executed?
Common guess: 'Starting', then 'Created generator'
Calling `tricky_gen()` only creates the generator object; it doesn't execute the function body at all. The code inside the generator function only runs when `next()` or a loop is used, so 'Starting' is never printed here.
Common mistakes
- Calling the function instead of iterating: Calling a generator function `gen()` doesn't execute its body; it just returns a generator object. You must iterate over it (e.g., using `for` or `next()`) to run the code.
- Consuming a generator more than once: Generators are exhausted once they yield all their values. If you try to iterate over the same generator object a second time, it will yield nothing. You must create a new generator object.
Glossary
- local state
- The current values of variables that exist only within a specific function and are usually lost when the function ends. Example: `count = 0` inside a function.
- execution pointer
- An internal marker that keeps track of exactly which line of code a program is currently running or paused at. Example: pausing at `yield 1`.
Recall questions
- What is the key difference in how a function's state is handled when it uses `yield` compared to `return`?
- Why would you choose to write a generator function instead of returning a list?
- If you call a generator function, what does it actually return before any iteration happens?
Understanding checks
What does the following code print?
1
Calling `next(g)` resumes the generator, executing it until the first `yield` statement, which returns 1.
A developer expects calling `process_data()` to immediately execute the first part of the function and print 'Starting...'. Why doesn't it?
Because `process_data()` is a generator function, calling it only returns a generator object. It doesn't start executing the body.
A generator function's body only begins running when `next()` is called on the resulting generator object or when it is iterated over.
Practice tasks
Convert List to Generator
The `countdown` function currently builds a complete list of numbers in memory and returns it. Modify it to be a generator function that `yield`s each number one by one instead.
Challenge
Batch Processor
Write a generator function `batch_items(items, batch_size)` that takes a list of `items` and yields sub-lists of length `batch_size`. If the remaining items are fewer than `batch_size`, yield them as the final batch. For example, `list(batch_items([1, 2, 3, 4, 5], 2))` should result in `[[1, 2], [3, 4], [5]]`.
Continue learning
Previous: Iterators & the iterator protocol