yield keyword and generator functions
Overview
A generator function is a function that contains the `yield` keyword. Unlike a normal function that computes a result and returns it all at once, a generator function pauses its execution at each `yield`. This returns a value to the caller while keeping its local variables and execution pointer intact. When `next()` is called on it again, it resumes exactly where it left off. def count_up_to(max_val): count = 1 while count <= max_val: yield count count += 1 You can think of a generator function as a resumable machine. Instead of building the entire product and handing it over at the end, it spits out one piece on demand, pauses, and waits for you to ask for the next piece. def return_in_gen(): yield 1 return "Finished" g = return_in_gen() next(g) # Returns 1 next(g) # Raises StopIteration("Finished") When a generator executes a `return` statement, it raises a `StopIteration` exception, and the returned value becomes the exception's argument. This behavior is critical when delegating generators, covered fully in yield-from.
Generators allow you to produce sequences of values over time without having to load all of them into memory at once, enabling efficient processing of large datasets, infinite sequences, or streams.
Where used: `FastAPI` dependency injection (`yield` for cleanup), `pytest` fixtures (`yield` for teardown), Processing large CSV files or database queries
Why learn this
- Processing files larger than your available RAM
- Writing `FastAPI` dependencies with setup and teardown steps
- Creating efficient data pipelines
Code walkthrough
def one_two_three():
yield 1
yield 2
yield 3
gen = one_two_three()
print(next(gen))
print(next(gen))Focus: print(next(gen))
Aha moment
def tricky_gen():
print('Starting')
yield 1
print('Continuing')
yield 2
g = tricky_gen()
print('Created generator')Prediction: What does this print when executed?
Common guess: `'Starting'`, then `'Created generator'`
Calling `tricky_gen()` only creates the generator object; it doesn't execute the function body at all. The code inside the generator function only runs when `next()` or a `for` loop is used, so `'Starting'` is never printed here.
Common mistakes
- Calling the function instead of iterating: Calling a generator function `gen()` doesn't execute its body; it just returns a generator object. You must iterate over it (e.g., using `for` or `next()`) to run the code.
- Consuming a generator more than once: Generators are exhausted once they `yield` all their values. If you try to iterate over the same generator object a second time, it will `yield` nothing. You must create a new generator object.
Glossary
- local state
- The current values of variables that exist only within a specific function and are usually lost when the function ends. Example: `count = 0` inside a function.
- execution pointer
- An internal marker that keeps track of exactly which line of code a program is currently running or paused at. Example: pausing at `yield 1`.
Recall questions
- What is the key difference in how a function's state is handled when it uses `yield` compared to `return`?
- Why would you choose to write a generator function instead of returning a `list`?
- If you call a generator function, what does it actually return before any iteration happens?
Understanding checks
What does the following code print?
`1`
Calling `next(g)` resumes the generator, executing it until the first `yield` statement, which returns `1`.
A developer expects calling `process_data()` to immediately execute the first part of the function and print `'Starting...'`. Why doesn't it?
Because `process_data()` is a generator function, calling it only returns a generator object. It doesn't start executing the body.
A generator function's body only begins running when `next()` is called on the resulting generator object or when it is iterated over.
What exception is raised when you call `next(g)` a second time on this generator?
`StopIteration('b')`
When a generator hits a `return` statement, it raises a `StopIteration` exception with the returned value attached to it.
Practice tasks
Convert List to Generator
The `countdown` function currently builds a complete `list` of numbers in memory and returns it. Modify it to be a generator function that `yield`s each number one by one instead.
Challenge
Batch Processor
Write a generator function `batch_items(items, batch_size)` that takes a `list` of `items` and yields sub-lists of length `batch_size`. If the remaining items are fewer than `batch_size`, `yield` them as the final batch. For example, `list(batch_items([1, 2, 3, 4, 5], 2))` should result in `[[1, 2], [3, 4], [5]]`.
Continue learning
Previous: Defining functions and return values
Previous: The Iterable Protocol
Previous: Iterator Protocol and StopIteration
Previous: iter() and next() builtins
Next: Lazy Evaluation and Memory Efficiency
Next: yield from and generator delegation