Generator expressions
Overview
A generator expression looks just like a list comprehension, but uses parentheses `()` instead of square brackets `[]`. Instead of building an entire list in memory all at once, a generator expression uses lazy evaluation. It calculates and yields exactly one item at a time, pausing execution until the next item is requested. This means a generator expression takes up virtually no memory, regardless of how many items it will eventually produce. You can use them directly inside aggregation functions (functions that compute a single result from many values) like `sum()`, `max()`, or `any()`.
When processing millions of database rows or gigabytes of log files, loading the entire dataset into memory with a list comprehension will crash your application with an OutOfMemory error. Generator expressions allow you to process massive datasets in a constant O(1) memory footprint.
Where used: Aggregating data directly (e.g. `sum(x for x in data)`), Reading large files line-by-line, Processing streams of API data
Why learn this
- Understanding lazy evaluation and memory efficiency
- Knowing when to trade the random-access capability of lists for the memory savings of generators
- Understanding why a generator appears 'empty' if you try to iterate over it twice
Code walkthrough
numbers = [1, 2, 3]
gen = (x * 10 for x in numbers)
print(next(gen))
print(next(gen))
print(next(gen))Focus: The `next()` function pulls exactly one value out of the generator expression at a time, proving it evaluates lazily.
Aha moment
gen = (x for x in [1, 2, 3])
first_pass = list(gen)
second_pass = list(gen)
print(second_pass)Prediction: What will `second_pass` contain?
Common guess: [1, 2, 3]
A generator expression is consumed as it is read. The `first_pass` exhausted the generator. When `second_pass` tried to read from it, there were no items left, returning an empty list.
Common mistakes
- Iterating a generator twice: It is exhausted as it produces items, so a second loop yields nothing because all items have already been consumed. If you need multiple passes, use a list.
- Trying to index or slice: Generators only know how to produce the *next* item. You cannot access `gen[5]` or check its `len()` because the items don't exist yet.
Glossary
- lazy evaluation
- Computing or fetching values only exactly when they are needed, rather than all at once in advance. Example: `(x*2 for x in range(10))` — values are computed one at a time, not all upfront.
- random-access
- The ability to instantly retrieve an item at any specific position in a collection, like getting the 5th item in a list. Example: `my_list[4]` fetches the 5th element in constant time.
Recall questions
- What is the primary difference in memory usage between a list comprehension and a generator expression?
- What syntax difference separates a list comprehension from a generator expression?
- What happens if you try to loop over the same generator expression twice?
Understanding checks
What is the output of the second print statement?
`[]`
A generator expression is evaluated lazily and exhausted as it is consumed. The first `list(gen)` consumes all items. The second `list(gen)` finds an empty generator.
A junior developer is confused why `my_gen[0]` raises a TypeError when `my_gen` is a generator expression. How do you explain it?
Generator expressions do not support indexing because their items are evaluated lazily one at a time.
Unlike lists, which hold all their items in memory at once and support random access, generators only know how to produce the next item.
Practice tasks
Memory-efficient sum
Change the list comprehension to a generator expression to calculate the sum of squares without allocating memory for the entire list.
Challenge
Any big numbers?
Using a generator expression with the `any()` function, check if there are any numbers greater than 100 in `nums`. Assign the boolean result to `has_big`.
Continue learning
Previous: Comprehensions