The Iterable Protocol
Overview
The Iterable Protocol is Python's mechanism for traversing containers. Think of an iterable as a book (a collection of pages) and an iterator as a bookmark (the current reading state); you can have many bookmarks in the same book. To support this protocol, objects implement two specific magic methods: - `__iter__()`: Makes an object an iterable. Returns a new iterator object, or raises `TypeError` if not implemented when passed to `iter()`. - `__next__()`: Makes an object an iterator. Returns the next item in the sequence, or raises `StopIteration` to signal the sequence is exhausted. When you run a `for` loop, Python automatically calls `iter()` to get an iterator, then repeatedly calls `next()` until the exception is caught. gen = (x for x in range(3)) list(gen) list(gen) # Returns [] Generator expressions return iterators, not independent iterables, meaning they are completely exhausted after one pass. This memory-saving feature is covered fully in `generator-expressions`.
It allows Python to loop over any object uniformly (such as `list`, `str`, `file`, or custom streams) without needing to know its internal structure. This abstraction enables infinite sequences and `lazy evaluation` to save memory.
Where used: SQLAlchemy query results, Streaming file lines, FastAPI request bodies
Why learn this
- Allows you to create custom collections that behave like built-in lists in `for` loops.
- Explains why iterators are exhausted after one pass, unlike a `list`.
- Underpins how `lazy evaluation` saves memory when processing massive datasets.
Code walkthrough
nums = [1, 2]
iterator = iter(nums)
print(next(iterator))
print(next(iterator))
try:
print(next(iterator))
except StopIteration:
print('Done!')Focus: print(next(iterator))
Aha moment
nums = [1, 2]
iterator = iter(nums)
for x in iterator:
print(x)
for x in iterator:
print('Again:', x)Prediction: What will be printed by the second `for` loop?
Common guess: It will print `Again: 1` and `Again: 2`.
The first `for` loop exhausts the iterator. The second `for` loop calls `iter(iterator)`, which returns the exhausted iterator itself, immediately raising `StopIteration`.
Common mistakes
- Exhausted Iterators: Trying to iterate over an iterator twice yields nothing the second time because it retains its exhausted state. Convert it to a `list` if you need multiple passes, or create a fresh iterator by calling `iter()` on the original iterable again.
- Confusing Iterable and Iterator: Returning `self` in `__iter__()` for a container that should not hold state. An iterable's `__iter__()` should return a *new* iterator object. Conversely, an iterator's `__iter__()` must return `self`.
Glossary
- traversing
- Systematically going through or visiting every single item in a collection or data structure. Example: a `for` loop traversing a list of names.
- lazy evaluation
- Computing or fetching values only exactly when they are needed, rather than all at once in advance. Example: yielding one line from a file at a time using lazy evaluation.
Recall questions
- What is the difference between an iterable and an iterator?
- How does a `for` loop know when to stop iterating?
- Why can you loop over a list multiple times, but an iterator only once?
Understanding checks
What will the code print?
Nothing.
The first `for` loop exhausts the iterator. The second `for` loop iterates over the exhausted iterator, which immediately raises `StopIteration`.
Why doesn't this class act as a proper iterable that can be reused?
It returns `self` in `__iter__()`, meaning it acts as its own iterator and gets exhausted.
To allow multiple passes, `__iter__()` must return a *new* iterator object each time, instead of returning the same exhausted object (`self`).
Practice tasks
Fix the Exhausting Iterable
The `Countdown` class below acts as both an iterable and an iterator, meaning it can only be looped over once. Modify the code by splitting it into two classes: `CountdownIterator` (which holds the state and implements `__next__()`) and `Countdown` (which returns a new `CountdownIterator` in its `__iter__()` method) so that it can be looped over multiple times.
Challenge
Paginated API Iterator
Write an iterable class `PaginatedResults` that simulates fetching data from an API. It takes a list of lists (representing pages of data). When iterated, it should yield individual items across all pages sequentially. You MUST separate the iterable and the iterator into two classes so it can be consumed multiple times.
Continue learning
Previous: __iter__ and __next__
Previous: for and while loops