Iterators and Generators

Produce values one at a time, on demand, without ever building the whole list

Big Idea: A for loop does not magically know how to walk a list, a file, or a range. Behind the scenes Python asks an object for its iterator and then keeps asking that iterator for the next value until it says "stop". A generator is the easiest way to build such an object: you write an ordinary-looking function, replace return with yield, and Python hands you a lazy, memory-light stream of values.

By the end you will be able to:

explain the iterator protocol (__iter__, __next__, StopIteration),
tell an iterable apart from an iterator,
write a generator function with yield and trace how it pauses and resumes,
reason about lazy evaluation and why it saves memory,
use compact generator expressions and know when they beat a list comprehension.

1. What really happens in a for loop

You have written for x in items: hundreds of times. What Python actually does is a two-step dance. First it calls iter(items) to get an iterator. Then, on every pass, it calls next() on that iterator to pull one value. When the iterator has nothing left, it raises a special exception, StopIteration, and the loop ends quietly.

You can run that dance by hand:

nums = [10, 20, 30]
it = iter(nums)        # get an iterator from the list
print(next(it))        # 10
print(next(it))        # 20
print(next(it))        # 30
print(next(it))        # raises StopIteration

A for loop is just this pattern with the StopIteration caught for you. Understanding it explains almost everything else in this topic.

2. Iterable vs iterator: two different jobs

These two words sound alike but name different roles, and mixing them up is the number-one source of confusion.

An iterable is anything you can loop over: a list, tuple, string, dict, set, file, or range. It knows how to produce an iterator via __iter__. It has no idea "where you are".
An iterator is the stateful cursor. It remembers your current position and knows how to give the next value via __next__. Every iterator is also iterable, because its __iter__ returns itself.

A restaurant analogy: the menu is the iterable, the same for everyone. The bookmark a specific diner keeps on the page they are reading is the iterator. Two diners can read the same menu with two separate bookmarks at the same time.

nums = [1, 2, 3]
a = iter(nums)         # bookmark A
b = iter(nums)         # bookmark B, independent
print(next(a))         # 1
print(next(a))         # 2
print(next(b))         # 1  (B started fresh)

Because a list is only iterable (not an iterator), you can loop over it again and again. An iterator, once exhausted, is used up for good.

Iterators and Generators