Threading for I/O-bound work

Roadmaps › Python

Scenario

You download 50 web pages, one per HTTP request, each taking ~200ms of network wait. Done serially it takes 10 seconds. You wrap it in threads and it drops to under a second — even though the GIL supposedly means 'only one thread runs Python at a time.' How is that not a contradiction?

Mental model

Ten people ordering pizza by phone. One phone, ten callers — but 95% of each call is waiting on hold. So one caller dials, and the moment they're on hold they hand the phone to the next. The phone (the CPU) is almost never the bottleneck; the waiting is, and waiting overlaps for free.

The GIL lets only one thread execute Python bytecode at a time, so threading gives no speedup for CPU-bound work. But I/O-bound work isn't running bytecode while it waits — it's blocked in the OS on a socket, disk, or lock. CPython *releases the GIL* around blocking I/O calls, so while thread A waits for bytes to arrive, thread B holds the GIL and runs. The waits overlap. That's why threading is the right tool for I/O-bound concurrency (network, files, DB calls) and the wrong tool for number-crunching.

Deep dive

The distinction that decides everything: is the work CPU-bound (busy computing — parsing, math, compression) or I/O-bound (mostly waiting — network, disk, database, subprocess)? Threading only helps the second kind. For I/O-bound work the thread spends nearly all its time blocked, not executing bytecode, so the GIL is free for other threads.

Threading speeds up I/O-bound (waiting) work, not CPU-bound (computing) work.

The reason it works: CPython releases the GIL before it makes a blocking system call (a socket read, a file read, `time.sleep`) and re-acquires it after. So a thread parked in `requests.get()` is holding no GIL — other threads run during that window. Ten threads each waiting 200ms overlap into ~200ms total instead of 2 seconds.

Blocking I/O calls release the GIL, letting other threads run during the wait.

In practice you rarely manage threads by hand — `concurrent.futures.ThreadPoolExecutor` gives you a pool and a clean `map`/`submit` API. Pick a worker count based on how many operations can be in-flight at once (for network calls, tens to low hundreds is fine — they're mostly idle). Because threads share memory, guard any shared mutable state with a `Lock`.

Use `ThreadPoolExecutor` for I/O fan-out; lock shared mutable state.

Code examples

Serial vs threaded I/O

import time
from concurrent.futures import ThreadPoolExecutor

def fetch(url):
    time.sleep(0.2)   # stand-in for a 200ms network round-trip
    return f"{url}: ok"

urls = [f"https://api/{i}" for i in range(50)]

# serial: ~10s (50 * 0.2)
start = time.perf_counter()
[fetch(u) for u in urls]
print(f"serial:   {time.perf_counter()-start:.2f}s")

# threaded: ~0.2s of overlapped waiting
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=50) as pool:
    list(pool.map(fetch, urls))
print(f"threaded: {time.perf_counter()-start:.2f}s")

Each `fetch` is 200ms of pure waiting. Serially that's 50 waits back-to-back (~10s). In the thread pool all 50 waits happen at once — while any thread is blocked in `time.sleep`/the network, it isn't holding the GIL, so the others proceed. Total collapses to ~one wait.

Locking shared mutable state

import threading
from concurrent.futures import ThreadPoolExecutor

counter = 0
lock = threading.Lock()

def work():
    global counter
    for _ in range(100_000):
        with lock:          # without this, updates race and get lost
            counter += 1

with ThreadPoolExecutor(max_workers=4) as pool:
    for _ in range(4):
        pool.submit(work)

print(counter)   # 400000 — correct only because of the lock

`counter += 1` is read-modify-write — not atomic — so two threads can read the same value and one increment is lost. The GIL does NOT save you here: it can switch threads mid-operation. A `Lock` makes the update mutually exclusive so the final count is right.

Common mistakes

Recall questions

Questions & answers

A script makes 200 independent REST API calls, each ~150ms, and takes 30 seconds. A teammate says 'Python has the GIL so threads won't help — use multiprocessing.' Are they right? What do you do?

They're wrong for this workload. The calls are I/O-bound — almost all of the 150ms is network waiting, during which CPython releases the GIL — so threads overlap the waits and cut the wall-clock dramatically. A `ThreadPoolExecutor` with, say, 50–100 workers brings 30s down toward ~a second. Multiprocessing would also work but adds process/serialization overhead that's unnecessary for pure I/O; threads (or asyncio) are the lean choice. Multiprocessing is the answer only when the bottleneck is CPU.

Approach: Classify the work as I/O-bound, explain the GIL is released during I/O so threads overlap waits, and reserve multiprocessing for CPU-bound work.

Two threads each increment a shared integer 1,000,000 times, but the final total is less than 2,000,000. Explain the bug and fix it.

`x += 1` compiles to load, add, store — three steps. A thread can be suspended between the load and the store; the other thread reads the same old value, and one increment is overwritten, so counts are lost (a race condition). The GIL doesn't prevent this because it can switch threads between bytecodes. Fix: wrap the update in a `threading.Lock` (`with lock: x += 1`) so it's atomic, or use a lock-free design like accumulating per-thread and summing at the end.

Approach: Identify the non-atomic read-modify-write race, note the GIL doesn't guarantee operation-level atomicity, and fix with a Lock.

Continue learning

Previous: The GIL (Global Interpreter Lock)

Next: Threading vs Multiprocessing

Return to Python Roadmap