Multiprocessing for CPU-bound work

Roadmaps › Python

Scenario

You have an 8-core machine and a CPU-heavy job: hash a million records. You thread it across 8 workers expecting an 8x speedup — and it runs exactly as slow as the single-threaded version, sometimes slower. Switch `ThreadPoolExecutor` to `ProcessPoolExecutor`, change nothing else, and it finally runs ~8x faster.

Mental model

One kitchen with one stove (the GIL) can only cook one dish at a time no matter how many cooks you hire — extra cooks just wait for the stove. Multiprocessing builds eight separate kitchens, each with its own stove. Now eight dishes cook at once. The cost: the kitchens don't share a fridge, so you have to carry ingredients (data) between them.

Each Python process has its own interpreter and its own GIL, so N processes run Python bytecode on N cores truly in parallel — the only way to get real CPU parallelism in CPython. The trade-off is isolation: processes don't share memory, so arguments and return values are pickled and shipped between them, and starting a process is far heavier than starting a thread. That overhead is worth it when the compute per task dwarfs the data shipped — the definition of CPU-bound work.

Deep dive

Because every process gets its own GIL, multiprocessing is the standard way to parallelize CPU-bound Python — math, parsing, image/number crunching, compression. Split the work across roughly `os.cpu_count()` workers and each runs on a separate core simultaneously. This is the exact case where threading fails.

Separate processes = separate GILs = true multi-core parallelism for CPU work.

The cost is data movement. Processes don't share address space, so inputs and outputs cross the boundary via pickling (serialize → send → deserialize). If each task does little compute but passes large objects, that serialization can cost more than you save. Multiprocessing wins when compute-per-task is large relative to the data shipped.

Args/results are pickled across the process boundary — cheap compute + big data can erase the gains.

Use `concurrent.futures.ProcessPoolExecutor` — the API mirrors `ThreadPoolExecutor`, so switching between them is often a one-word change. Two gotchas: the target function and its arguments must be picklable (no lambdas/local closures), and on the 'spawn' start method (default on Windows and macOS) your entry point must be guarded by `if __name__ == '__main__':` or child processes re-import and re-run your module.

`ProcessPoolExecutor` mirrors the thread API; require picklable targets and guard `__main__`.

Code examples

Threads vs processes on CPU-bound work

import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def crunch(n):
    # pure CPU: sum of squares, no I/O
    return sum(i * i for i in range(n))

tasks = [10_000_000] * 8

if __name__ == "__main__":   # required for the process pool (spawn)
    for Pool, label in [(ThreadPoolExecutor, "threads"), (ProcessPoolExecutor, "processes")]:
        start = time.perf_counter()
        with Pool(max_workers=8) as pool:
            list(pool.map(crunch, tasks))
        print(f"{label}: {time.perf_counter()-start:.2f}s")

# threads:   ~4.0s  (GIL serializes -> no parallelism)
# processes: ~0.6s  (8 cores in parallel)

`crunch` never releases the GIL (no I/O), so the thread pool runs the 8 tasks one-at-a-time — no faster than serial. The process pool runs 8 interpreters on 8 cores at once, so the same work finishes in roughly an eighth of the time.

The __main__ guard and picklable targets

from concurrent.futures import ProcessPoolExecutor

def square(x):        # top-level, importable, picklable
    return x * x

def run():
    with ProcessPoolExecutor() as pool:
        return list(pool.map(square, range(10)))

if __name__ == "__main__":   # without this, spawned children re-run run() -> recursion / error
    print(run())

# NOTE: pool.map(lambda x: x*x, range(10)) would raise
# 'cannot pickle local object' — lambdas/closures aren't picklable.

With the default 'spawn' start method, each child process imports this module fresh. Without the `__main__` guard, importing would re-trigger `run()` in every child. And because arguments/functions are pickled to reach the child, the target must be a top-level function — a `lambda` or nested closure can't be pickled.

Common mistakes

Recall questions

Questions & answers

A data pipeline applies a heavy CPU transform to 10 million rows and is CPU-bound on one core. How do you parallelize it in Python, and what's the expected ceiling?

Use multiprocessing (e.g. `ProcessPoolExecutor` with ~`os.cpu_count()` workers), chunking the rows so each worker transforms a batch — each process has its own GIL and runs on a separate core, so the speedup ceiling is roughly the core count minus serialization/coordination overhead. Keep data crossing the process boundary small (pass chunks/paths, not the whole dataset repeatedly). Threads wouldn't help because the transform is CPU-bound and the GIL would serialize it. For very heavy numeric work, a vectorized/native library (NumPy, Polars) that releases the GIL can beat or complement multiprocessing.

Approach: Identify CPU-bound → multiprocessing, size to cores, chunk to limit pickling, and note the near-linear-minus-overhead ceiling.

An engineer parallelizes a function with `ProcessPoolExecutor` but it's slower than the serial version. Give two plausible causes.

(1) The work is I/O-bound or too small per task, so process startup and pickling overhead dominate the tiny compute — threads/asyncio or plain serial would be faster. (2) Large arguments/results are being serialized across the boundary each call, and that data movement costs more than the parallel compute saves. Related causes: too many workers thrashing cores, or the task itself already using a native library that parallelizes internally.

Approach: Point to overhead exceeding benefit — either work-too-small/I/O-bound, or serialization of large payloads dominating.

Continue learning

Previous: Threading vs Multiprocessing

Return to Python Roadmap