Concurrency: Multiprocessing
Use every CPU core by running real, separate Python processes
multiprocessing module runs your code in separate operating-system processes, each with its own Python interpreter and its own Global Interpreter Lock (GIL). That is the one reliable way to make CPU-bound Python code run on many cores at once. The price is that processes do not share memory, so every argument and every result must be pickled and shipped across a boundary.
- explain why the GIL forces you to processes (not threads) for CPU-bound work,
- launch work with
Process, aPool, andProcessPoolExecutor, - choose
mapvsimapvsapply_asyncand pick a sensiblechunksize, - move data with
Queue,Pipeandshared_memory, - avoid the classic traps: the
if __name__ == "__main__"guard, unpicklable arguments, and start methods (spawnvsfork).
1. The GIL and why threads do not help CPU work
CPython protects its internal state with the Global Interpreter Lock: only one thread may execute Python bytecode at any instant. For I/O-bound work (waiting on the network, disk or a database) threads are excellent, because a thread releases the GIL while it waits. But for CPU-bound work (crunching numbers, parsing, compression, image processing) threads take turns on a single core and give you no speed-up at all.
Multiprocessing sidesteps the GIL entirely. Each child process is a fresh Python interpreter with its own GIL, so N processes can genuinely run Python bytecode on N cores simultaneously.
| Workload | Best tool | Why |
|---|---|---|
| I/O-bound (requests, files, DB) | threading / asyncio | the GIL is released while waiting; no extra processes needed |
| CPU-bound (math, parsing, encode) | multiprocessing | only separate processes escape the single-GIL bottleneck |
Rule of thumb: if your code is busy computing, reach for processes. If it is busy waiting, reach for threads or async.