Concurrency Patterns: Thread Pools and Producer-Consumer
From raw threads to reusable, deadlock-free concurrency designs
Big Idea: You already know how to launch a
std::thread, guard shared state with a std::mutex, and reason about the memory model with atomics. Those are the primitives. Real systems rarely spawn a thread per task and rarely hand-roll synchronisation at every call site. Instead they assemble a small number of reusable patterns: a thread pool that owns a fixed set of workers, a thread-safe queue that hands work between them, and the producer-consumer handshake built on a std::condition_variable. Master these three and you can build almost any concurrent pipeline without leaking threads, busy-waiting, or racing on shared data.
By the end you will be able to:
- explain why spawning one thread per task does not scale, and what a thread pool fixes;
- build a bounded, thread-safe queue using
std::mutexandstd::condition_variable; - implement the producer-consumer pattern with correct blocking and wake-up;
- assemble a small thread pool that accepts tasks and returns a
std::future; - recognise and prevent the classic hazards: lost wake-ups, spurious wake-ups, deadlock, and data races.
1. Why not one thread per task?
The naive approach to concurrency is: for every unit of work, call std::thread(doWork). It is easy to write and it is almost always wrong at scale. Threads are not free:
- Creation cost: spawning an OS thread costs microseconds and a fresh stack (often 1 MB reserved). A request handler that creates a thread per request pays this repeatedly.
- Oversubscription: if 10,000 tasks each spawn a thread on an 8-core machine, the scheduler thrashes context-switching between far more threads than cores, and throughput drops.
- Unbounded resource use: nothing caps how many threads exist, so a burst of work can exhaust memory or OS handles.
A thread pool inverts the relationship. It creates a fixed number of long-lived worker threads once (typically std::thread::hardware_concurrency()), then feeds them a stream of small tasks through a shared queue. Workers loop: take a task, run it, repeat. The threads are reused, the count is bounded, and the queue naturally smooths out bursts. This is the pattern behind virtually every server, thread-based async runtime, and parallel algorithm library.
The mental shift: stop thinking "a thread is a task" and start thinking "a thread is a worker that consumes tasks." Tasks become data; threads become fixed infrastructure.