Streams in Depth
Process data piece by piece instead of loading it all into memory
- tell the four stream types apart -- Readable, Writable, Duplex, Transform;
- consume a Readable in flowing mode, paused mode, and with
for await; - connect streams with
pipe()andpipeline(); - explain backpressure and why it protects memory;
- write your own Transform to reshape data as it flows.
1. Why streams exist
Imagine you must copy a 4 GB video file. The naive approach reads the whole file into a variable, then writes that variable out:
const data = fs.readFileSync('movie.mp4'); // 4 GB now lives in RAM
fs.writeFileSync('copy.mp4', data);This works for small files but is a disaster for large ones: your process needs 4 GB of memory just to hold the buffer, and nothing happens until the entire read finishes. Streams solve both problems by moving the data in chunks (typically 64 KB at a time). Memory stays tiny and constant, and work starts on the first chunk immediately.
Analogy. A non-stream approach is like emptying a swimming pool by carrying the whole pool to a new location. A stream is a hose: water flows through continuously, you never hold more than what is in the hose at any instant, and the far end starts filling right away.
Because streams are an interface (an agreed set of methods and events), the same code that reads a file can read from a network socket or a decompressor. That composability is their real power.