The Data Stack: NumPy and pandas
Arrays, DataFrames and vectorization — the doorway from general Python into data work
for loops are flexible but slow for numeric work, because every element is a full Python object and every operation runs interpreted bytecode. NumPy stores numbers in one contiguous, typed block and pushes whole-array operations down into compiled C — this is vectorization. pandas builds a labeled, columnar table (the DataFrame) on top of NumPy so you can select, align, group and aggregate real-world data by name instead of by hand. Together they are the foundation the rest of the Python data-science ecosystem stands on.
- explain why a NumPy
ndarrayis faster and smaller than a list of numbers, - replace element-by-element loops with vectorized array expressions,
- reason about dtype, contiguous memory, and views vs copies,
- apply broadcasting rules to operate on differently shaped arrays,
- build and query a pandas
DataFramewithloc/ilocand boolean masks, - summarise data with the split-apply-combine pattern (
groupby), - decide when to reach for the data stack and when plain Python is the better tool.
1. Why a data stack exists: the cost of a Python loop
You already know how to sum a million numbers with a comprehension or a loop. It works, but it is expensive. In CPython, a list of integers is really an array of pointers, each aimed at a separate heap-allocated int object with its own header. Adding two lists element-by-element means: dereference a pointer, unbox an object, dispatch __add__, box the result, store a new pointer — per element, in interpreted bytecode.
# Pure-Python element-wise add of two lists
def add_lists(a, b):
return [x + y for x, y in zip(a, b)] # a Python object op per elementNumPy attacks all three costs at once: it stores the numbers as raw machine values (no per-element objects), keeps them contiguous in memory (cache-friendly), and performs the loop itself in compiled C. A vectorized a + b on two arrays of a million elements is typically tens of times faster than the comprehension above — and the code is shorter.