1. Why SVD? — Transcending Eigendecomposition
Callback — the limitation we are about to overcome. Eigendecomposition $A = X\Lambda X^{-1}$ has three serious restrictions:
- $A$ must be square ($n \times n$).
- Even when square, $A$ may be defective — not enough linearly independent eigenvectors to form $X$.
- Eigenvalues may be complex, even for a real matrix (recall the rotation matrix $\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ with eigenvalues $\pm i$).
The SVD Promise
Every matrix $A \in \mathbb{R}^{m \times n}$ — square or rectangular, full-rank or singular — admits a factorization
$$A = U\,\Sigma\,V^{\!\top}$$where
- $U \in \mathbb{R}^{m \times m}$ is an orthogonal matrix (its columns $\mathbf{u}_1,\dots,\mathbf{u}_m$ form an orthonormal basis of $\mathbb{R}^m$).
- $V \in \mathbb{R}^{n \times n}$ is an orthogonal matrix (its columns $\mathbf{v}_1,\dots,\mathbf{v}_n$ form an orthonormal basis of $\mathbb{R}^n$).
- $\Sigma \in \mathbb{R}^{m \times n}$ is a diagonal matrix with entries $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_{\min(m,n)} \ge 0$ on the diagonal.
The numbers $\sigma_i$ are the singular values. The columns of $U$ are the left singular vectors; the columns of $V$ are the right singular vectors.
The Geometric Sentence
Read the factorization right-to-left as a sequence of three operations on a vector $\mathbf{x}$:
$$A\mathbf{x} = U\,(\Sigma\,(V^{\!\top}\mathbf{x}))$$- $V^{\!\top}$: Rotate (in the input space $\mathbb{R}^n$). Express $\mathbf{x}$ in the orthonormal basis $\{\mathbf{v}_1,\dots,\mathbf{v}_n\}$.