The Singular Value Decomposition
~7 min read · Tier toggle changes depth (T1 plain · T2 standard · T3 rigour)
You should be comfortable with inner products, orthonormal bases, and the spectral theorem for self-adjoint operators, as covered in previous lectures on quadratic forms and principal axes.
Why this matters
The Singular Value Decomposition (SVD) is the most general factorization of a linear map, revealing its geometric action as a sequence of rotations and scalings.
While the spectral theorem decomposes self-adjoint operators into eigenvalues and eigenvectors, it fails for non-square or non-normal matrices. The SVD extends this idea to any linear map between finite-dimensional inner-product spaces. It provides the mathematical foundation for data compression, signal processing, and the stability analysis of physical systems, allowing us to quantify how a transformation stretches space in its most significant directions.
Geometric Intuition
Imagine a linear map \(A\) acting on a vector space \(\mathcal{V}\). In general, \(A\) does not simply scale vectors along fixed axes; it can rotate, shear, and stretch them in complex ways. However, the SVD tells us that any such map can be understood as a composition of three simpler operations: a rotation (or reflection) in the input space, a pure scaling along orthogonal axes, and a final rotation in the output space.
Specifically, the map first rotates the input basis vectors \(\{v_i\}\) to align with the directions of maximum stretch. It then scales these directions by factors \(\sigma_i\), known as singular values. Finally, it rotates the resulting vectors to align with the output basis \(\{u_j\}\). This decomposition is unique in terms of the singular values, though the singular vectors are unique only up to sign or phase. This geometric clarity makes the SVD indispensable for understanding the "shape" of a linear transformation.
The SVD Statement
Let \(A: \mathcal{V} \to \mathcal{W}\) be a linear map between finite-dimensional inner-product spaces over \(\mathbb{C}\) (or \(\mathbb{R}\)), with \(\dim \mathcal{V} = n\) and \(\dim \mathcal{W} = m\). Then there exist an orthonormal basis \(\{v_i\}\) of \(\mathcal{V}\), an orthonormal basis \(\{u_j\}\) of \(\mathcal{W}\), and real numbers \(\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_p \ge 0\) with \(p = \min(m, n)\), such that in these bases \(A = U \Sigma V^{\dagger}\), where \(U\) and \(V\) are unitary and \(\Sigma \in \mathbb{R}^{m \times n}\) is diagonal with non-negative entries.
The singular values \(\sigma_i\) are the square roots of the eigenvalues of \(A^{\dagger}A\) (or \(AA^{\dagger}\)). The vectors \(v_i\) are the eigenvectors of \(A^{\dagger}A\), and \(u_i\) are the normalized images \(Av_i\). This connection allows us to compute the SVD by solving a standard eigenvalue problem for a self-adjoint operator.
Connection to Self-Adjoint Operators
The SVD is deeply rooted in the spectral theory of self-adjoint operators. Consider the operator \(A^{\dagger}A\), which maps \(\mathcal{V}\) to \(\mathcal{V}\). Since \((A^{\dagger}A)^{\dagger} = A^{\dagger}A\), this operator is self-adjoint. By the spectral theorem, \(A^{\dagger}A\) has an orthonormal basis of eigenvectors \(\{v_i\}\) with real non-negative eigenvalues \(\lambda_i\).
We define the singular values as \(\sigma_i = \sqrt{\lambda_i}\). If \(\sigma_i > 0\), we define \(u_i = \frac{1}{\sigma_i} A v_i\). It can be shown that the set \(\{u_i\}\) is orthonormal. If \(\sigma_i = 0\), the vector \(v_i\) lies in the kernel of \(A\), and we can choose \(u_i\) arbitrarily to complete an orthonormal basis of \(\mathcal{W}\). This construction ensures that \(A v_i = \sigma_i u_i\), which is the core relationship underlying the SVD.
Computing the SVD
To compute the SVD of a matrix \(A\), we first form the Gram matrix \(A^{\dagger}A\). We then find the eigenvalues \(\lambda_i\) and eigenvectors \(v_i\) of this self-adjoint matrix. The singular values are \(\sigma_i = \sqrt{\lambda_i}\). The matrix \(V\) is formed by placing the eigenvectors \(v_i\) as columns. The matrix \(\Sigma\) is a diagonal matrix with \(\sigma_i\) on the diagonal. Finally, the columns of \(U\) are given by \(u_i = \frac{1}{\sigma_i} A v_i\) for non-zero singular values.
This method is numerically stable and widely used in scientific computing. For large matrices, iterative methods such as the Lanczos algorithm are employed to find the dominant singular values and vectors without computing the full decomposition. The SVD is particularly useful for determining the rank of a matrix, as the rank is equal to the number of non-zero singular values.
The derivation
Singular Value Decomposition
Let \(A:\mathcal{V}\to\mathcal{W}\) be a linear map between finite-dimensional inner-product spaces over \(\mathbb{C}\) (or \(\mathbb{R}\)), with \(\dim\mathcal{V}=n\) and \(\dim\mathcal{W}=m\). Then there exist an orthonormal basis \(\{v_i\}\) of \(\mathcal{V}\), an orthonormal basis \(\{u_j\}\) of \(\mathcal{W}\), and real numbers \(\sigma_1\ge\sigma_2\ge\cdots\ge\sigma_p\ge 0\) with \(p=\min(m,n)\), such that in these bases \(A=U\Sigma V^{\dagger}\), where \(U\) and \(V\) are unitary and \(\Sigma\in\mathbb{R}^{m\times n}\) is diagonal with non-negative entries \(\Sigma_{ii}=\sigma_i\).
A common misconception is that the singular vectors \(v_i\) and \(u_i\) are eigenvectors of \(A\). In general, \(A v_i = \sigma_i u_i\), but \(u_i\) is not necessarily a scalar multiple of \(v_i\) unless \(A\) is normal. The eigenvectors of \(A\) exist only if \(A\) is square and diagonalizable, whereas the SVD exists for any linear map. Confusing these two concepts leads to errors in applications such as principal component analysis, where the left singular vectors represent the directions of maximum variance in the data space, not the eigenvectors of the data matrix itself. C
Worked example
Problem. Consider the real \(2 \times 2\) matrix \(A = \begin{pmatrix} 3 & 0 \\ 0 & 4 \end{pmatrix}\). Find the singular values and the matrices \(U\), \(\Sigma\), and \(V\) in the SVD \(A = U \Sigma V^{\dagger}\).
- Compute \(A^{\dagger}A = A^T A = \begin{pmatrix} 3 & 0 \\ 0 & 4 \end{pmatrix} \begin{pmatrix} 3 & 0 \\ 0 & 4 \end{pmatrix} = \begin{pmatrix} 9 & 0 \\ 0 & 16 \end{pmatrix}\).
- Find the eigenvalues of \(A^{\dagger}A\). The characteristic equation is \((9-\lambda)(16-\lambda)=0\), so \(\lambda_1 = 16\) and \(\lambda_2 = 9\).
- Compute the singular values: \(\sigma_1 = \sqrt{16} = 4\) and \(\sigma_2 = \sqrt{9} = 3\). Order them such that \(\sigma_1 \ge \sigma_2\).
- Find the eigenvectors of \(A^{\dagger}A\). For \(\lambda_1 = 16\), the eigenvector is \(v_1 = \begin{pmatrix} 0 \\ 1 \end{pmatrix}\). For \(\lambda_2 = 9\), the eigenvector is \(v_2 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}\).
- Form the matrix \(V = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}\). Note that \(V\) is orthogonal, so \(V^{\dagger} = V^T = V\).
- Compute the columns of \(U\). \(u_1 = \frac{1}{\sigma_1} A v_1 = \frac{1}{4} \begin{pmatrix} 3 & 0 \\ 0 & 4 \end{pmatrix} \begin{pmatrix} 0 \\ 1 \end{pmatrix} = \frac{1}{4} \begin{pmatrix} 0 \\ 4 \end{pmatrix} = \begin{pmatrix} 0 \\ 1 \end{pmatrix}\). Similarly, \(u_2 = \frac{1}{\sigma_2} A v_2 = \frac{1}{3} \begin{pmatrix} 3 & 0 \\ 0 & 4 \end{pmatrix} \begin{pmatrix} 1 \\ 0 \end{pmatrix} = \frac{1}{3} \begin{pmatrix} 3 \\ 0 \end{pmatrix} = \begin{pmatrix} 1 \\ 0 \end{pmatrix}\).
- Form the matrix \(U = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}\). The matrix \(\Sigma = \begin{pmatrix} 4 & 0 \\ 0 & 3 \end{pmatrix}\).
- Verify \(A = U \Sigma V^{\dagger}\). \(U \Sigma V^{\dagger} = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix} \begin{pmatrix} 4 & 0 \\ 0 & 3 \end{pmatrix} \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix} = \begin{pmatrix} 0 & 3 \\ 4 & 0 \end{pmatrix} \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix} = \begin{pmatrix} 3 & 0 \\ 0 & 4 \end{pmatrix} = A\).
Units check. The matrix \(A\) is dimensionless in this abstract example, so the singular values are dimensionless. If \(A\) represented a physical quantity with units, the singular values would carry the same units as \(A\).
Three quick tests
- What is the relationship between the singular values of \(A\) and the eigenvalues of \(A^{\dagger}A\)?
Answer
The singular values \(\sigma_i\) are the square roots of the eigenvalues \(\lambda_i\) of \(A^{\dagger}A\), i.e., \(\sigma_i = \sqrt{\lambda_i}\). - Is the SVD unique?
Answer
The singular values are unique. The singular vectors are unique up to sign (in the real case) or phase (in the complex case), provided the singular values are distinct. - How does the SVD relate to the rank of a matrix?
Answer
The rank of a matrix is equal to the number of non-zero singular values. This provides a robust numerical method for determining the rank, as small singular values can be treated as zero within a tolerance.
Problem set for PU-104 →
Each derivation in this unit ends with five graded problems with full solutions.
Lecture notes drafted with a local model and checked line by line by the editor; every number in the worked example was recomputed.