Linear Subspaces

PCA can be understood as a reconstruction problem. It searches for two matrices that work together: W compresses an original vector x, and U uses that compressed representation to produce a recovered vector. The recovered vector is not an unrelated replacement for x. It is specifically the result of applying the combined mapping UW to x, written as UWx.

The mapping has two stages. First, W changes x into a compressed representation Wx. Next, U uses Wx to produce UWx. Combining the two stages gives one mapping from the original vector directly to its recovered version: x ↦ UWx. This notation matters because the reconstruction comparison is between x and the output produced by this complete mapping.

PCA chooses U and W to minimize the total squared distance between original vectors and their recovered versions. For an original vector x, the recovered version used in that comparison is UWx. Across the data being reconstructed, PCA therefore prefers a matrix pair whose combined mapping produces recovered vectors close to the corresponding originals under the squared-distance objective.

The minimizing solution has two connected properties. First, the columns of U are orthonormal, expressed by UᵀU = I. Second, W is the transpose of U, expressed by W = Uᵀ. Thus W is not an unrelated matrix selected independently from U. The two matrices have a precise relationship, and their combined mapping is UW.

Fix U and W, then let x vary over possible input vectors. Every resulting recovered vector has the form UWx. The collection of all such outputs is the range of the mapping UW. The source describes this range as an n-dimensional linear subspace of Rᵈ.

This geometric view changes how to interpret PCA. PCA is not merely producing separate recovered vectors one at a time. It selects a matrix structure whose possible recovered outputs occupy one shared n-dimensional linear subspace of Rᵈ. Each particular input x produces one point in that subspace through UWx.