How SVD and PCA compress millions of data dimensions with minimal information loss

Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) are two foundational linear algebra techniques that power recommendation systems, image compression, and much of modern machine learning. A book titled 'Mathematics of Data Science,' published on arXiv on July 11, 2026, by Afonso Bandeira, Amit Singer, and Thomas Strohmer, dedicates a full chapter to these methods among 16 chapters covering mathematical foundations of data science. SVD factorizes any matrix into three component matrices, while PCA applies SVD to identify the directions of greatest variance in high-dimensional data, enabling significant dimensionality reduction. The Eckart-Young theorem, detailed in the book, mathematically proves that retaining the top k singular values yields the best possible rank-k approximation of the original matrix. Together, these techniques allow practitioners to compress data, remove noise, and accelerate model training without destroying essential information.