24 Linear Algebra
Vectors represent data and parameters, while matrices represent the linear maps used throughout a network. We begin with the geometry of vectors, dot products, projections, and matrix transformations. We then study two matrix decompositions used repeatedly in deep learning. Eigendecompositions support stability analysis, PCA, and the analysis of loss curvature; singular value decompositions support low-rank approximation, conditioning, and parameter-efficient fine-tuning.
The three parts answer progressively more specific questions:
| Object | Question | Principal tool |
|---|---|---|
| Vectors and subspaces | Which directions agree, and what information does a map preserve or discard? | Angles, projections, range, and null space |
| Repeated square maps | Which directions evolve independently under iteration? | Eigenvalues and eigenvectors |
| An arbitrary rectangular map | Which input directions are amplified most, and how well can the map be approximated at low rank? | Singular values and left/right singular vectors |
The eigendecomposition requires a square matrix and may lack a complete eigenbasis. The SVD uses separate input and output bases and exists for every finite matrix; this distinction organizes the chapter.
Resources and Further Reading
The references below develop the geometry, decompositions, and numerical methods used in this chapter. Most are freely accessible online.
Books
- Mathematics for Machine Learning — Deisenroth, Faisal & Ong — free PDF; chapters 2 (Linear Algebra), 3 (Analytic Geometry), and 4 (Matrix Decompositions) cover this chapter’s material from a machine-learning perspective.
- Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares (VMLS) — Boyd & Vandenberghe — free PDF, slides, and code; an applications-first treatment organized around least squares and data.
- Introduction to Linear Algebra, 6th ed. — Gilbert Strang — a foundational text (a commercial book; sample chapters and problems are online) covering the four subspaces, eigenvalues, and the SVD.
- Linear Algebra and Learning from Data — Gilbert Strang — the SVD-centric companion to MIT 18.065, focused on matrix methods for signal processing and deep learning. It assumes the foundations developed in Strang’s introductory text and closely matches this chapter’s SVD and low-rank material.
- Linear Algebra Done Right, 4th ed. — Sheldon Axler — free, open-access; a rigorous, determinant-free path to eigenvalues and the spectral theorem for readers who want the proofs.
- Numerical Linear Algebra — Trefethen & Bau — covers conditioning, stability, QR, and floating-point algorithms for the SVD and eigendecomposition.
Courses and video lectures
- MIT 18.06 Linear Algebra (OCW) — Gilbert Strang — a full-semester lecture series with notes, exams, and solutions.
- MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning (OCW) — Gilbert Strang — the graduate follow-on that connects linear algebra directly to deep learning and optimization.
- Essence of Linear Algebra — 3Blue1Brown — a short, visually driven series that builds geometric intuition for vectors, transformations, determinants, and eigenvectors.
- Singular Value Decomposition — Steve Brunton — a lecture series on SVD geometry, low-rank approximation, PCA, and randomized SVD; it accompanies Data-Driven Science and Engineering.
Tutorials, notes, and visual introductions
- Linear Algebra Review and Reference — Stanford CS229 (Kolter, updated by Do) — a compact PDF refresher covering the notation and matrix calculus assumed in machine-learning courses.
- Immersive Linear Algebra — Ström, Åström & Akenine-Möller — a free online textbook with interactive figures for exploring transformations and eigenvectors.
- LoRA: Low-Rank Adaptation of Large Language Models — Hu et al. — applies low-rank structure to adapt large models by training only a low-rank update to the weight matrices.