24  Calculus and Automatic Differentiation

Training a network means asking, repeatedly, how a scalar loss changes when we nudge a parameter. This chapter builds the calculus that answers it: derivatives and the local-linear view in one variable, gradients and the chain rule in many, the matrix-calculus and automatic-differentiation machinery that makes backpropagation cheap, and finally integration—the operation we will need for probability (Chapter 26) and for the differential equations of Chapter 28.

Resources and Further Reading

If you want to go deeper, the following are the canonical references we recommend, from refreshers on single-variable calculus to the matrix calculus and automatic-differentiation machinery behind backpropagation.

Books

• Mathematics for Machine Learning — Deisenroth, Faisal & Ong — free Cambridge text; Chapter 5, “Vector Calculus,” is the cleanest treatment of gradients, Jacobians, and the chain rule aimed squarely at ML.
• The Matrix Cookbook — Petersen & Pedersen — the standard quick-reference for matrix-derivative identities; reach for it when you need \(\partial(\mathbf{x}^\top \mathbf{A}\mathbf{x})\) and friends without re-deriving them.
• Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation — Griewank & Walther — the definitive monograph (SIAM, 2nd ed.) on automatic differentiation, including forward/reverse modes and checkpointing.

Courses and video lectures

• Essence of Calculus — 3Blue1Brown — a twelve-part visual series that builds derivatives, integrals, and the chain rule from intuition; the best on-ramp if calculus feels rusty.
• Single Variable Calculus (18.01SC) — MIT OpenCourseWare — full self-study course with lecture videos and graded problem sets for the one-variable foundations.
• Multivariable Calculus (18.02SC) — MIT OpenCourseWare — partial derivatives, gradients, and vector calculus, the multivariable companion to 18.01.
• Matrix Calculus for Machine Learning and Beyond (18.S096) — Edelman & Johnson, MIT OCW — treats derivatives of matrix-valued maps holistically rather than entry-by-entry; lecture notes and videos included.

Tutorials, notes, and visual introductions

• The Matrix Calculus You Need For Deep Learning — Parr & Howard — a free, self-contained primer that develops exactly the matrix calculus needed to read a backprop derivation, assuming only Calculus 1.
• Calculus on Computational Graphs: Backpropagation — Christopher Olah — a short, diagram-driven post showing how the chain rule on a computational graph is backpropagation.
• matrixcalculus.org — Laue, Mitterreiter & Giesen — an online symbolic calculator for vector and matrix derivatives; useful for checking hand-derived gradients and exporting LaTeX or Python.

Automatic differentiation

• Automatic Differentiation in Machine Learning: a Survey — Baydin, Pearlmutter, Radul & Siskind — the standard survey (JMLR, 2018); read it to understand why AD is neither symbolic nor numerical differentiation.
• The Art of Differentiating Computer Programs — Naumann — SIAM’s introduction to algorithmic differentiation from the compiler’s point of view, by the author of the NP-completeness result for optimal Jacobian accumulation cited in this chapter.
• The Autodiff Cookbook — JAX documentation — grad, JVPs/VJPs, Jacobians, and Hessian-vector products with runnable examples.
• Autodidax: JAX core from scratch — JAX documentation — builds forward- and reverse-mode autodiff (and jit/vmap) from the ground up, demystifying what a modern AD system actually does.
• A Gentle Introduction to torch.autograd — PyTorch documentation — how PyTorch records a dynamic computational graph and replays it in reverse to compute gradients via .backward().