25  Calculus and Automatic Differentiation

Training a network requires derivatives of a scalar loss with respect to its parameters. This chapter follows two tracks. The first three sections develop the differentiation needed for optimization: scalar derivatives, gradients and chain rules, then Jacobian products and automatic differentiation. The final section begins a second track on integration for continuous probability (Chapter 27), expectations, and the differential equations of Chapter 29. Integration is not a prerequisite for backpropagation.

Resources and Further Reading

The following references range from single-variable calculus reviews to matrix calculus and automatic differentiation.

Books

• Mathematics for Machine Learning — Deisenroth, Faisal & Ong — free Cambridge text; Chapter 5, “Vector Calculus,” provides a treatment of gradients, Jacobians, and the chain rule for machine learning.
• The Matrix Cookbook — Petersen & Pedersen — a quick reference for matrix-derivative identities, including \(\partial(\mathbf{x}^\top \mathbf{A}\mathbf{x})\) and friends without re-deriving them.
• Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation — Griewank & Walther — a detailed monograph (SIAM, 2nd ed.) on automatic differentiation, including forward/reverse modes and checkpointing.

Courses and video lectures

• Essence of Calculus — 3Blue1Brown — a twelve-part visual introduction to derivatives, integrals, and the chain rule.
• Single Variable Calculus (18.01SC) — MIT OpenCourseWare — full self-study course with lecture videos and graded problem sets for the one-variable foundations.
• Multivariable Calculus (18.02SC) — MIT OpenCourseWare — partial derivatives, gradients, and vector calculus, the multivariable companion to 18.01.
• Matrix Calculus for Machine Learning and Beyond (18.S096) — Edelman & Johnson, MIT OCW — treats derivatives of matrix-valued maps holistically rather than entry-by-entry; lecture notes and videos included.

Tutorials, notes, and visual introductions

• The Matrix Calculus You Need For Deep Learning — Parr & Howard — a free, self-contained primer that develops exactly the matrix calculus needed to read a backprop derivation, assuming only Calculus 1.
• Calculus on Computational Graphs: Backpropagation — Christopher Olah — a short, diagram-driven post showing how the chain rule on a computational graph is backpropagation.
• matrixcalculus.org — Laue, Mitterreiter & Giesen — an online symbolic calculator for vector and matrix derivatives; useful for checking hand-derived gradients and exporting LaTeX or Python.

Automatic differentiation

• Automatic Differentiation in Machine Learning: a Survey — Baydin, Pearlmutter, Radul & Siskind — the standard survey (JMLR, 2018); read it to understand why AD is neither symbolic nor numerical differentiation.
• The Art of Differentiating Computer Programs — Naumann — SIAM’s introduction to algorithmic differentiation from the compiler’s point of view, by the author of the NP-completeness result for optimal Jacobian accumulation cited in this chapter.
• The Autodiff Cookbook — JAX documentation — grad, JVPs/VJPs, Jacobians, and Hessian-vector products with runnable examples.
• Autodidax: JAX core from scratch — JAX documentation — builds forward- and reverse-mode autodiff (and jit/vmap) from basic components to show how a modern AD system operates.
• A Gentle Introduction to torch.autograd — PyTorch documentation — how PyTorch records a dynamic computational graph and replays it in reverse to compute gradients via .backward().