How much of something is there in total Riemann sums · the fundamental theorem · change of variables · probability.
Why integration
Motivation
Differentiation answered a local question: how does f change when we nudge x? Integration answers a global one, how much is there in total, and the fundamental theorem joins the two together.
An area under a curve, a volume under a surface.
Every continuous probability is an integral, \int p = 1.
Every expectation is an integral average, \int x\,p(x)\,dx.
The change-of-variables rule computes the Gaussian normalizer.
01
The integral and the theorem
a limit of sums, and how to compute it
The definite integral as a limit
The integral
Archimedes’ recipe: chop [a,b] into N slices of width \epsilon, stand a rectangle of height f(x_i) on each, and add. As the slices shrink, the staircase squeezes onto the curve:
The continuous analogue of a sum: \sum \to \int, spacing \epsilon \to dx.
Watching the limit converge
The integral
The definition converges, but slowly. Refine the partition for \int_0^2 \tfrac{x}{1+x^2}\,dx = \tfrac12\log 5 and the left-rule error shrinks in step with \epsilon:
First-order convergence: cut \epsilon by ten, cut the error by ten. It works, but it is slow and gives no closed form. We need a better idea.
The fundamental theorem of calculus
The integral
Let the upper limit move: the area-so-far function F(x)=\int_a^x f accumulates signed area out to x. Nudge x by \epsilon and only one thin sliver of new area appears, of height f(x), so
\frac{dF}{dx}(x) = f(x).
The rate at which area grows is the current height. Finding areas becomes the search for an antiderivative.
Integration is differentiation reversed
The integral
If G'=f then every definite integral is a difference of two values:
\int_a^b f(x)\,dx = G(b) - G(a).
Hard sums become table lookups. We check it numerically: build F as a cumulative Riemann sum, finite-difference it, and compare against f itself.
max |dF/dx - f| : 5.551115123125783e-14
Riemann F(2) vs G(2)-G(0): 0.8045188628837103 0.8047189562170501
The error is float32 roundoff, not discretization: differencing the cumulative sum telescopes back to the term it just added.
Improper integrals
The integral
Densities live on unbounded domains, so we integrate to infinity as a limit, \int_a^\infty f = \lim_{b\to\infty}\int_a^b f. The power law is the test case:
The partials approach 1 as b grows (truncation error e^{-b}); the left rule settles near 1.0005, not 1 (discretization error \approx\tfrac{\epsilon}{2}). Two independent knobs: neither fixes the other.
Integration by parts
The integral
The product rule, run backwards through the theorem:
With the right u this collapses hard integrals to trivial ones.
03
Multiple integrals
double integrals, Fubini, the Jacobian, and the Gaussian
Multiple integrals: a volume under a surface
Double integrals
For f(x,y) on a box, \int_U f\,d\mathbf{x} is the volume between the surface and the base plane. The rectangle recipe survives: tile the base with \epsilon\times\epsilon squares, stand a box on each, and total the volumes.
Our running example is the bell e^{-x^2-y^2} over [-2,2]^2.
The box volume, numerically
Double integrals
Total the boxes on a fine grid over [-2,2]^2. The exact value involves the error function\operatorname{erf}, the Gaussian’s antiderivative, which has no elementary formula but ships as a library routine:
Riemann volume over the box [-2, 2]^2 : 3.11227
exact volume pi * erf(2)^2 : 3.11227
Over the whole plane the volume becomes exactly \pi, computed below.
Fubini: sum in any order
Fubini
A grid sum totals in any order. Summing columns-first and passing to the limit splits the double integral into iterated single ones:
The determinant is exactly the factor by which a linear map scales volume, so a tiny cube of volume d\mathbf{x} becomes a parallelepiped of volume |\det D\boldsymbol{\phi}|\,d\mathbf{x}.
Read for densities instead of areas, this same factor (as -\log|\det D\boldsymbol{\phi}|) is the engine behind normalizing flows.
The Gaussian integral
The Gaussian integral
The one-dimensional \int_{-\infty}^\infty e^{-x^2}\,dx has no elementary antiderivative. Go up a dimension: square it, merge the copies by Fubini, then switch to polar coordinates,
The polar map contributes Jacobian |\det D\boldsymbol{\phi}|=r, the factor that makes the integrand elementary. So \int e^{-x^2}\,dx=\sqrt\pi:
grid integral of e^(-x^2-y^2): 3.141593
exact value pi : 3.141593
04
Integration meets probability
densities, expectations, and Monte Carlo
Densities and expectations
Probability
This is why a deep-learning reader needs integration. A continuous density is a non-negative function that is normalized; an expectation is an integral average:
The Gaussian integral supplies the normalizer: p(x)=\tfrac{1}{\sqrt\pi}e^{-x^2} integrates to 1, and symmetry sends its mean to 0.
total mass integral p dx : 1.0
mean integral x p dx: 0.0
Monte Carlo beats the curse of dimensionality
Probability
With no closed form, Monte Carlo estimates an expectation by sampling, \mathbb{E}[g(X)] \approx \tfrac1n\sum_i g(x_i), at rate 1/\sqrt nin every dimension (the constant, the standard deviation of g(X), may grow with d; the exponent never does).
A grid to resolution \epsilon in d dimensions costs N=\epsilon^{-d} points and decays only as N^{-2/d}: the exponent is divided by d. Past a handful of dimensions, sampling is the only practical choice.
n=10 Monte Carlo=0.87741 error=1.3e-01
n=100 Monte Carlo=0.75454 error=7.7e-03
n=1000 Monte Carlo=0.74619 error=6.3e-04
n=10000 Monte Carlo=0.74833 error=1.5e-03
n=100000 Monte Carlo=0.74719 error=3.7e-04
midpoint quadrature =0.74682
Differentiating under the integral
Probability
Training through an expectation means swapping \nabla_{\boldsymbol\theta} and \int; the plain swap is what shows the score \nabla_{\boldsymbol\theta}\log
p_{\boldsymbol\theta} has mean zero, the identity at the heart of policy gradients. When the limits move too, the Leibniz rule adds the boundary terms:
