Proof. Insert \pm\mathbf m(X); the cross term vanishes by the tower rule. \blacksquare Least squares against a noisy target fits its conditional mean: used once for scores, once for velocities.
Denoising score matching & Tweedie
Denoising
Perturb \tilde{\mathbf x}=\mathbf x+\sigma\boldsymbol\epsilon; the conditional score is closed-form -\boldsymbol\epsilon/\sigma, and regressing on it (Vincent) recovers the marginal score. Rearranged, that is Tweedie:
One step up the score lands exactly on the posterior mean.
Estimating the score and optimal denoising are the same function.
A score network in 1-D
Denoising
A tiny MLP trained by denoising score matching matches the analytic score, landing on the irreducible loss floor:
DSM loss 2.036 vs irreducible floor 2.038; max |s_theta - score| on [-4, 4]: 0.199
02
Score-based diffusion
forward noise, DDPM, Langevin, DDIM, guidance
One noise level → all of them
Forward process
Small \sigma approximates p but ignores empty space; large \sigma covers but blurs. The fix: a noise-conditional score \mathbf s_\theta(\mathbf x,t) trained along a forward SDE (VE or VP), then a reverse pass to generate:
A weighting \lambda(t) allocates effort across noise levels: \lambda = g^2 makes the loss a likelihood bound; DDPM’s 1-\bar\alpha_t trades that for sample quality.
Two clocks
Conventions
Diffusion runs data→noise and samples backward; flow matching runs noise→data and samples forward. To compare, substitute t\to 1-t:
DDPM is three propositions
DDPM
The DDPM step is Euler–Maruyama on the VP-SDE (to O(\beta_t)).
The simple \|\boldsymbol\epsilon-\boldsymbol\epsilon_\theta\|^2 loss is denoising score matching, reweighted by \lambda(t)=1-\bar\alpha_t.
t = 10: Var(x_t) chain 1.000, formula 1.000, alpha_bar 0.9981
t = 100: Var(x_t) chain 0.994, formula 1.000, alpha_bar 0.8970
t = 1000: Var(x_t) chain 1.024, formula 1.000, alpha_bar 0.0000
chain vs one-shot at T: mean -0.027 vs +0.005, std 1.012 vs 1.000
Langevin: stationary but slow
Sampling
dX = \tfrac12\nabla\log p\,dt + dW has stationary density p (substitute \rho=p into Fokker–Planck → 0). But it mixes slowly across modes:
From one mode, almost no walker crosses (P(X>0)=0.012). Fix: anneal the noise, or use predictor–corrector.
DDIM: slide along your own curve
Sampling
Invert the marginal for \hat{\mathbf x}_0 = \bigl(\mathbf x_t - \sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon_\theta\bigr)/\sqrt{\bar\alpha_t}, then re-use the predicted noise instead of resampling:
Each sample slides deterministically along its own curve: strides can skip levels.
Ten strides for a thousand staggers
Sampling
On the closed-form mixture (no learning in the loop), ten strides land every sample in the same mode as the thousand-step reference; by fifty the terminal laws are statistically indistinguishable:
10 strides vs 1000: mean |gap| 0.083, KS 0.080, mode fraction 0.495
50 strides vs 1000: mean |gap| 0.015, KS 0.018, mode fraction 0.495
mode fraction at 1000 steps: 0.495; 5% KS threshold: 0.021
Same network; \eta controls reinjected noise. In the fine-step limit, deterministic DDIM is related to the probability-flow ODE; finite strides are numerical approximations, including for Gaussian marginals.
Classifier-free guidance trains one network with label dropout and extrapolates through the conditional: \tilde{\mathbf s} = (1-\gamma)\,\mathbf s_\varnothing + \gamma\,\mathbf s_y.
Measured on the closed-form mixture: \gamma=1 reproduces the exact conditional (mean 0.966 vs the analytic 0.970); at \gamma=3, 10 there is no more mass to move, so the mode distorts, drifting to 1.04, then 1.07, and narrowing.
For \gamma>1 the tilt p_t(\mathbf x)\,p_t(y\mid\mathbf x)^\gamma is the noised marginal of no clean distribution: a useful controlled distortion, not a consistent diffusion.
03
Flow matching
prescribe the path, regress the velocity, translate the targets
Probability paths and velocities
Flow matching
Prescribe a path (p_t) from noise to data; its velocity obeys the continuity equation. The intractable marginal velocity is again a posterior mean:
The tractable CFM loss (closed-form per-pair velocity) and the intractable FM loss have the same gradients.
Proof. Apply the regression lemma with target \mathbf u_t(\mathbf x\mid\mathbf z); its conditional mean is the marginal velocity. \blacksquare Identical structure to Vincent’s theorem.
Score, noise, velocity: one function
The dictionary
On a Gaussian path \mathbf x_t = \alpha_t\,\mathbf x_1 + \sigma_t\,\boldsymbol\epsilon, the marginal velocity and the marginal score determine each other:
The invariant clock is the log-SNR \lambda_t = \log(\alpha_t^2/\sigma_t^2): schedules covering the same \lambda range are re-clockings of the same model: EDM’s \sigma(t)=t is exactly such a re-clocking.
Rectified flow: straight paths
Flow matching
The simplest path is a straight line, \mathbf x_t=(1-t)\mathbf x_0+t\mathbf x_1, with constant target \mathbf x_1-\mathbf x_0. Conditional paths are straight; the marginal flow bends only where paths cross:
Gaussian → two moons
Flow matching
A small MLP trained by the rectified-flow loss, then Euler-integrated, sharpens the crescents as step count grows:
Integrate the trained ODE once, keep the couplings (\mathbf z, \hat{\mathbf x}_1(\mathbf z)), and retrain the same architecture on those pairs, which now almost never cross:
final reflow loss 0.003
1 step(s): energy distance CFM 0.603 -> reflow 0.003
2 step(s): energy distance CFM 0.129 -> reflow 0.003
32 step(s): energy distance CFM 0.003 -> reflow 0.003
One Euler step scores 0.016: within noise of the original’s 32-step quality, 40\times better than its one-step 0.676. The loss floor collapses too: the coupling’s posterior variance is gone.
Every method here is the same template: a probability path, a closed-form conditional regression target, and a numerical integrator.
DDPM, score-SDE, PF-ODE, DDIM, and flow matching differ only in the path, the loss reweighting, and whether the sampler injects noise; the stochastic interpolants framework writes the whole family in one formalism.
