Naive Bayes

Dive into Deep Learning · §26.7

The chapter’s ideas at work in one system
count, smooth, argmax, then check the result.

Why not fit p(y | x) directly?

Motivation

A classifier wants \hat y = \operatorname{argmax}_y p(y\mid\mathbf x). Estimating that table directly is hopeless: 2^d feature patterns (2^{784} for MNIST, more than atoms in the universe).

The plan: maximum likelihood (the maximum-likelihood section) fits a generative model by counting; Bayes flips it into a classifier; the statistics of the statistics section then judge the result: error bar, failure map, calibration.

Bayes’ rule → a generative classifier

The route

Flip to the generative direction; the label-independent denominator drops:

\hat y = \operatorname*{argmax}_y\, p(y\mid\mathbf x) = \operatorname*{argmax}_y\, p(\mathbf x\mid y)\,p(y).

We still owe a model for p(\mathbf x\mid y), a distribution over all 2^d patterns. No savings yet.

Generative vs. discriminative

Two routes

Generative (here): learn class-conditionals p(\mathbf x\mid y) + prior, flip through Bayes.
Discriminative (softmax / logistic): fit the boundary p(y\mid\mathbf x) directly.

In some matched model comparisons, generative fitting reaches its error floor sooner while discriminative fitting reaches a lower floor; discriminative wins given enough data.

The naive assumption

conditional independence, and why it helps

The factorization

Independence

Assume features are conditionally independent given the label:

p(\mathbf x\mid y) = \prod_{i=1}^d p(x_i\mid y).

This collapses \mathcal O(2^d) parameters to \mathcal O(d) per class: the curse of dimensionality broken by fiat.

The graphical model

Independence

The label fans out to every feature; no edges run between features. The right panel shows the dependence we throw away.

The assumption is false (pixels are correlated), but the classifier only needs the argmax right, not calibrated probabilities.

Log space and linearity

underflow, the log-sum, and why naive Bayes is linear

Predict in log space

Numerical

Multiplying 784 probabilities underflows: in single precision nearly every class score becomes an exact zero. Since \log is increasing it preserves the argmax and turns the product into a sum:

\hat y = \operatorname*{argmax}_y\, \log p(y) + \sum_{i=1}^d \log p(x_i\mid y).

The score is affine → linear classifier

Linearity

For binary pixels, \log p(x_i\mid y) = x_i\log p_{iy} + (1-x_i)\log(1-p_{iy}), so each class score is

s_y(\mathbf x) = \mathbf w_y^\top\mathbf x + b_y.

Naive Bayes draws the same kind of decision hyperplanes as softmax regression; only the way it fits the weights differs.

Training is counting

priors, likelihoods, and Laplace smoothing

Maximum likelihood = frequencies

Counting

Both ingredients are counts: the class prior \hat p(y)=n_y/n and the per-pixel firing rate, Laplace-smoothed as \hat p(x_i{=}1\mid y) = \tfrac{n_{iy}+1}{n_y+2} so a never-seen pixel cannot send a log-score to -\infty:

n_y = np.array([(Y == y).sum() for y in range(10)])
P_y = n_y / n_y.sum()                                  # class prior, p(y)

n_x = np.stack([X[Y == y].sum(axis=0) for y in range(10)])
P_xy = (n_x + 1) / (n_y + 2).reshape(10, 1, 1)         # Laplace-smoothed p(x_i=1|y)
P_y

array([0.09871667, 0.11236667, 0.0993    , 0.10218333, 0.09736667,
       0.09035   , 0.09863333, 0.10441667, 0.09751667, 0.09915   ])

The +1/+2 is the posterior mean under a \text{Beta}(1,1) prior: pseudo-observations as regularization.

MNIST, end to end

binarize, learn templates, 84% accuracy

Load and binarize

MNIST

Threshold each 28\times28 grayscale image at 128 to get binary pixels x_i\in\{0,1\}:

def load(train):
    ds = torchvision.datasets.MNIST(root='../data', train=train, download=True)
    X = np.floor(ds.data.numpy() / 128).astype('float32')
    return X, ds.targets.numpy()

X, Y = load(train=True)
X_test, Y_test = load(train=False)
X.shape, Y.shape

((60000, 28, 28), (60000,))

A Gaussian naive Bayes would instead keep a per-class mean and variance for each continuous pixel.

What the model learns

MNIST

Each class is just an averaged template. The blur is the naive assumption: per-pixel marginals, nothing about co-occurrence.

d2l.show_images([P_xy[y] for y in range(10)], 2, 5,
                titles=[str(y) for y in range(10)]);

Classify and evaluate

MNIST

Sum the log-likelihoods, take the argmax, and measure what the raw products would have done:

float32 underflow: 99.1% of class scores; smallest float64 survivor = 1e-323
0.8427

84.27%: far above 10\% chance, far below modern nets (<1\% error). The gap traces back to the wrong independence assumption.

Where it shines: text

Domain

Pixels are tightly coupled, so the assumption hurts. Words in a bag-of-words model are far closer to independent: naive Bayes dominated spam filtering for decades.

The multinomial event model counts word occurrences (with a +|V| denominator) instead of presence/absence.

Held to the chapter’s standards

error bar · failure map · calibration

How precise is 84.27%?

Error bar

An accuracy is an estimate from 10{,}000 random test examples, so it carries a standard error, and the statistics section’s bootstrap delivers it: resample the test set, recompute, read off the spread:

test accuracy = 0.8427, bootstrap 95% CI = (0.8354, 0.8496)

The report is 84.3\% \pm 0.7: the third decimal is noise. Comparing against a competitor needs a paired test on the shared test set; a model inside this interval can still be significantly better.

The failure map

A confusion matrix localizes the missing 16%

most confused true->predicted: 4->9 (138), 5->3 (106), 8->3 (68), 7->9 (67)

The worst failures (4\to9, 5\to3, 8\to3) largely track the pairs whose learned templates overlap most: their difference lives in the joint behavior of neighboring pixels, invisible to marginals.

The confidence cannot be trusted

Calibration

Normalized scores are a genuine posterior. Do the claimed probabilities match reality? Bin test examples by claimed confidence and compare with achieved accuracy:

mean claimed confidence = 0.986,   actual accuracy = 0.843
confidence in (0.0, 0.9]:   447 examples,  claimed 0.7229,  achieved 0.3736
confidence in (0.9, 0.99]:   455 examples,  claimed 0.9606,  achieved 0.4747
confidence in (0.99, 0.999]:   406 examples,  claimed 0.9960,  achieved 0.5640
confidence in (0.999, 1.0]:  8692 examples,  claimed 1.0000,  achieved 0.8991

Mean claim 98.6\%, delivery 84.3\%; the top bin asserts near-certainty and is right 89.9\% of the time.

Each of 784 correlated pixels contributes its evidence as if independent, so it is counted many times over: the median best-vs-second score gap is about 30 nats, already enough to saturate the softmax. Modern nets are overconfident in the same direction, though less severely: run this reliability check on any probabilities you intend to consume.

Recap

Wrap-up

Bayes’ rule + conditional independence = naive Bayes.
Breaks the curse of dimensionality: \mathcal O(d) not \mathcal O(2^d) parameters.
Training is one counting pass; smooth (a \text{Beta} prior), predict in log space.
The log-space score is affine → a linear classifier, like softmax.

Great where features are near-independent (text); weak on images, where they are not.
Checked with the chapter’s own tools: 84.3\%\pm0.7 (bootstrap), failures where templates overlap (confusion), confidence wildly inflated (calibration).

Never ship an accuracy without its error bar, its failure map, and a calibration check.