4  Multilayer Perceptron

A multilayer perceptron (MLP) composes fully connected layers with nonlinear activation functions. This simple change extends linear models to nonlinear functions and provides the first deep networks in the book.

This chapter derives and implements MLPs, explains backpropagation through their computational graphs, and studies parameter initialization and numerical stability. It then revisits generalization and introduces dropout before applying the resulting methods to house-price prediction. Later chapters treat computational efficiency and scaling.

Resources and Further Reading

The references below develop the multilayer perceptron and the core mechanics of training deep networks that this chapter introduces: hidden layers and activation functions, the universal approximation theorem, backpropagation, initialization and numerical stability, modern generalization, and dropout. Later parts of the book develop the optimizers, normalization layers, and architectures these topics point toward; the sources here go deeper on the foundations. All are freely accessible online except where noted.

Books

• Deep Learning — Goodfellow, Bengio & Courville — free HTML; Chapter 6 (Deep Feedforward Networks) is the canonical reference for hidden units, activation functions, the universal approximation theorem, and backpropagation as reverse-mode differentiation; Chapter 7 covers dropout and regularization.
• Neural Networks and Deep Learning — Michael Nielsen — free online; Chapter 2 derives the four equations of backpropagation from first principles step by step, complementing this chapter’s backpropagation section.
• Understanding Deep Learning — Simon J. D. Prince — free PDF; a modern (2023), figure-rich treatment whose early chapters on shallow and deep networks, initialization, and training map directly onto this chapter.
• Deep Learning: Foundations and Concepts — Bishop & Bishop — free online edition; a probabilistically-flavoured (2024) account of MLPs, backpropagation, initialization, and regularization.

Courses and video lectures

• CMU 11-785 Introduction to Deep Learning — Bhiksha Raj et al. — free slides and recordings; devotes multiple lectures to the perceptron, universal approximation, and backpropagation at exactly this chapter’s depth.
• Stanford CS231n: Deep Learning for Computer Vision — free course notes; the “Neural Networks” and “Backpropagation, Intuitions” modules are one of the most widely cited online treatments of the computational-graph view of backprop and the activation-function comparison.
• MIT 6.S191: Introduction to Deep Learning — Amini & Amini — free, updated annually; the opening lectures build the MLP and training loop from scratch with accompanying visual explanations.
• NYU Deep Learning — LeCun & Canziani — free notebooks and videos; gradient descent, backpropagation, and training taught by Yann LeCun, a founder of the field, and Alfredo Canziani.

Foundational papers

• Learning Representations by Back-propagating Errors — Rumelhart, Hinton & Williams (1986), Nature — the paper that brought backpropagation and learned hidden representations to a broad audience (paywalled, noted; widely reproduced online).
• Approximation by Superpositions of a Sigmoidal Function — Cybenko (1989) — the first proof of the universal approximation theorem for single-hidden-layer sigmoid networks (paywalled, noted).
• Approximation Capabilities of Multilayer Feedforward Networks — Hornik (1991) — extends universal approximation to arbitrary bounded, nonconstant activations (the non-polynomial characterization is Leshno et al. (1993); paywalled, noted).
• Understanding the Difficulty of Training Deep Feedforward Networks — Glorot & Bengio (2010) — free PDF; derives Xavier/Glorot initialization from variance preservation and explains sigmoid-saturation vanishing gradients, the backbone of this chapter’s initialization section.
• Delving Deep into Rectifiers — He, Zhang, Ren & Sun (2015) — free; derives He/Kaiming initialization, the current default for ReLU networks.
• Rectified Linear Units Improve Restricted Boltzmann Machines — Nair & Hinton (2010) — free; the systematic case for ReLU as a hidden-unit nonlinearity.
• Dropout: A Simple Way to Prevent Neural Networks from Overfitting — Srivastava et al. (2014), JMLR — free; the dropout paper, with the ensemble and co-adaptation interpretations this chapter uses.
• Reconciling Modern Machine-Learning Practice and the Bias–Variance Trade-off — Belkin, Hsu, Ma & Mandal (2019), PNAS — free; introduces the double-descent curve central to this chapter’s modern-generalization section.

Tutorials, notes, and interactive

• TensorFlow Playground — Smilkov & Carter — free, zero-install; tune depth, width, activation, and regularization and watch a decision boundary train in real time, a hands-on way to build hidden-layer intuition.
• Backpropagation, Intuitions — Stanford CS231n — free; the gate/computational-graph view with worked numerical examples, a concrete complement to this chapter’s backprop derivation.
• micrograd — Andrej Karpathy — free; a ~100-line scalar autograd engine plus a video walkthrough that builds it from scratch, a compact implementation related to the backpropagation section and its capstone exercise.
• Yes you should understand backprop — Andrej Karpathy — free; a short argument, for practical analysis of saturated sigmoids, inactive ReLUs, exploding gradients, and gradient clipping.
• Double Descent — MLU-Explain — free, interactive; animates the interpolation threshold and second descent, a clear visual companion to the modern-generalization section.
• KAN: Kolmogorov–Arnold Networks — Liu et al. (2024) — free; a recent alternative that places learnable activations on edges rather than fixed activations on nodes, useful context for where the MLP sits among modern parameterizations.

Leshno, Moshe, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. 1993. “Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any Function.” Neural Networks 6 (6): 861–67.