Dive into Deep Learning
Slides
Courses
GitHub
Discuss
PDF
PyTorch
JAX
TensorFlow
MXNet
Notebooks
PyTorch
JAX
TensorFlow
MXNet
Language Models
25
Preference Optimization and Reinforcement Learning
index.html
Preface
Installation
Notation
Introduction
Basics
1
Preliminaries
1.1
Data Manipulation
1.2
Data Preprocessing
1.3
Linear Algebra
1.4
Calculus
1.5
Automatic Differentiation
1.6
Probability and Statistics
1.7
Documentation
2
Linear Regression
2.1
Linear Regression
2.2
Object-Oriented Design for Implementation
2.3
Synthetic Data
2.4
Implementation from Scratch
2.5
Concise Implementation
2.6
Generalization
2.7
Weight Decay
3
Linear Classification
3.1
Softmax Regression
3.2
Image Classification Data
3.3
Measuring Classifiers
3.4
Softmax Regression from Scratch
3.5
Concise Implementation
3.6
Generalization in Classification
3.7
Environment and Distribution Shift
4
Multilayer Perceptron
4.1
Multilayer Perceptrons
4.2
Implementation
4.3
Backpropagation
4.4
Numerical Stability and Initialization
4.5
Generalization in Deep Learning
4.6
Dropout
4.7
Predicting House Prices on Kaggle
5
Computation
5.1
Modules and Model Construction
5.2
Parameters, State, and Memory
5.3
Initialization
5.4
Custom Layers and Functions
5.5
Numerical Formats
5.6
Saving, Loading, and Pretrained Weights
5.7
GPUs, Devices, and Memory
5.8
Reproducibility and Inspection
6
Convolutional Neural Networks
6.1
From Fully Connected Layers to Convolutions
6.2
Convolutions for Images
6.3
Padding, Stride, and Dilation
6.4
Multiple Input and Multiple Output Channels
6.5
Pooling
6.6
Convolutional Neural Networks (LeNet)
7
Modern Convnets
7.1
AlexNet and Learned Image Representations
7.2
Blocks, Bottlenecks, and Branches: VGG, NiN, GoogLeNet
7.3
Normalization Layers
7.4
Residual Networks: ResNet, ResNeXt, and DenseNet
7.5
Efficient Convnets
7.6
Modern Training Recipes for Convnets
7.7
ConvNeXt: A ConvNet for the 2020s
7.8
Convolutional Network Design Spaces
8
Sequence Models
8.1
Working with Sequences
8.2
From Text to Tokens
8.3
Language Models
8.4
Recurrent Neural Networks
8.5
Implementing RNNs
8.6
Backpropagation Through Time
8.7
Decoding and Generation
Advanced
9
Optimization Algorithms
9.1
Landscapes
9.2
Gradient Descent
9.3
Stochastic and Minibatch Gradient Descent
9.4
Momentum
9.5
Adam and AdamW
9.6
Schedules
9.7
Muon
9.8
Batch Size
9.9
Scaling Up
9.10
Practice
9.11
Further Reading
10
Attention
10.1
Queries, Keys, and Values
10.2
Attention Scoring and Masking
10.3
Multi-Head and Cross-Attention
10.4
Positional Information
10.5
The Cost of Attention
10.6
Attention Circuits
10.7
Further Reading
11
Transformers
11.1
The Transformer Block
11.2
A GPT from Scratch
11.3
Generation and the KV Cache
11.4
Encoders, Decoders, and Cross-Attention
11.5
Vision Transformer
11.6
Mixture of Experts
11.7
Scaling Laws
11.8
Further Reading
12
State Space Models
12.1
Gated Recurrent Networks
12.2
Linear Recurrences and State Space Models
12.3
Selective State Space Models
12.4
Matrix-State Recurrences
12.5
The Delta Rule
12.6
Hybrid Architectures
12.7
Further Reading
13
Computational Performance
13.1
The Performance Model
13.2
Hardware
13.3
Compute Graphs and Compilation
13.4
Memory and Precision
13.5
Multi-GPU from First Principles
13.6
Multi-GPU in Practice
13.7
Case Study: Optimizing a Transformer
13.8
Further Reading
14
Reinforcement Learning
14.1
Markov Decision Processes
14.2
Dynamic Programming
14.3
Temporal-Difference Learning
14.4
Exploration
14.5
Learning from Demonstrations
14.6
Policy Gradient
14.7
Baselines and Variance Reduction
14.8
Function Approximation in Reinforcement Learning
15
Deep Reinforcement Learning
15.1
Actor-Critic Methods and Multi-Step Returns
15.2
Trust Regions and Proximal Policy Optimization
15.3
KL-Regularized Policy Optimization
15.4
Deep Q-Networks
15.5
Soft Actor-Critic
15.6
Learned Models and Planning
15.7
Offline Reinforcement Learning
15.8
Reinforcement Learning for Sequence Generation
15.9
Further Reading
16
Generative Adversarial Networks
16.1
Generative Adversarial Networks
16.2
Adversarial Objectives and Divergences
16.3
Relativistic Objectives
16.4
Gradient Penalties and Convergence
16.5
Adversarial Image Generation
16.6
Conditional Generation
16.7
Adversarial Losses Beyond GANs
16.8
Further Reading
17
Diffusion Models
17.1
Denoising and the Score
17.2
Gaussian Paths: Diffusion and Flow Matching
17.3
Diffusion for Images
17.4
Samplers, Inversion and Editing
17.5
Likelihood, Weighting and the Design Space
17.6
Latent Diffusion and Transformers
17.7
Few-Step Generation
17.8
Discrete Diffusion
17.9
Further Reading
Language Models
18
Working with Pretrained Language Models
18.1
Pretrained Models with Hugging Face
18.2
A Released Checkpoint in the Book’s Own Code
18.3
Prompting and In-Context Learning
18.4
Scoring Language Models
18.5
Evaluation Suites, Judges and Error Bars
18.6
Interpretability
18.7
Models behind an API
19
Encoders, Embeddings and Retrieval
19.1
Word Embeddings
19.2
Pretrained Encoders
19.3
Text Embeddings from Contrastive Training
19.4
Sparse, Dense and Approximate Retrieval
19.5
Retrieval-Augmented Generation
19.6
Further Reading
20
Data for Language Models
20.1
Document Extraction from Web Crawls
20.2
Quality Filtering
20.3
Deduplication and Decontamination
20.4
Data Mixtures
20.5
Synthetic and Rephrased Data
20.6
Tokenizers at Scale
21
Pretraining
21.1
The Pretraining Data Path
21.2
The Reference Pretraining Run
21.3
Low-Precision Training
21.4
Continued Pretraining and Mid-Training
21.5
Long-Context Training
21.6
Training Budgets beyond the Compute Optimum
22
Hardware for Training and Inference
22.1
Accelerator Design
22.2
Kernels, Tiles and Fusion
22.3
Training, Prefill and Decode Workloads
22.4
Memory and Storage Tiers
22.5
Data Paths inside a Server
22.6
Scale-Up Domains and Scale-Out Networks
22.7
Collective Communication at Scale
22.8
Energy, Power and Cost
23
Parallelism Beyond One Node
23.1
Multi-Node Jobs and Hang Diagnosis
23.2
Sharded Data Parallelism at Scale
23.3
Tensor and Sequence Parallelism
23.4
Pipeline Parallelism
23.5
Context Parallelism
23.6
Expert Parallelism
23.7
Composed Parallelism and Topology
23.8
Profiling and Performance Debugging of Distributed Runs
23.9
Checkpoints at Scale
23.10
Failures, Elasticity and Goodput
24
Fine-Tuning
24.1
Full Fine-Tuning
24.2
Supervised Instruction Tuning
24.3
Instruction Data
24.4
Low-Rank Adaptation
24.5
Quantized Fine-Tuning
24.6
Fine-Tuning at Scale and Toolchains
24.7
Merging and Retaining Capabilities
25
Preference Optimization and Reinforcement Learning
25.1
Reward Models
25.2
RLHF with PPO at Language-Model Scale
25.3
Direct Preference Optimization and Its Variants
25.4
RL with Verifiable Rewards
25.5
RL Training Systems
25.6
Reward Overoptimization
25.7
Safety Evaluation and Red-Teaming
25.8
Further Reading
26
Distillation and Reasoning
26.1
Knowledge Distillation
26.2
On-Policy Distillation
26.3
Training Reasoning Models
26.4
Verifiers and Process Supervision
26.5
Parallel Test-Time Compute
26.6
Sequential and Adaptive Test-Time Compute
27
Agents and Tool Use
27.1
Tool Calling
27.2
Structured Outputs and Constrained Decoding
27.3
Agent Loops and Harnesses
27.4
Training Agents with Reinforcement Learning
27.5
Evaluating and Securing Agents
28
Model Compression
28.1
Quantization for Inference
28.2
Activation Outliers and Equivalent Rescaling
28.3
Reconstruction-Based Quantization
28.4
Rotations and Incoherence
28.5
Quantization-Aware Training and Native Low-Bit Formats
28.6
Pruning and Sparsity
28.7
Further Reading
29
Efficient Inference
29.1
A Fast Decoder on One GPU
29.2
Decode Attention Kernels
29.3
Deterministic and Batch-Invariant Inference
29.4
Speculative Decoding
29.5
Drafters and Multi-Token Prediction
29.6
KV-Cache Quantization
29.7
KV-Cache Eviction and Selective Reading
29.8
Inference on CPUs and Devices
30
Serving Systems
30.1
Workloads and Service Objectives
30.2
Continuous Batching and Chunked Prefill
30.3
Paged KV Memory
30.4
Prefix Caching and Shared-Prefix Workloads
30.5
Scheduling under Overload
30.6
KV-Cache Offload, Stores and Prefix-Aware Routing
30.7
Tensor and Expert Parallelism for Serving
30.8
Prefill and Decode Disaggregation
30.9
Serving Many Adapters
31
Advanced Architectures
31.1
Architecture Configurations
31.2
Mixture-of-Experts Design at Scale
31.3
Latent Attention and Cross-Layer KV Sharing
31.4
Softmax Attention Refinements
31.5
Learned Sparse Attention
31.6
Hybrid Models at Scale
31.7
The Residual Stream across Depth
31.8
Further Reading
Image Models
32
Computer Vision
32.1
Image Augmentation
32.2
Fine-Tuning
32.3
Object Detection and Bounding Boxes
32.4
Anchor Boxes
32.5
Multiscale Object Detection
32.6
The Object Detection Dataset
32.7
Single Shot Multibox Detection
32.8
Region-based CNNs (R-CNNs)
32.9
Semantic Segmentation and the Dataset
32.10
Transposed Convolution
32.11
Fully Convolutional Networks
32.12
Neural Style Transfer
32.13
Image Classification (CIFAR-10) on Kaggle
32.14
Dog Breed Identification (ImageNet Dogs) on Kaggle
Attic
33
Gaussian Processes
33.1
Introduction to Gaussian Processes
33.2
Gaussian Process Priors
33.3
Gaussian Process Inference
34
Hyperparameter Optimization
34.1
What Is Hyperparameter Optimization?
34.2
Hyperparameter Optimization API
34.3
Asynchronous Random Search
34.4
Multi-Fidelity Hyperparameter Optimization
34.5
Asynchronous Successive Halving
35
Recommender Systems
35.1
Overview of Recommender Systems
35.2
The MovieLens Dataset
35.3
Matrix Factorization
35.4
AutoRec: Rating Prediction with Autoencoders
35.5
Personalized Ranking for Recommender Systems
35.6
Neural Collaborative Filtering for Personalized Ranking
35.7
Sequence-Aware Recommender Systems
35.8
Feature-Rich Recommender Systems
35.9
Factorization Machines
35.10
Deep Factorization Machines
36
Natural Language Processing: Pretraining
36.1
Encoder-Decoder Models for Sequence Transduction
36.2
Word Embedding (word2vec)
36.3
Approximate Training
36.4
The Dataset for Pretraining Word Embeddings
36.5
Pretraining word2vec
36.6
Word Embedding with Global Vectors (GloVe)
36.7
Subword Embedding
36.8
Word Similarity and Analogy
36.9
Bidirectional Encoder Representations from Transformers (BERT)
36.10
The Dataset for Pretraining BERT
36.11
Pretraining BERT
37
Natural Language Processing: Applications
37.1
Sentiment Analysis and the Dataset
37.2
Sentiment Analysis: Using Recurrent Neural Networks
37.3
Sentiment Analysis: Using Convolutional Neural Networks
37.4
Natural Language Inference and the Dataset
37.5
Natural Language Inference: Using Attention
37.6
Fine-Tuning BERT for Sequence-Level and Token-Level Applications
37.7
Natural Language Inference: Fine-Tuning BERT
Mathematics for Deep Learning
38
Linear Algebra
38.1
Vectors, Angles, and Projections
38.2
Matrices as Linear Maps
38.3
Eigendecompositions
38.4
Singular Value Decomposition and Low-Rank Approximation
38.5
Further Reading
39
Calculus and Automatic Differentiation
39.1
Single Variable Calculus
39.2
Multivariable Calculus
39.3
Matrix Calculus
39.4
Automatic Differentiation
39.5
Integral Calculus
39.6
Further Reading
40
Optimization
40.1
Gradient Descent and Curvature
40.2
Convexity
40.3
Stochastic Gradient Methods
40.4
Adaptive Methods
40.5
Constrained Optimization
40.6
Duality
40.7
Numerical Stability and Conditioning
40.8
Further Reading
41
Probability and Statistical Learning
41.1
Random Variables
41.2
Distributions
41.3
The Exponential Family
41.4
Statistics
41.5
Maximum Likelihood
41.6
Bayesian Computation
41.7
Concentration Inequalities
41.8
Uniform Convergence and Generalization
41.9
Naive Bayes
41.10
Further Reading
42
Information Theory and Divergences
42.1
Entropy, Cross-Entropy, and KL Divergence
42.2
Codes, Compression, and Perplexity
42.3
Divergences and Distances Between Distributions
42.4
Mutual Information
42.5
Estimating Mutual Information
42.6
Further Reading
43
Dynamics: Differential Equations and Generative Flows
43.1
Ordinary Differential Equations and Numerical Solvers
43.2
Neural ODEs, Adjoints, and Continuous Flows
43.3
Stochastic Differential Equations
43.4
The Fokker–Planck Equation and Probability Flow
43.5
Score Matching and Diffusion Models
43.6
Flow Matching and Optimal Transport
43.7
Further Reading
Tools for Deep Learning
44
Tools for Deep Learning
44.1
Notebooks
44.2
Colab and Kaggle
44.3
Cloud Computing
44.4
Hardware
44.5
Ecosystem
44.6
Distributed Model Training
44.7
Model Serving
44.8
Contributor Guide
44.9
Utility Functions and Classes
44.10
The
d2l
API Reference
References
Language Models
25
Preference Optimization and Reinforcement Learning
25.1
Reward Models
25
Preference Optimization and Reinforcement Learning
25.2
RLHF with PPO at Language-Model Scale