8  Sequence Models

A great deal of intelligent behavior looks like predicting what comes next. We finish each other’s sentences, anticipate the next note in a melody, and extrapolate a trend from the numbers seen so far. Each of these tasks takes a sequence of observations and asks for its continuation. The parts of this book that came before treated learning as a mapping from a fixed-length input to a single output, one example at a time. That framing served us well for tabular data and for images, but it quietly assumed away two features that define sequential data, and it is exactly those two features that this chapter is built to handle.

The first assumption was that examples are independent and identically distributed. When we fit linear and logistic regression in Chapter 2 and Chapter 3, or multilayer perceptrons in Chapter 4, we drew each example without regard to the others. In a sentence or a time series the opposite is true: every element depends on the ones before it, and that dependence is precisely the signal we want to model. The second assumption was that each input has a fixed shape. Even the images of Chapter 6 arrived as a fixed grid of pixels. A document, a recording, or a price history has no such fixed length, and two examples rarely share one.

Two ideas carry us through the whole chapter. The first is autoregressive factorization. Rather than model the probability of a whole sequence at once, we factor it into a product of one-step-ahead predictions: the probability of each element given the elements that precede it. This turns an unwieldy generative problem into an ordinary supervised one, in which the input is a prefix and the label is the next element. Every position in a sequence thus becomes another training signal, and generation becomes nothing more than repeating that prediction and feeding each output back in as the next input.

The second idea is the hidden state. A prefix grows without bound as a sequence unrolls, yet we cannot afford a memory that grows along with it. So we insist that the model carry a fixed-size summary of everything it has read so far, revising that summary as each new element arrives. This is what a recurrent neural network does, and the tension at its heart runs through the rest of the part: how much of an unbounded past can a bounded state honestly remember? Getting a useful answer to that question is what makes the difference between a model that forgets within a few steps and one that holds a thought.

Language modeling is the running application that ties these ideas together, and by the end we will have trained a small language model and sampled fresh text from it properly. The path there is a single build, one section handing off to the next: we tokenize a corpus, set a baseline to beat, learn a recurrence, face down the gradients that make it hard to train, and finally sample from it. We begin with sequences in the abstract in Section 8.1, then turn text into tokens a model can consume in Section 8.2. Section 8.3 frames the language-modeling task and fits a simple counting baseline to beat. Section 8.4 introduces the recurrent network itself, and Section 8.5 implements an RNN language model end to end. Section 8.6 confronts the gradient realities of training through time, and Section 8.7 closes the loop by turning a trained model’s predictions back into readable text.

A word on where this material stands. Recurrent networks powered the deep-learning breakthroughs of the 2010s in speech recognition and machine translation, and for a while they were the default model for anything sequential. Transformers later displaced them at scale, and much of today’s attention goes there. Yet recurrence has returned in modern guise, which we take up in Chapter 12, precisely because a bounded-memory state makes inference cheap when the alternative grows with the length of the sequence. Whichever architecture wins a given task, the concepts introduced here, namely autoregressive factorization, perplexity, backpropagation through time, and decoding, are exactly the ones on which the later material on large language models stands. They are worth learning once, and learning carefully.

Resources and Further Reading

The references below retrace this chapter’s arc — text into tokens, a counting baseline, the recurrence that replaces it, the gradients that make training hard, and the decoding that turns predictions back into text. Because tokenization, perplexity, and sampling are practiced today essentially as these sources describe, the list runs unusually far back and unusually far forward: Shannon’s 1951 guessing game and a 2025 sampling rule sit here comfortably side by side. All are freely accessible online except where noted.

Books

• Speech and Language Processing, 3rd ed. — Jurafsky & Martin — free draft (complete PDF, updated January 2026); Chapter 2 (words and tokens) and Chapter 3 (\(n\)-gram language models) are the standard treatment of Section 8.2 and Section 8.3, and develop the Kneser–Ney smoothing our sparsity discussion names but does not derive; Chapter 13 covers RNNs and LSTMs.
• Deep Learning — Goodfellow, Bengio & Courville — free HTML; Chapter 10 (Sequence Modeling) develops recurrent networks, teacher forcing, and backpropagation through time with fuller gradient derivations than Section 8.4 and Section 8.6 carry, and previews the gated architectures of the next chapter.
• Natural Language Processing — Jacob Eisenstein — free PDF of the notes that became Introduction to Natural Language Processing (MIT Press, 2019); Chapter 6 treats \(n\)-gram and RNN language models as one continuous story, exactly the handoff this chapter makes between Section 8.3 and Section 8.4.

Courses and video lectures

• Stanford CS224N: Natural Language Processing with Deep Learning — free slides and notes, with complete lecture videos on YouTube; its language-modeling and RNN lectures cover the span from Section 8.3 to Section 8.6, vanishing-gradient argument included, before heading toward transformers.
• Neural Networks: Zero to Hero — Andrej Karpathy — free video series; the makemore lectures build character-level language models from a counting bigram model up to neural networks — the arc from Section 8.3 to Section 8.5, compressed and typed live — and “Let’s build the GPT Tokenizer” does the same for the byte-level BPE of Section 8.2.
• MIT 6.S191: Introduction to Deep Learning — Amini & Amini — free, updated annually; the “Deep Sequence Modeling” lecture is a fast, visual pass over sequence tasks, RNNs, and their gradient pathologies (Section 8.4, Section 8.6).

Foundational papers

• Prediction and Entropy of Printed English — Claude E. Shannon (1951), Bell System Technical Journal — free scan; frames language as a next-symbol guessing game and measures the entropy of English with human predictors — the direct ancestor of the perplexity and bits-per-byte of Section 8.3, and still a delight to read.
• Finding Structure in Time — Jeffrey Elman (1990), Cognitive Science — the network of Section 8.4 in its original habitat: a hidden state fed back on itself discovers word boundaries and grammatical structure from raw sequences (paywalled, noted; widely reproduced online).
• A Neural Probabilistic Language Model — Bengio, Ducharme, Vincent & Jauvin (2003), JMLR — free; the answer to the sparsity wall of Section 8.3: replace counting with learned embeddings and a neural network, the embedding-plus-softmax design that Section 8.5 inherits.
• Recurrent Neural Network Based Language Model — Mikolov et al. (2010), Interspeech — free; the demonstration that RNN language models beat smoothed \(n\)-grams in practice — the model of Section 8.5, evaluated the way Section 8.3 teaches.
• On the Difficulty of Training Recurrent Neural Networks — Pascanu, Mikolov & Bengio (2013), ICML — free; the modern account of vanishing and exploding gradients and the source of the gradient-norm clipping used in Section 8.5; the difficulty itself was first formalized by Bengio, Simard & Frasconi (1994) (paywalled, noted).
• Neural Machine Translation of Rare Words with Subword Units — Sennrich, Haddow & Birch (2016), ACL — free; the paper that brought byte pair encoding into NLP — the merge algorithm implemented from scratch in Section 8.2.
• Language Models are Unsupervised Multitask Learners — Radford et al. (2019) — free; the GPT-2 report, source of both the byte-level BPE with regex pre-tokenization that Section 8.2 rebuilds and checks token-for-token against tiktoken, and the top-\(k\) truncation of Section 8.7.
• The Curious Case of Neural Text Degeneration — Holtzman et al. (2020), ICLR — free; diagnoses why maximization loops and why the unreliable tail spoils pure sampling, then proposes nucleus (top-\(p\)) sampling — the argument on which Section 8.7 is built.
• Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs — Nguyen et al. (2025), ICLR — free; the confidence-scaled truncation rule in the unified sampler of Section 8.7, with evidence that it degrades most gracefully at the high temperatures where creative sampling wants to run.

Tutorials, notes, and interactive

• The Unreasonable Effectiveness of Recurrent Neural Networks — Andrej Karpathy — free; the 2015 essay that showed character-level RNN language models writing Shakespeare, Wikipedia markup, LaTeX, and C code, and visualized single hidden units tracking quotes and line lengths — the best advertisement for what Section 8.5 trains, sampled the way Section 8.7 explains.
• minbpe — Andrej Karpathy — free; a minimal, thoroughly commented byte-level BPE codebase (the companion to the tokenizer video above) that reaches parity with GPT-4’s tiktoken tokenizer — the natural second implementation to compare against the one built in Section 8.2.
• Tiktokenizer — free, zero-install; type text and watch production tokenizers segment it live — an instant way to rerun the fertility, digit-splitting, and whitespace experiments of Section 8.2 on GPT-2 versus current vocabularies.
• How to Generate Text — Patrick von Platen (Hugging Face) — free; greedy search, beam search, top-\(k\), and top-\(p\) with runnable transformers code — the experiments of Section 8.7 repeated on models large enough for each failure mode to be vivid.
• Hugging Face LLM Course, Chapter 6: Tokenizers — free; train and dissect production BPE, WordPiece, and Unigram tokenizers with the tokenizers library — the industrial-strength counterpart to the tokenizers-in-the-wild tour closing Section 8.2.