8 Sequence Models
After two chapters on images, the last chapter of this part turns to sequences. Every model so far reads an input of fixed shape, such as a \(28 \times 28\) image, and treats each example on its own. A sentence, a recording, or a series of measurements is different: it varies in length, and its entries depend on each other and on their order. Take the opening sentence of H. G. Wells’ The Time Machine, which the models of this chapter read: “The Time Traveller (for so it will be convenient to speak of him) was expounding a recondite matter to us.” To guess its last word, a reader uses the nineteen words before it, in order. How can a model do the same when the text before it can have any length?
Claude Shannon made a game of this. In 1951 he measured how predictable English is by asking people to guess a text one letter at a time. A language model plays the same game: it predicts each token, which may be a character or a word, from the tokens before it. A model that can predict can also write, by feeding each token it chooses back in as its next input.
The difficulty is the text already read, which grows with every token. A window of the last few tokens forgets everything older. A recurrent neural network instead summarizes all of it in a hidden state of fixed size, which the same layer updates after every token. Today’s large language models also predict one token at a time, so what this chapter teaches about cutting text into tokens, scoring a model, and turning its predictions into text carries over to them.
The chapter sets up the problem before any recurrent network appears. We predict a sequence from its own past and see why forecasting many steps ahead is harder than forecasting one (Section 8.1), cut text into tokens (Section 8.2), and build language models that only count, which also gives us the measure by which language models are compared (Section 8.3). The counts run out, and the recurrent network is our answer. Section 8.4 replaces the window by a hidden state, and Section 8.5 implements and trains the model. We then put it to the test: against the counts, without the safeguard that its training turns out to need, and position by position, to see how much of what it has read the model uses. Section 8.6 explains what we find from the gradients of that same model. The last section, Section 8.7, turns next-token distributions into text.
Gated cells and state space models, which let a recurrent network remember further back, follow in Chapter 12, and attention, which keeps every past token, in Chapter 10.
Resources and Further Reading
The resources below accompany the sections of this chapter; each names the section it goes with. All are freely accessible online except where noted.
Books
- Speech and Language Processing, 3rd ed. — Jurafsky & Martin — free draft (complete PDF, updated January 2026); Chapter 2 (words and tokens) and Chapter 3 (\(n\)-gram language models) are the standard treatment of Section 8.2 and Section 8.3, and treat at length the interpolated and discounted smoothing schemes, of which Section 8.3 derives Kneser–Ney; Chapter 13 covers RNNs and LSTMs.
- Deep Learning — Goodfellow, Bengio & Courville — free HTML; Chapter 10 (Sequence Modeling) develops recurrent networks, teacher forcing, and backpropagation through time with fuller gradient derivations than Section 8.4 and Section 8.6, and introduces the gated architectures of Chapter 12.
- Natural Language Processing — Jacob Eisenstein — free PDF of the notes that became Introduction to Natural Language Processing (MIT Press, 2019); Chapter 6 treats \(n\)-gram and RNN language models as one story, the step this chapter takes from Section 8.3 to Section 8.4.
Courses and video lectures
- Stanford CS224N: Natural Language Processing with Deep Learning — free slides and notes, with complete lecture videos on YouTube; its language-modeling and RNN lectures cover the material from Section 8.3 to Section 8.6, the vanishing-gradient argument included, before turning to transformers.
- Neural Networks: Zero to Hero — Andrej Karpathy — free video series; the makemore lectures build character-level language models from a counting bigram model up to neural networks, the path from Section 8.3 to Section 8.5 typed live, and “Let’s build the GPT Tokenizer” builds the byte-level BPE of Section 8.2.
- MIT 6.S191: Introduction to Deep Learning — Amini & Amini — free, updated annually; the “Deep Sequence Modeling” lecture is a fast, visual pass over sequence tasks, RNNs, and their gradient pathologies (Section 8.4, Section 8.6).
Foundational papers
- Prediction and Entropy of Printed English — Claude E. Shannon (1951), Bell System Technical Journal — free scan; frames language as a game of guessing the next symbol and measures the entropy of English with human predictors, the origin of the perplexity and bits per byte of Section 8.3.
- Finding Structure in Time — Jeffrey Elman (1990), Cognitive Science — paywalled, widely reproduced online; introduces the network of Section 8.4, whose hidden state, fed back on itself, discovers word boundaries and grammatical categories from raw sequences.
- A Neural Probabilistic Language Model — Bengio, Ducharme, Vincent & Jauvin (2003), JMLR — free; replaces the sparse count tables of Section 8.3 by learned embeddings and a neural network, the neural \(n\)-gram of Section 8.4 and the embedding and softmax layers of Section 8.5.
- A Hierarchical Bayesian Language Model Based on Pitman–Yor Processes — Yee Whye Teh (2006), ACL — free; gives the Kneser–Ney smoothing of Section 8.3 a Bayesian reading, as approximate inference in a hierarchical Pitman–Yor process, a model that the sequence memoizer of Wood et al. (2011) extends to contexts of unbounded length.
- Recurrent Neural Network Based Language Model — Mikolov et al. (2010), Interspeech — free; shows that RNN language models beat smoothed \(n\)-grams in practice, with the model of Section 8.5 evaluated by perplexity as in Section 8.3.
- On the Difficulty of Training Recurrent Neural Networks — Pascanu, Mikolov & Bengio (2013), ICML — free; the analysis of vanishing and exploding gradients in Section 8.6 and the source of the gradient-norm clipping of Section 8.5. The difficulty itself was first formalized by Bengio, Simard & Frasconi (1994) (paywalled).
- Neural Machine Translation of Rare Words with Subword Units — Sennrich, Haddow & Birch (2016), ACL — free; brought byte pair encoding into NLP, the merge algorithm implemented from scratch in Section 8.2.
- Language Models are Unsupervised Multitask Learners — Radford et al. (2019) — free; the GPT-2 report, source of the byte-level BPE with regex pre-tokenization that Section 8.2 rebuilds and checks token for token against
tiktoken, and of the top-\(k\) sampling of Section 8.7. - The Curious Case of Neural Text Degeneration — Holtzman et al. (2020), ICLR — free; explains why maximization loops and why the unreliable tail spoils pure sampling, and proposes nucleus (top-\(p\)) sampling, the argument Section 8.7 follows.
- Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs — Nguyen et al. (2025), ICLR — free; the min-\(p\) rule in the sampler of Section 8.7, which keeps tokens in proportion to the model’s confidence, with evidence that it degrades less than top-\(p\) at high temperatures.
Tutorials, notes, and interactive
- The Unreasonable Effectiveness of Recurrent Neural Networks — Andrej Karpathy — free; the 2015 essay in which character-level RNN language models write Shakespeare, Wikipedia markup, LaTeX, and C code, and single hidden units track quotes and line lengths; a larger version of what Section 8.5 trains, sampled as in Section 8.7.
- minbpe — Andrej Karpathy — free; a minimal, commented byte-level BPE codebase (the companion to the tokenizer video above) that reproduces GPT-4’s
tiktokentokenizer, a second implementation to compare with the one of Section 8.2. - Tiktokenizer — free, nothing to install; shows how production tokenizers segment text as it is typed, a quick way to repeat the fertility and digit experiments of Section 8.2.5.3 on GPT-2 and current vocabularies.
- How to Generate Text — Patrick von Platen (Hugging Face) — free; greedy search, beam search, top-\(k\), and top-\(p\) with runnable
transformerscode, the experiments of Section 8.7 repeated on models large enough to show each failure clearly. - Hugging Face LLM Course, Chapter 6: Tokenizers — free; trains and dissects production BPE, WordPiece, and Unigram tokenizers with the
tokenizerslibrary, a companion to the production tokenizers examined at the end of Section 8.2.