8  Sequence Models

Many learning problems involve ordered observations: words in a sentence, notes in a melody, or measurements in a time series. Earlier chapters mapped a fixed-length input to an output and usually treated examples as independent. Sequence models must instead represent dependence across positions and inputs whose lengths may vary.

Earlier objectives usually treated examples as exchangeable, fixed-shape records. Sequence tasks retain the order within each example: conditioning on earlier words, notes, or measurements can change the distribution of later ones. Sequence lengths may also vary, even though implementations often pad or batch them into equal-sized arrays.

The chapter develops two main ideas. The first is autoregressive factorization. Rather than model the probability of a whole sequence at once, we factor it into a product of one-step-ahead predictions: the probability of each element given the elements that precede it. This turns an unwieldy generative problem into an ordinary supervised one, in which the input is a prefix and the label is the next element. Every position in a sequence thus becomes another training signal. Generation repeats this prediction while feeding each generated output back as the next input.

The second idea is the hidden state. A prefix grows with time, but a recurrent model carries a fixed-size state \(h_t=f(x_t,h_{t-1})\). The state is a function of the prefix, and training determines which predictive information it retains. Its fixed size makes incremental inference inexpensive but may discard information from the distant past.

Language modeling provides the running application. We turn text into tokens, factor sequence probabilities into next-token conditionals, and replace a fixed context window with a recurrent state. The implementation then exposes two remaining problems: gradients must propagate through repeated state updates, and a probability distribution still needs a decoding rule to produce text.

Recurrent networks powered the deep-learning breakthroughs of the 2010s in speech recognition and machine translation, and for a while they were the default model for anything sequential. Transformers later displaced them at scale, and much of today’s attention goes there. Yet recurrence has returned in modern guise, which we take up in Chapter 12, precisely because a bounded-memory state makes inference cheap when the alternative grows with the length of the sequence. Whichever architecture wins a given task, the later material on large language models uses the same concepts introduced here: autoregressive factorization, perplexity, backpropagation through time, and decoding. They are worth learning once, and learning carefully.

Resources and Further Reading

The references below retrace this chapter’s arc — text into tokens, a counting baseline, the recurrence that replaces it, the gradients that make training hard, and the decoding that turns predictions back into text. Because tokenization, perplexity, and sampling are practiced today essentially as these sources describe, the list runs unusually far back and unusually far forward: Shannon’s 1951 guessing game and a 2025 sampling rule sit here comfortably side by side. All are freely accessible online except where noted.

Books

• Speech and Language Processing, 3rd ed. — Jurafsky & Martin — free draft (complete PDF, updated January 2026); Chapter 2 (words and tokens) and Chapter 3 (\(n\)-gram language models) are the standard treatment of Section 8.2 and Section 8.3, and develop the Kneser–Ney smoothing our sparsity discussion names but does not derive; Chapter 13 covers RNNs and LSTMs.
• Deep Learning — Goodfellow, Bengio & Courville — free HTML; Chapter 10 (Sequence Modeling) develops recurrent networks, teacher forcing, and backpropagation through time with fuller gradient derivations than Section 8.4 and Section 8.6 carry, and previews the gated architectures of the next chapter.
• Natural Language Processing — Jacob Eisenstein — free PDF of the notes that became Introduction to Natural Language Processing (MIT Press, 2019); Chapter 6 treats \(n\)-gram and RNN language models as one continuous story, exactly the handoff this chapter makes between Section 8.3 and Section 8.4.

Courses and video lectures

• Stanford CS224N: Natural Language Processing with Deep Learning — free slides and notes, with complete lecture videos on YouTube; its language-modeling and RNN lectures cover the span from Section 8.3 to Section 8.6, vanishing-gradient argument included, before heading toward transformers.
• Neural Networks: Zero to Hero — Andrej Karpathy — free video series; the makemore lectures build character-level language models from a counting bigram model up to neural networks — the arc from Section 8.3 to Section 8.5, compressed and typed live — and “Let’s build the GPT Tokenizer” does the same for the byte-level BPE of Section 8.2.
• MIT 6.S191: Introduction to Deep Learning — Amini & Amini — free, updated annually; the “Deep Sequence Modeling” lecture is a fast, visual pass over sequence tasks, RNNs, and their gradient pathologies (Section 8.4, Section 8.6).

Foundational papers

• Prediction and Entropy of Printed English — Claude E. Shannon (1951), Bell System Technical Journal — free scan; frames language as a next-symbol guessing game and measures the entropy of English with human predictors — the direct ancestor of the perplexity and bits-per-byte of Section 8.3, and still a delight to read.
• Finding Structure in Time — Jeffrey Elman (1990), Cognitive Science — the network of Section 8.4 in its original habitat: a hidden state fed back on itself discovers word boundaries and grammatical structure from raw sequences (paywalled, noted; widely reproduced online).
• A Neural Probabilistic Language Model — Bengio, Ducharme, Vincent & Jauvin (2003), JMLR — free; replaces the sparse count tables of Section 8.3 with learned embeddings and a neural network, the embedding-plus-softmax design that Section 8.5 inherits.
• Recurrent Neural Network Based Language Model — Mikolov et al. (2010), Interspeech — free; the demonstration that RNN language models beat smoothed \(n\)-grams in practice — the model of Section 8.5, evaluated the way Section 8.3 teaches.
• On the Difficulty of Training Recurrent Neural Networks — Pascanu, Mikolov & Bengio (2013), ICML — free; the modern account of vanishing and exploding gradients and the source of the gradient-norm clipping used in Section 8.5; the difficulty itself was first formalized by Bengio, Simard & Frasconi (1994) (paywalled, noted).
• Neural Machine Translation of Rare Words with Subword Units — Sennrich, Haddow & Birch (2016), ACL — free; the paper that brought byte pair encoding into NLP — the merge algorithm implemented from scratch in Section 8.2.
• Language Models are Unsupervised Multitask Learners — Radford et al. (2019) — free; the GPT-2 report, source of both the byte-level BPE with regex pre-tokenization that Section 8.2 rebuilds and checks token-for-token against tiktoken, and the top-\(k\) truncation of Section 8.7.
• The Curious Case of Neural Text Degeneration — Holtzman et al. (2020), ICLR — free; diagnoses why maximization loops and why the unreliable tail spoils pure sampling, then proposes nucleus (top-\(p\)) sampling — the argument on which Section 8.7 is built.
• Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs — Nguyen et al. (2025), ICLR — free; the confidence-scaled truncation rule in the unified sampler of Section 8.7, with evidence that it degrades most gracefully at the high temperatures where creative sampling wants to run.

Tutorials, notes, and interactive

• The Unreasonable Effectiveness of Recurrent Neural Networks — Andrej Karpathy — free; the 2015 essay that showed character-level RNN language models writing Shakespeare, Wikipedia markup, LaTeX, and C code, and visualized single hidden units tracking quotes and line lengths — the best advertisement for what Section 8.5 trains, sampled the way Section 8.7 explains.
• minbpe — Andrej Karpathy — free; a minimal, thoroughly commented byte-level BPE codebase (the companion to the tokenizer video above) that reaches parity with GPT-4’s tiktoken tokenizer — the natural second implementation to compare against the one built in Section 8.2.
• Tiktokenizer — free, zero-install; type text and watch production tokenizers segment it live — an instant way to rerun the fertility, digit-splitting, and whitespace experiments of Section 8.2 on GPT-2 versus current vocabularies.
• How to Generate Text — Patrick von Platen (Hugging Face) — free; greedy search, beam search, top-\(k\), and top-\(p\) with runnable transformers code — the experiments of Section 8.7 repeated on models large enough for each failure mode to be vivid.
• Hugging Face LLM Course, Chapter 6: Tokenizers — free; train and dissect production BPE, WordPiece, and Unigram tokenizers with the tokenizers library — the industrial-strength counterpart to the tokenizers-in-the-wild tour closing Section 8.2.