8  Sequence Models

After two chapters on images, the last chapter of this part turns to sequences. Every model so far reads an input of fixed shape, such as a \(28 \times 28\) image, and treats each example on its own. A sentence, a recording, or a series of measurements is different: it varies in length, and its entries depend on each other and on their order. Take the opening sentence of H. G. Wells’ The Time Machine, which the models of this chapter read: “The Time Traveller (for so it will be convenient to speak of him) was expounding a recondite matter to us.” To guess its last word, a reader uses the nineteen words before it, in order. How can a model do the same when the text before it can have any length?

Claude Shannon made a game of this. In 1951 he measured how predictable English is by asking people to guess a text one letter at a time. A language model plays the same game: it predicts each token, which may be a character or a word, from the tokens before it. A model that can predict can also write, by feeding each token it chooses back in as its next input.

The difficulty is the text already read, which grows with every token. A window of the last few tokens forgets everything older. A recurrent neural network instead summarizes all of it in a hidden state of fixed size, which the same layer updates after every token. Today’s large language models also predict one token at a time, so what this chapter teaches about cutting text into tokens, scoring a model, and turning its predictions into text carries over to them.

The chapter sets up the problem before any recurrent network appears. We predict a sequence from its own past and see why forecasting many steps ahead is harder than forecasting one (Section 8.1), cut text into tokens (Section 8.2), and build language models that only count, which also gives us the measure by which language models are compared (Section 8.3). The counts run out, and the recurrent network is our answer. Section 8.4 replaces the window by a hidden state, and Section 8.5 implements and trains the model. We then put it to the test: against the counts, without the safeguard that its training turns out to need, and position by position, to see how much of what it has read the model uses. Section 8.6 explains what we find from the gradients of that same model. The last section, Section 8.7, turns next-token distributions into text.

Gated cells and state space models, which let a recurrent network remember further back, follow in Chapter 12, and attention, which keeps every past token, in Chapter 10.

Resources and Further Reading

The resources below accompany the sections of this chapter; each names the section it goes with. All are freely accessible online except where noted.

Books

Courses and video lectures

Foundational papers

Tutorials, notes, and interactive