28  Model Compression

In Section 24.5 the reader stored a frozen model in four bits so that a fine-tuning run would fit, and the model ran no faster. A served model has to be fast, its activations enter the arithmetic, and it is judged on held-out quality at a stated speed. This chapter builds, one mechanism at a time, the methods that compress a trained model for that purpose, and checks each against brute force, an exact identity, or a paired interval.