30  Serving Systems

The reader has Section 29.1’s decoder, which answers one request near the bandwidth bound. A service never sees one request: it sees an arrival process, a mix of prompt and output lengths, a latency contract, and a hundred users who are each waiting. This chapter builds the mechanisms inside a serving engine, each with an invariant a test can check and a measurement against a reference server, from the contract a service makes to the layouts that spread one model over a node.