31  Advanced Architectures

The reader has the transformer of Advanced (RoPE, GQA, RMSNorm, SwiGLU, a mixture of experts, sliding windows and sinks, DeltaNet and hybrid stacks) and a configuration file from Section 18.1 with fields that match none of it: a latent rank, an indexer that keeps 512 of a million keys, a layer type called linear_attention, an hc_mult of 4, an embedding table with 384 million rows. This chapter teaches those fields, one design axis per section, and for each change asks what it saves, what it costs, and which training and serving setup makes the trade worthwhile.