20  Data for Language Models

The reader has trained on a text file someone else prepared. Released models start from billions of pages, and every choice made before the first batch (what to extract, what to drop, what to repeat, how to cut it into tokens) changes what the model can ever learn.