What a model trains on: a corpus split into training and validation text, cut into windows, batched, and (for most models) turned into tokens by byte-pair encoding. Then what changes at scale: packing, deduplication, filtering, mixtures and tokens per parameter.
This deck follows chapter 2, data and tokenisation, where BPE learns its first merges on the whole book in your browser. Measured numbers: results.md §1; the BPE demo below runs the site’s algorithm on the book’s first paragraph.
The Training site trains on Lewis Carroll’s Alice’s Adventures in Wonderland (1865, Project Gutenberg eBook #11, public domain), reduced to the Decoder’s 64-character alphabet: lower case, straight quotes, no italics marks.
A model can lower its training loss by memorising. Only text it has never seen shows whether it learnt something that carries over, so the end of the book is never trained on, and runs are compared on it (deck 01).
142,041 characters in all; SHA-256 8a2b46fd9524cc57… recorded in results.md §1, so every run can prove which text it saw. Unigram entropy of the training characters: 3.0000 nats.
A training example is a window of \(S + 1 = 9\) consecutive characters. \(S = 8\) is the model’s context length: how many tokens it sees at once (real models: thousands). The inputs are \(x_i = w_i\) and the targets \(y_i = w_{i+1}\), so one window gives 8 training signals.
Each step draws 16 windows at random places in the training text: the batch. A window starts at \(\lfloor u\,(N - S - 1)\rfloor\) for \(u\) uniform in \([0,1)\) from the seeded generator, \(N\) the number of training characters:
for (let b = 0; b < B; b++) {
const start = Math.floor(rng() * (train.length - S - 1));
const w = train.slice(start, start + S + 1);
starts.push(start);
inputs.push(w.slice(0, S));
targets.push(w.slice(1));
}The 2,000-step default run sees 256,000 characters: about 2.0 passes (epochs) over the training text.
The Decoder’s model reads characters, so it must spend its context and its layers assembling “the” from three pieces every time. Most real models read tokens: frequent fragments learnt from the text itself.
BPE (Sennrich, Haddow & Birch 2015) starts from single symbols and repeatedly merges the most frequent adjacent pair into a new symbol, until the vocabulary has the size wanted.
GPT-2 runs the same algorithm over bytes rather than characters, so any text can be encoded, with a vocabulary of 50,257 (Radford et al. 2019).
The site keeps training on characters, so that the model being trained is exactly the Decoder’s; the tokeniser is there to show how real models see text. With tokens, the same 8-position context covers more text, and a per-token loss is a larger number for the same skill (so perplexities compare only under one tokeniser).
Below, the site’s BPE (a port of scripts/training/data.py: words split as a leading space plus a run of non-spaces, ties broken by the smaller pair of ids) learns 12 merges on the book’s first paragraph. Step through it.
On the full training text the first merges are the language’s commonest fragments: “␣t”, “he”, “␣a”, “␣s”, “in”, and by the sixth “␣the”, a space and the whole word (the site’s chapter 2 animates them).
merges (a vocabulary of 128)
characters
tokens after the merges
characters per token
So an 8-token context would see about two thirds more text than 8 characters. Each merge removes exactly \(\operatorname{count}(a^\ast b^\ast)\) symbols from the corpus (fewer if the pair overlaps itself, as in “a a a”); ties go to the smaller pair of ids, so the Python reference and the browser learn identical merges.
Real corpora are many documents of different lengths. Rather than pad short ones, trainers pack: concatenate them into one long stream with a separator token and cut fixed-length windows, so no compute is wasted on padding.
A window can then straddle two documents. Some runs mask attention across the boundary, so a document never attends to the one before it. Llama 3’s report describes doing this and finds it matters most for very long sequences (Grattafiori et al. 2024, §3.2). With a single book there is no boundary here.
Small experiments often use small, simple corpora such as TinyStories, short stories written in a child’s vocabulary, on which models of a few million parameters already write fluent English (Eldan & Li 2023).
Duplicated text is memorised and wastes compute; removing near-duplicates helps (Lee et al. 2021).
Pretraining corpora are web crawls plus books, code and papers, in trillions of tokens; most of the work is quality filtering and choosing the mixture of sources. FineWeb documents one such pipeline in detail (Penedo et al. 2024).
How much data a model of a given size should see is a scaling question (deck 13). The Chinchilla analysis found about 20 tokens per parameter compute-optimal (Hoffmann et al. 2022).
127,837 training characters for 5,376 parameters: 23.8 characters per parameter, by coincidence in the same range.
The default run sees that text about 2.0 times; real runs mostly see each token once. Deck 13’s toy scaling sweep trains far past one pass, which is one reason its fitted law should not be over-read.
Animated: chapter 2, data and tokenisation · previous: deck 01 · next: deck 03, backpropagation