Running the model & the world

Training ends; this track is everything after. It’s the least linear track on the site on purpose - it’s two clusters of pages, not one chain.

Making it fast and affordable. Sampling turns a distribution into a word; speculative decoding makes the expensive model a checker instead of a typist; quantization stores the same numbers smaller; the serving layer manages the KV cache economics of running many requests at once; the machine underneath explains why memory bandwidth, not arithmetic, decided which ideas won; and batching is the GPU-throughput discipline that ties all of it together - including the token-sorting Mixture of Experts needs to run fast.

What it still can’t do. A model never consults a store of facts - it samples plausible continuations, which is why hallucination and tool use are one problem, not two. Jagged intelligence is the same root cause showing up as bafflingly uneven capability. What it can’t remember is the gap between weights (permanent, expensive to change) and context (cheap, temporary). What’s coming is the open list: several of these are unsolved, not solved.

Sampling - the model outputs a distribution; something has to pick the word.
Speculative decoding - a small model drafts, the big model verifies in one pass.
Quantization - the same numbers, stored in fewer bits.
The serving layer - prefix caching, paged KV memory, continuous batching.
The machine underneath - the ideas that won are the ideas the hardware liked.
Batching & GPU kernels - one big contiguous multiply beats a thousand scattered small ones.
Hallucination & tool use - the model doesn’t know it’s guessing; tools give it something checkable to read.
Jagged intelligence - olympiad problems solved, letter-counting sometimes missed.
What it can’t remember - facts live in weights; conversation lives in context.
What’s coming - the open fronts, extending machinery already on this map.

Next: World models, a different objective - predict the next state of the world, not the next token.