Sampling

Running the model & the world

The idea

The model outputs a distribution; something must pick the word. Greedy takes the top, temperature reshapes, top-p trims the tail. Generation quality lives in this last, cheap step.

Proof the dice are real: run the same prompt three times against an open model and three differently-worded, equally-valid solutions come back (ch. 2:20-2:21). The dark twin of that demo: ask a weak model about an invented name three times and get three contradictory, equally confident biographies - sampling is not retrieval, and it never was (ch. 1:23; see Hallucination & tool use).

The story, animated

A pile, then a pick. Same prompt, three rolls. Greedy, temperature, top-p — one pile, three rules.

Watch

Sampling — a pile, then a pick; the dice are real · 18s, companion clip.

Where it sits

Worth having first: Softmax & temperature · The next-token objective

Unlocks: Speculative decoding

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).