Softmax & temperature

Foundations

The idea

A list of scores becomes a probability distribution: exponentiate, then normalise. Temperature divides the scores first - same ranking, different sharpness. Try it below.

The output contract is worth stating precisely: at every step the model emits one probability for every vocabulary entry - at GPT-4 scale, 100,277 numbers, a full distribution over what could come next (ch. 0:17). Picking a word from it is a biased dice roll, which is the whole subject of Sampling (ch. 0:26).

Try it

Where it sits

Worth having first: Embeddings

Unlocks: The next-token objective · Sampling

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).