LayerNorm

Inside the blocks

The idea

Between blocks, each vector is rescaled to a standard size before being read. A plumbing detail with outsized importance: it keeps hundreds of successive additions from blowing up or dying out.

In the walkable 3D model Karpathy tours (bbycroft.net/llm), this is the first stop inside every block: normalise, then the matrix products, softmax, attention, MLP - a plain checklist for what one block contains, with the normalisations as the quiet joints between the loud parts (ch. 0:24).

The story, animated

The residual additions can blow up. LayerNorm rescales this token — then Pre-LN leaves the highway itself clean.

Watch

LayerNorm — same shape, standard size; Pre-LN leaves the highway clean · 20s, companion clip.

Where it sits

Worth having first: The residual stream

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).