The residual stream

Inside the blocks

The idea

Every block reads from and writes back into one running vector per token - the residual stream. Attention adds, the MLP adds; nothing overwrites. The stream is the model’s working memory across layers.

The stream’s fixed depth has a consequence bigger than plumbing: a token’s answer can only use as much computation as one pass provides. Hard problems must spread across many tokens’ passes - the argument Tokens to think makes concrete (ch. 1:48).

The story, animated

The tempting picture is a replacement. What actually happens: one running vector, attention writes, the MLP writes, a quiet block gets out of the way.

Watch

The residual stream — attention and the MLP add, they do not overwrite · 18s, companion clip.

Where it sits

Worth having first: Attention

Unlocks: LayerNorm · Rows ask questions

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).