Attention & memory

Attention is the mechanism that does the pushing foundations promised: every token asks a question, every earlier token offers a label and a payload, and a softmax over the row decides who hands over what. That’s the whole idea - the rest of this track is what it costs to run.

Each token turns itself into a query, a key and a value; heads run many of these matchmakings in parallel. From there the path forks in two directions that both matter: RoPE tells the mechanism where each token sits (position has to be injected - attention alone is order-blind), and the KV cache is what makes generating the next token affordable at all, by remembering instead of recomputing. Remembering is not free - grow the context far enough and the cache itself becomes the bottleneck. That bottleneck is the memory wall, and it’s the reason the efficiency lineage exists.

1. Attention - every token asks a question, every earlier one offers a key and a value.
2. Queries, keys, values & heads - each token becomes three things of itself; heads run the matchmaking in parallel.
3. RoPE - position is injected by rotation, not by adding a vector.
4. KV cache - ask once, remember twice; the price is a cache that grows with every token.
5. The memory wall - at long context, memory - not math - is what stops you.

Also: Attention by shapes — eleven live matrix-flow diagrams (QKV → self/cross → MHA → MQA/GQA) where dragging a dimension recomputes every box. Parallel track; the pages above stay the word-token story.

Next: The efficiency lineage, a decade of architecture answering one question - keep the cache, shrink it, or replace it?