RoPE
Attention & memory · 2021
The idea
Section titled “The idea”Attention is order-blind on its own; position has to be injected. RoPE does it by rotation.
RoPE replaced the original Transformer’s fixed sinusoidal position encoding - added once, to the embedding, before any attention happened. RoPE instead rotates the query and key vectors inside attention, every layer, which is what lets the dot product see relative offset instead of absolute position.
The story, animated
Section titled “The story, animated”Same bag of words, two readings. Rotate the query and the key by their seat — the dot product sees the offset, not the seat numbers.
Verdict
Section titled “Verdict”What it fixedRelative position falls out of the geometry - queries and keys are rotated by position, so the dot product sees the offset.
What it still costsStretch far past the trained context and the rotations land in territory the model never saw.
Where it sits
Section titled “Where it sits”Worth having first: Queries, keys, values & heads