Cross Attention

Attention by shapes · 4 of 11

The idea

What if the answers live in a different pile of tokens? A decoder generating French looks at an English sentence; a captioner looks at image patches; a RAG reader looks at retrieved chunks. Queries still come from X, but K and V come from a second sequence E.

The shapes

X and E need not share embedding size. After projection they must meet in the same key dimension. Unequal lengths make S a rectangle M×N — the fingerprint of cross-attention.

Live diagram

Next: Self vs Cross