Rotary Position Embeddings encode token position by rotating queries and keys

Transformers lack inherent sequence order, so early models added position vectors to token embeddings, which blended meaning and location in one sum. Rotary Position Embeddings (RoPE), introduced as RoFormer in 2021, instead rotate query and key vectors by an angle tied to their absolute position. Because the rotations partly cancel in the dot product, the resulting attention score depends only on the relative distance between tokens. In practice, vectors are split into coordinate pairs rotated at different frequencies, giving the model a multi-scale sense of distance, and the same operation can be expressed with complex-number multiplication.