RoPE Explained: How Rotating Query-Key Vectors Gives Transformers Long-Context Memory

Rotary Position Embedding (RoPE), introduced by Su et al. in 2021, addresses a core limitation of earlier positional encoding methods used in transformer-based language models. Traditional absolute encodings added sine-cosine vectors directly to input embeddings, causing models to fail on sequences longer than their training length. Relative bias approaches patched the attention matrix but broke GPU-level optimizations like FlashAttention and increased memory overhead. RoPE instead rotates Query and Key vectors in 2D sub-planes before attention is computed, ensuring that dot products depend only on the relative distance between tokens, not their absolute positions. This design preserves vector norms, remains compatible with efficient attention kernels, and scales naturally to longer contexts without retraining.