RoPE: How 2D Rotations Solved Transformer Long-Context

Short summary

This article provides a mechanics-first explanation of Rotary Position Embedding (RoPE), showing how 2D rotations of Query and Key vectors make attention depend only on relative token distance. It contrasts RoPE with absolute encodings and relative bias approaches, explaining why RoPE preserves FlashAttention kernel fusions. A production-ready PyTorch implementation with cosine/sine caching is included.

• •RoPE rotates Q and K vectors in 2D sub-planes so attention depends only on relative distance (m-n)
• •Unlike relative bias methods, RoPE doesn't modify the N×N attention matrix, preserving FlashAttention fusions
• •Includes a vectorized PyTorch implementation using the real-valued slice trick with precomputed cos/sin caches

Generated with AI, which can make mistakes.