Source-linked AI summary
RoFormer: Enhanced Transformer with Rotary Position Embedding
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu
TL;DR
Transformer self-attention needs positional information, but existing approaches commonly add it to context representations and are unsuitable for linear self-attention. The paper proposes RoPE, which combines absolute-position rotations with explicit relative dependencies, and evaluates RoFormer across long-text benchmarks. RoFormer consistently outperforms alternatives, while the authors note that some convergence and long-text performance explanations remain unresolved.
Problem
Transformer self-attention is position-agnostic, while existing positional approaches commonly add position information to context representations and are unsuitable for linear self-attention.
Method
RoPE encodes absolute positions with rotation matrices and incorporates explicit relative-position dependency into self-attention.
Results
RoFormer consistently achieves better performance than alternative methods across various long-text benchmark datasets.
Takeaways & Limitations
RoPE provides sequence-length flexibility, distance-based dependency decay, and compatibility with relative position encoding in linear self-attention.
Takeaways & Limitations
The authors lack faithful explanations for faster convergence than other position-encoding strategies and for superior long-text performance despite similar long-term decay properties.
Abstract
from arXiv · showhide
Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \url{https://huggingface.co/docs/transformers/model_doc/roformer}.
1 Introduction
Transformer language models require positional information because self-attention is position-agnostic, motivating RoPE, which combines absolute-position rotations with explicit relative dependencies. RoFormer evaluates this approach on long-text benchmarks and reports better performance than alternatives.
- Self-attention in current pretrained language models is position-agnostic, motivating methods that encode sequence order during learning.
- RoPE encodes absolute position with a rotation matrix while incorporating explicit relative-position dependency into self-attention.Its key idea is multiplying context representations by a rotation matrix to encode relative position.
- RoPE is presented as supporting flexible sequence lengths, distance-based decay of inter-token dependency, and relative position encoding for linear self-attention.The paper contrasts these properties with approaches that add position information to context representations.
- The paper examines existing positional-encoding approaches, develops RoPE, studies its properties, and reports experiments.
- RoFormer is evaluated on various long-text benchmark datasets and consistently achieves better performance than alternative methods.
2 Background and Related Work
Transformer position encoding methods incorporate positional information into self-attention through absolute or relative representations. RoPE instead derives relative position encoding by rotating context representations rather than directly adding position embeddings.
- Preliminary: Self-attention incorporates positional information into word embeddings before transforming them into queries, keys, and values.Queries and keys compute attention weights, while values produce the weighted-sum output.
- Absolute position embedding: Existing transformer position-encoding work mainly chooses functions for adding positional information to contextual representations.Absolute encodings may be generated sinusoidally or represented by trainable position vectors.
- Absolute position embedding: Sinusoidal position functions provide position-dependent vectors, whereas RoPE uses sinusoidal functions to multiply context representations and encode relative position.
- Relative position embedding: Relative-position approaches include clipped trainable relative embeddings and modifications that replace or decompose absolute-position terms in attention.Clipping represents relative distance only within a specified range, based on the hypothesis that precise distance beyond that range is unhelpful.
- Relative position embedding: These approaches commonly derive relative encoding by decomposing self-attention and adding position information to context representations.RoPE instead derives relative position encoding under constraints and interprets it through rotations of context representations.
3 Proposed approach
RoPE solves relative position encoding by rotating transformed token representations, so self-attention depends explicitly on relative positions while retaining absolute position information. Its formulation also supports long-distance decay and integration with linear attention.
- Relative position formulation: RoPE formulates relative position encoding by requiring query-key inner products to depend on word embeddings and relative position m − n.The encoding functions transform position-indexed queries and keys so their inner product conforms to this relative-position formulation.
- 2D construction: In the 2D case, RoPE rotates affine-transformed word embeddings by angle multiples of their position indices.The construction uses complex representations and a preset non-zero angle parameter.
- General form: For even-dimensional representations, RoPE divides the space into d/2 two-dimensional subspaces and applies paired rotation blocks.The resulting rotary matrix uses predefined parameters Θ with dimension-specific angular frequencies.
- Efficient realization: The rotary matrix is orthogonal, preserving stability during position encoding, while its sparse structure permits a more computationally efficient multiplication realization.The paper notes that direct matrix multiplication is inefficient and motivates an alternative implementation.
- General form: RoPE is multiplicative rather than additive and incorporates relative position through products of rotation matrices in self-attention.This avoids altering expanded additive-position terms when relative position information is introduced.
- Properties: With θ_i = 10000^-2i/d, RoPE exhibits long-term decay: the inner product decreases as relative position increases.This matches the intended weaker connection between tokens separated by larger relative distances.
- Properties: RoPE can be combined with linear attention because rotation preserves hidden-representation norms and can be applied to outputs of non-negative feature functions.The linear-attention formulation avoids the original pairwise query-key computation with quadratic complexity O(N^2).
4 Experiments and Evaluation
RoFormer is evaluated across translation, pre-training, downstream classification, linear attention, and Chinese long-text tasks. It generally improves convergence or performance over the corresponding baselines, while the authors note unresolved explanations for some gains and hardware requirements for pre-training.
- Machine translation: RoFormer achieves better BLEU scores than the baseline Transformer on WMT 2014 English-to-German translation.Both models were trained under the same settings.
- Language-model pre-training: RoFormer converges faster than vanilla BERT during masked-language-model pre-training.The comparison uses MLM loss over training, with BERT as the baseline.
- Downstream tasks: RoFormer significantly outperforms BERT on three of six GLUE datasets after fine-tuning.The evaluation uses task-specific metrics including F1-score, Spearman correlation, and accuracy.
- Linear attention: Adding RoPE to Performer produces rapid convergence and lower loss under the same training steps while retaining linear attention complexity.The experiment compares Performer with and without RoPE on Enwik8 language-model pre-training.
- Chinese long-text evaluation: At a maximum input length of 1024, RoFormer outperforms WoBERT by an absolute 1.5% on CAIL2019-SCM.At length 512, RoFormer is comparable to WoBERT and slightly better than BERT; the Chinese experiments also vary sequence-length bounds during pre-training.
- Limitations: The authors do not thoroughly explain RoFormer’s faster convergence or its superior long-text performance, and the Transformer-based design requires hardware resources for pre-training.These are stated limitations of the work.
5 Conclusions
RoFormer introduces a position embedding method that incorporates explicit relative position dependency into Transformer self-attention. Experiments on English and Chinese benchmarks show faster pre-training convergence and better performance on long-text tasks.
- RoFormer combines absolute position encoding through a rotation matrix with explicit relative position dependency in self-attention.Theoretical analysis formulates relative position through vector products in self-attention.
- Experiments on English and Chinese benchmark datasets show faster pre-training convergence and better performance on long-text tasks.