PaperlayerView exact revision on arXiv

Source-linked AI summary

ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices

Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, Huaisong Zhang, Chun Yuan

arXiv:2506.03737v1cs.CVcs.AI

TL;DR

Attention lacks inherent position information, while existing RoPE methods rely on manually designed rotation matrices with limited transformation space. ComRoPE formalizes RoPE through trainable commuting angle matrices and a necessary-and-sufficient commutativity condition. It surpasses LieRE on ImageNet-1K classification at training and higher resolutions.

  • Problem

    Attention lacks inherent positional information, and existing RoPE methods rely on restricted 2D rotation groups or manually designed matrices that can fail to preserve relative-position dependency.

    Source: §1 · pp. 1–2

  • Method

    ComRoPE formalizes RoPE functions parameterized by angle matrices and uses two types of trainable commuting matrix sets as sufficient solutions to the RoPE equation.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    ComRoPE surpasses LieRE by 1.6% at training resolution and 2.9% at higher resolutions on ImageNet-1K classification.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework generalizes existing RoPE formulations and offers insights for future positional-encoding research.

    Source: §1 · p. 2

    ComRoPE's matrix-exponential implementation is slow and memory-intensive on large models, and its strict commutativity requirements may restrict embedding expressiveness.

    Source: H. Limitations · p. 17

Abstract

from arXiv · show

The Transformer architecture has revolutionized various regions since it was proposed, and its effectiveness largely depends on the ability to encode positional information. Traditional position encoding methods exhibit significant limitations due to lack of robustness and flexibility of position. Therefore, Rotary Positional Encoding (RoPE) was proposed to alleviate these issues, which integrates positional information by rotating the embeddings in the attention mechanism. However, RoPE requires manually defined rotation matrices with limited transformation space, constraining the model's capacity. In this work, we propose ComRoPE, which generalizes RoPE by defining it in terms of trainable commuting angle matrices. Specifically, we demonstrate that pairwise commutativity of these matrices is essential for RoPE to achieve scalability and positional robustness. We formally define the RoPE Equation, which is an essential condition that ensures consistent performance with position offsets. Based on the theoretical analysis, we present two types of trainable commuting angle matrices as sufficient solutions to the RoPE equation, which significantly improve performance, surpassing the current state-of-the-art method by 1.6% at training resolution and 2.9% at higher resolution on the ImageNet-1K dataset. Furthermore, our framework shows versatility in generalizing to existing RoPE formulations and offering new insights for future positional encoding research. To ensure reproducibility, the source code and instructions are available at https://github.com/Longin-Yu/ComRoPE

1. Introduction

Positional encodings supply the position information that attention lacks, and RoPE encodes relative positions by rotating token embeddings. ComRoPE generalizes RoPE with trainable commuting angle matrices to address limits in existing rotation designs while preserving positional robustness.

1. Introduction

  • Attention lacks inherent position information, so positional embeddings are added to represent sequence order and structure.Source: §1 · p. 1
  • Existing RoPE methods rely on 2D rotation groups, often use manually designed matrices, and can struggle to preserve relative-position dependency.Source: §1 · pp. 1–2
  • Pairwise commutativity is a necessary and sufficient condition for RoPE functions parameterized by angle matrices, unifying existing RoPE variants.Source: §1 · p. 2
  • ComRoPE surpasses LieRE by 1.6% at training resolution and 2.9% at higher resolutions on ImageNet-1K classification.Source: §1 · p. 2
  • ComRoPE uses two types of trainable commuting angle matrix sets to provide sufficient solutions to the RoPE equation without manual design.Source: §1 · p. 2

2. Related work

Related work traces positional encoding from absolute embeddings to relative methods, then describes RoPE and its extensions for language and vision Transformers.

2.1. Position information in attention

  • APE adds fixed positional embeddings to inputs but is limited on long sequences and sensitive to positional shifts; RPE instead represents relative positions within attention.Source: §2.1 · p. 2

2.2. Rotary position embedding

  • RoPE combines APE and RPE, while 2D-RoPE extensions include LieRE, axial and mixed axial frequencies, and applications in multimodal and vision models.Source: §2.2 · p. 2

3. Method

ComRoPE formalizes when angle-matrix RoPE depends only on relative position and constructs commuting trainable matrices to satisfy that condition. It also normalizes image coordinates and perturbs patch locations during training.

3.1. Preliminaries

  • The method formalizes RPE and RoPE for multi-axial attention, with the RoPE Equation requiring position encoding to depend only on relative positions.The formal definitions specify a rotation-matrix encoding and dot-product similarity; this relative-position dependence makes the encoding robust to offsets.Source: §3.1 · p. 3

3.2. Main theorems

  • Pairwise commutativity of the angle matrices is necessary and sufficient for rotation differences to depend only on location differences, yielding offset-robust RoPE.The paper calls this framework ComRoPE and uses the condition to guide construction of its angle matrices.Source: §3.2 · p. 4

3.3. Construction of pairwise commuting matrices

  • ComRoPE-AP partitions angle-matrix blocks among axes, making all but one matrix zero in each set; the number of blocks must be a multiple of the axes.This construction follows the sufficient condition that a set commutes when all but one matrix are zero.Source: §3.3 · p. 5
  • ComRoPE-LD scales a shared base matrix to form linearly dependent angle matrices, providing a second construction that satisfies the commuting requirement.The construction uses a base matrix P and scaling factors θ_i.Source: §3.3 · p. 5

3.4.1 Relative scaling and center offset

  • For images, the method scales each spatial dimension to 1 and uses relative coordinates because patch-span relationships can vary with patch size.A patch’s center is used as its aggregation location in the feature tensor.Source: §3.4.1 · p. 5

3.4.2 Position perturbation

  • During training, patch locations are perturbed around their centers with intensity σ, and the resulting locations are truncated to the patch area.The perturbation scale is based on patch size.Source: §3.4.2 · p. 5

4. Experiments

The experiments assess positional encoding on ImageNet-1K classification across resolutions, object detection on MS COCO, and additional robustness, block-size, perturbation, and integration properties. ComRoPE variants are evaluated against APE, vanilla RoPE, and LieRE.

4.1.1 Setup

  • Models are trained at 224×224 and evaluated across resolutions; Figure 3 groups results by perturbation intensity and omits 112×112.The comparison uses ViT-B/16 on ImageNet-1K, with models trained from scratch.Source: §4.1.1 · p. 6, Figure 3 · p. 6

4.1.2 Main results

  • At 224×224, ComRoPE-LD surpasses LieRE by 1.6%; at 512×512, it outperforms LieRE by 2.9%.The trainable-angle methods outperform APE and vanilla RoPE across resolutions; ComRoPE-LD performs best among the three trainable variants.Source: §4.1.2 · pp. 6–7

4.2. Object detection

  • On MS COCO, ComRoPE-LD slightly outperforms LieRE with nearly half its extra parameters; ComRoPE-LD and LieRE need 29% fewer iterations than APE to reach the same results.Table 2 reports object-detection results; Figure 4 shows results over the training procedure.Source: Table 2 · p. 7, §4.2 · p. 7, Figure 4 · p. 7

4.3.1 Impact of commutativity

  • With increasing coordinate-offset standard deviation, LieRE performance deteriorates while ComRoPE remains consistent, attributed to commuting angle matrices.The shared random offset leaves relative positional dependencies between patches unchanged.Source: §4.3.1 · p. 7, Figure 5 · p. 8

4.3.2 Impact of block size

  • Across block sizes 2–8, larger blocks improve accuracy by expanding the rotation space, at the cost of additional parameters and computation.The authors cap block size at 8 to balance performance and computational cost.Source: §4.3.2 · pp. 7–8, Figure 6 · p. 8

4.3.3 Utility of position perturbation

  • Increasing perturbation intensity from 0 to 1 improves APE by 19.5%, while angle-matrix RoPE gains 2.9%.The authors attribute the smaller gain for angle-matrix RoPE to its inherent robustness to positional variation.Source: §4.3.3 · p. 8, Figure 7 · p. 8

4.4. Applications

  • With an all-zero angle matrix, ComRoPE reduces to standard attention; with block size 2, ComRoPE-AP reduces to commonly used RoPE attention.The paper states that these properties allow ComRoPE to be integrated during fine-tuning even when it was not used during pre-training.Source: §4.4 · p. 8

5. Conclusion

ComRoPE generalizes RoPE using trainable angle matrices and formalizes the condition for consistent behavior under position offsets. The paper reports improved performance across tasks and generalization to existing RoPE formulations.

5. Conclusion

  • ComRoPE parameterizes Rotary Position Embedding with trainable angle matrices and formalizes a necessary-and-sufficient condition for the RoPE Equation.The framework is presented as a more flexible and scalable alternative to manually designed rotation matrices.Source: §5 · p. 8
  • Experiments show that ComRoPE outperforms existing positional encoding methods across various tasks.Source: §5 · p. 8
  • The framework generalizes existing RoPE formulations and offers a foundation for further positional encoding research.Source: §5 · p. 8

A.1. Proof of the main theorem

This appendix proves the commuting-matrix condition underlying the main theorem, explains the exponential representation of rotation matrices, and reports 3D classification results across resolutions.

A.1. Proof of the main theorem

  • Pairwise-commuting matrices satisfy the exponential product identity, which the proof uses to establish the commuting-matrix condition in both directions.For two matrices, the proof states that e^(Ax)e^(By) = e^(Ax+By) for all real x and y if and only if A and B commute.Source: §A.1 · p. 11

B.1. 3D classification

  • ComRoPE performs best when UCF-101 evaluation resolution exceeds the 224 × 224 training resolution, showing resolution robustness in 3D classification.Table 3 reports accuracy for models trained at 224 × 224 and evaluated at varying resolutions.Source: Table 3 · p. 12, §B.1 · p. 13

A.2. Explanation of rotation matrix and its exponential representation

  • Every rotation matrix can be represented as the exponential of a skew-symmetric matrix.The appendix identifies the rotation matrices with SO(n) and the associated skew-symmetric matrices with its Lie algebra.Source: §A.2 · pp. 12–13

B.2. Fine-tune on pre-trained model

ComRoPE can replace standard attention or vanilla RoPE during fine-tuning, including for models not using it in pre-training. The section also reports fine-tuning comparisons and provides experiment-configuration details.

B.2. Fine-tune on pre-trained model

  • Zero angle matrices recover standard attention, and the Appendix D initialization recovers vanilla RoPE, allowing either attention type to be replaced during fine-tuning.Pre-trained weights can be loaded and fine-tuned with ComRoPE even when it was not used during pre-training.Source: §B.2 · p. 13
  • ComRoPE-LD achieves the best performance among the fine-tuned methods, while vanilla RoPE has the lowest accuracy among all five methods.Table 4 reports accuracy for fine-tuned models using different positional encoding methods on ImageNet.Source: Table 4 · p. 13, §B.2 · p. 13

C.1. Configuration of 2D classification

C.2. Configuration of 3D classification

  • For 3D experiments, the authors modify model parameters because vanilla RoPE and ComRoPE-AP require head dimensions to be multiples of the coordinate dimension, and use a smaller model for UCF-101.Table 6 gives the 3D model and training configuration.Source: §C.2 · p. 13, Table 6 · p. 14

D. Reformulation of baseline RoPE methods in detail

The appendix expresses vanilla RoPE as a special case of ComRoPE-AP and describes the trainable block construction used for LieRE.

D.1. Vanilla RoPE

  • Vanilla RoPE represents each 2D sine-and-cosine rotation as an exponential of a skew-symmetric matrix and is a ComRoPE-AP special case with block size 2 and non-trainable P_j.In practice, RoFormer uses θ = 10000−1 as the rotation-base hyperparameter.Source: §D.1 · p. 14

D.2. LieRE

  • LieRE uses independent trainable blocks, with each B_ij defined from a trainable matrix P_ij.Source: §D.2 · p. 14

E. Analysis and comparison of complexity and extra consumption

The section compares positional-encoding methods by parameter and time costs, then examines angle-matrix element distributions from 2D classification experiments. ComRoPE-LD shows the broadest distribution and largest overall variance.

E. Analysis and comparison of complexity and extra consumption

E.1. APE

  • APE adds a position-code tensor of n × d parameters and incurs O(n × d) extra time complexity.Its extra computation is adding position embeddings to the original features.Source: §E.1 · p. 14

E.2. RoPE parameterized by angle matrices

  • The angle-matrix RoPE algorithm combines axis-wise angle bases with patch positions, then uses the resulting rotations to transform queries and keys.Its stated inputs include query and key matrices, an angle-base matrix, and patch positions; the outputs are rotated query and key matrices.Source: §E.2 · p. 14

E.2.1 Vanilla RoPE

  • Vanilla RoPE uses block size b = 2, has no extra parameters, and can calculate its angle-base matrix during preprocessing.The comparison describes its extra parameter count as zero.Source: §E.2.1 · pp. 14–15

E.2.2 LieRE

  • LieRE forms each angle-base matrix as A = P − P⊤, with independent parameters in P.The passage identifies the subtraction used to obtain A as an additional step.Source: §E.2.2 · p. 15

E.2.3 ComRoPE-AP

  • ComRoPE-AP constructs angle-base matrices by zeroing blocks irrelevant to the corresponding coordinate axes.The parameter count is reported as db.Source: §E.2.3 · p. 15

E.2.4 ComRoPE-LD

  • ComRoPE-LD makes angle-base matrices linearly dependent across axes and represents them with a shared matrix and multiplication factors.The described dependencies are along the first, axis dimension.Source: §E.2.4 · p. 15

F. Distribution of elements in angle matrices

  • ComRoPE-LD has the largest overall variance; its distribution extends farther from zero, likely reflecting axis-wise linear dependencies that require frequency differences.The analysis extracts upper-triangular matrix elements and presents their density on linear and logarithmic scales.Source: F. Distribution of elements in angle… · p. 15, Table 8 · p. 15

G. More Analysis

Under the DeiT recipe, ComRoPE-LD consistently outperforms RoPE-Mixed. The analysis also notes computation costs and a memory bottleneck for end-to-end training on large language models.

Computational complexity and time consumption

  • Small block sizes should have minimal impact, but torch.matrix exp optimization issues lower GPU utilization and increase time costs for LieRE and ComRoPE.Computational costs are compared on A800×4.Source: G. More Analysis · p. 15, Table 9 · p. 16

Application to LLMs

  • ComRoPE can replace rotary embeddings in most pretrained LLM checkpoints without architectural changes, but current matrix-exponential memory overhead makes end-to-end large-model training prohibitively expensive.For single-sequence language modeling, the angle matrices’ commuting property holds automatically.Source: G. More Analysis · p. 16

Implementation with sota codebase and settings

  • ComRoPE-LD consistently outperforms RoPE-Mixed under the DeiT recipe, where RoPE-Mixed uses ComRoPE-LD with 2 × 2 blocks.The comparison is reported in Table 10.Source: G. More Analysis · p. 16, Table 10 · p. 17

H. Limitations

The authors identify computational overhead and strict commutativity requirements as constraints on ComRoPE.

H. Limitations

  • The current implementation is slow and memory-intensive on large models, while strict commutativity requirements may restrict embedding expressiveness.The authors identify reducing training time and GPU memory use, and finding weaker sufficient conditions, as future-work priorities.Source: H. Limitations · p. 17

Related papers

Loading 2506.03737v1…