PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Representational Position Encoding

Yifan Zhang, Zixiang Chen, Yifeng Liu, Zhen Qin, Huizhuo Yuan, Kangping Xu, Yang Yuan, Quanquan Gu, Andrew Chi-Chih Yao

arXiv:2512.07805v6cs.LGcs.AIcs.CL

TL;DR

Existing positional mechanisms offer different properties: RoPE uses fixed coordinate planes, while absolute codes and table-based relatives have other limitations. GRAPE unifies multiplicative rotations and additive logit biases through group actions, recovering familiar mechanisms as exact cases. Its formulation also includes learned and contextual extensions with streaming and cache support.

  • Problem

    RoPE fixes coordinate planes and typically a log-uniform spectrum, while absolute codes break translation equivariance and table-based relatives add window-dependent overhead.

    Source: §1 · p. 1

  • Method

    GRAPE represents positional mechanisms as group actions, using rank-2 skew generators for multiplicative rotations and unipotent lifts for additive logit biases.

    Source: §1 · p. 1, Additive GRAPE · pp. 2–3

  • Results

    GRAPE recovers RoPE and ALiBi as exact special cases and proves FoX is an exact instance, with multiplicative relative laws holding for commuting and coupled non-commuting subspaces.

    Source: Additive GRAPE · p. 3, §2.2 · p. 4

  • Takeaways & Limitations

    The framework offers an extensible design space with learned-basis and contextual variants, while retaining streaming and cacheability for additive mechanisms.

    Source: Additive GRAPE · p. 3, §7 · p. 10

    Canonical RoPE equality requires coordinate 2D planes and the canonical log-uniform spectrum.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

We present GRAPE (Group Representational Position Encoding), a unified framework for positional encoding based on group actions. GRAPE unifies two families of mechanisms: (i) multiplicative rotations (Multiplicative GRAPE) in $\operatorname{SO}(d)$ and (ii) additive logit biases (Additive GRAPE) arising from unipotent actions in the general linear group $\mathrm{GL}$. In Multiplicative GRAPE, a position $n \in \mathbb{Z}$ (or $t \in \mathbb{R}$) acts as $\mathbf{G}(n) = \exp(n \, \omega \, \mathbf{L})$ with a rank-2 skew-symmetric generator $\mathbf{L} \in \mathbb{R}^{d \times d}$, yielding a relative, compositional, norm-preserving map with a closed-form matrix exponential. RoPE is recovered exactly when the $d/2$ planes correspond to canonical coordinate pairs with a log-uniform spectrum. Learned commuting subspaces and compact non-commuting mixtures strictly extend this geometry to capture cross-subspace feature coupling at $O(d)$ and $O(r d)$ cost per head, respectively. In Additive GRAPE, additive logits arise from rank-1 (or low-rank) unipotent actions, recovering ALiBi and the Forgetting Transformer (FoX) as exact special cases while preserving an exact relative law and streaming cacheability. Overall, GRAPE provides a principled design space for positional geometry in long-context models, subsuming RoPE and ALiBi as special cases. Project page: https://github.com/model-architectures/GRAPE.

1 INTRODUCTION

GRAPE frames positional encoding as group actions, bringing multiplicative rotations and additive logit biases into one framework. Its additive family uses unipotent actions to recover ALiBi and FoX and support contextual extensions.

1 INTRODUCTION

  • GRAPE unifies norm-preserving multiplicative rotations in SO(d) with additive positional mechanisms represented as unipotent actions in GL.The framework is motivated by RoPE’s fixed coordinate planes and typical log-uniform spectrum, which limit cross-subspace coupling and contextual phase warping.Source: §1 · p. 1

Additive GRAPE

  • GRAPE also proposes contextual and path-integral additive forms while retaining linear-in-offset logit biases.The introduction describes these as principled, streaming-friendly contextual extensions.Source: Additive GRAPE · pp. 2–3
  • Multiplicative GRAPE encodes positions with rank-2 skew generators, while Additive GRAPE uses homogeneous lifts to produce linear-in-offset logit biases.The multiplicative map is an isometry with an exact relative law; the additive formulation recovers ALiBi and FoX.Source: Additive GRAPE · p. 2
  • The framework recovers RoPE and ALiBi as exact special cases and establishes FoX as an exact instance.Its contributions include fast closed-form multiplicative maps and streaming-cacheable additive biases, including query- or key-gated slopes and path-integral forms.Source: Additive GRAPE · pp. 2–3

2 MULTIPLICATIVE GROUP REPRESENTATIONAL POSITION ENCODING

Multiplicative GRAPE maps positions to rotations generated by rank-2 skew-symmetric operators. Its one-parameter subgroup structure gives exact relative attention, and its closed-form action is efficient to apply.

2.1 PRELIMINARIES AND RANK-2 GENERATOR

  • A rank-2 skew generator built from vectors a and b acts on their spanned plane, with spectrum ±is and zero eigenvalues elsewhere.The paper fixes an initialization gauge with unit-norm, orthogonal vectors and absorbs scale into ω.Source: §2.1 · p. 3

2.2 EXACT RELATIVE LAW

  • The one-parameter subgroup law makes multiplicative GRAPE exactly relative, whether its generator combines commuting or coupled non-commuting subspaces.For rotations, the group inverse is the transpose, so interactions depend only on offsets.Source: §2.2 · p. 4

2.3 CLOSED-FORM FAST MATRIX EXPONENTIAL

  • A rank-2 generator’s exponential has a closed-form Rodrigues-type expression, rotating span{a,b} while leaving its orthogonal complement unchanged.Applying the map requires O(d) flops without materializing the full matrix.Source: §2.3 · p. 4

2.4 THE b = J a CONSTRAINT

  • With b = J a, the map becomes a planar rotation; when ||a||=1 its angle is nω, and otherwise its effective frequency is ω||a||².The scale of a can therefore be absorbed into ω.Source: §2.4 · p. 4

2.5 APPLICATION TO RELATIVE ENCODING AND EQUIVARIANCE

  • Applying the map to queries and keys makes each attention score depend only on the relative offset j − i, not absolute positions.In streaming, each key is transformed once when it arrives; no cache rotation is needed as the query position advances.Source: §2.5 · p. 5

3 MULTI-SUBSPACE MULTIPLICATIVE GRAPE

Multi-Subspace GRAPE combines rank-2 generators on orthogonal planes, forming a commuting construction that includes RoPE. Learned bases and frequencies extend the coordinate-plane formulation.

3 MULTI-SUBSPACE MULTIPLICATIVE GRAPE

  • Combining generators across distinct orthogonal 2D planes yields a commuting multi-subspace GRAPE construction for encoding position across the hidden dimension.A single rank-2 generator affects only one plane; multiple generators extend the action across the space.Source: §3 · p. 5, §3.1 · p. 5

3.1 MULTI-SUBSPACE GRAPE-M AND ROPE AS A SPECIAL CASE

  • Multi-Subspace GRAPE equals RoPE when its planes are coordinate pairs and its frequencies follow the canonical log-uniform spectrum.In that case, the map is a block-diagonal product of planar rotations.Source: §3.1 · p. 5
  • GRAPE can learn or tie per-plane frequencies and replace coordinate selectors with a learned orthogonal basis.This preserves commuting subspaces while allowing learned planes.Source: §3.1 · p. 5

4 ADDITIVE GROUP REPRESENTATIONAL POSITION ENCODING

Additive GRAPE frames additive positional mechanisms as unipotent actions in GL, preserving an exact relative law and streaming/cache rules. Homogeneous lifts yield content-adaptive biases, with ALiBi and FoX recovered as exact instances.

4 ADDITIVE GROUP REPRESENTATIONAL POSITION ENCODING

  • The lifted unipotent formulation gives additive positional mechanisms, including ALiBi, an exact relative law and streaming/cache rules.The framework covers absolute feature shifts as well as additive logit biases.Source: §4 · p. 6

4.1 HOMOGENEOUS LIFT AND A UNIPOTENT ACTION

  • A homogeneous lift embeds affine transformations in a higher-dimensional linear space, where a nilpotent generator with A^2 = 0 exponentiates to a unipotent action.The lift augments x in R^d to a vector in R^(d+k) and works in GL(d+k).Source: §4.1 · p. 6
  • GRAPE-A-QK derives a learnable content-adaptive linear bias from query- and key-dependent unipotent actions; its slope is non-positive for causal offsets.The contribution is (j−i)ω(λq + λk) for j ≤ i.Source: §4.1 · p. 6

4.2 EXACT ALIBI AS A RANK-1 UNIPOTENT IN GL(d+2)

  • ALiBi is exactly a rank-1 unipotent GL(d+2) action, whose two constant coordinates enable a content-independent scalar logit bias.The two coordinates represent query and key, avoiding self-loops that would violate the generator's trace-zero property.Source: §4.2 · p. 7
  • FoX's forgetting bias coincides with a cumulative sum of per-token log forget scalars, making FoX an exact additive GRAPE instance.The per-token forget scalars lie in (0, 1], and their logarithms form the additive bias.Source: §4.2 · p. 7

5 PATH INTEGRAL ADDITIVE GRAPE

Path-Integral Additive GRAPE builds additive attention biases by summing endpoint-indexed edge potentials along a path. Endpoint-independent potentials recover familiar relative biases, while endpoint-dependent potentials allow more expressive path-integral biases with row-wise composition.

5 PATH INTEGRAL ADDITIVE GRAPE

  • For each query endpoint, the bias sums causal edge potentials from key position j to query position t, combining with either raw or orthogonally rotary attention logits.The link function is monotone increasing and 1-Lipschitz; the experiments use log(Sigmoid(z)).Source: §5 · p. 7
  • The path-product construction uses a fixed rank-1 nilpotent E with E^2 = 0, so factors along the path collapse additively.The construction defines endpoint-indexed unipotent factors for each fixed query endpoint.Source: §5 · p. 8
  • GRAPE-AP strictly contains GRAPE-A: endpoint-independent edge potentials give GRAPE-A, while endpoint-dependent potentials permit more expressive path-integral biases with row-wise composition.The row-wise semigroup law is preserved even though endpoint dependence relaxes the global one-parameter group law.Source: §5 · p. 8
  • Endpoint-independent edge potentials recover exact ALiBi or phase-modulated additive biases, both depending only on a phase difference and retaining the relative law and streaming/cache policy.Setting aℓ ≡ 1 gives ALiBi; setting aℓ = ωℓ with ωℓ = g(xℓ) ≥ 0 yields the phase-modulated form.Source: §5 · p. 8
  • Streaming computes each query's edge potentials in one similarity sweep, then uses a prefix sum, with O(t) per-step overhead and O(L) cached-probe memory per head.Cached rotated probes require O(1) recomputation per key; recomputing rotations on the fly reduces memory to O(d) per head.Source: §5 · p. 8

6 EXPERIMENTS

The experiments compare GRAPE with RoPE, ALiBi, and FoX on language modeling using Llama-based models trained on FineWeb-Edu. Across the reported loss curves, GRAPE maintains an edge over other mechanisms, while RoPE shows training instability in the large-model plot.

6 EXPERIMENTS

  • The evaluation compares positional encodings on language modeling, including RoPE, ALiBi, and FoX.The reported evaluation tables cover medium and large models using zero-shot lm-evaluation-harness results.Source: §6 · p. 8, Table 1 · p. 9, Table 2 · p. 10

6.1 IMPLEMENTATION DETAILS

  • Experiments use Llama-based models while changing only the positional encoding, with 50B FineWeb-Edu training tokens and a context length of 4,096.The medium model has 353M parameters and the large model 770M; FoX-Pro is not used and its KV-shift module is disabled for fair comparison.Source: §6.1 · pp. 8–9

6.2 RESULT ANALYSIS

  • GRAPE maintains a persistent loss-curve edge over RoPE and FoX, while RoPE shows training instability and GRAPE steadily improves in the large-model run.The figures report training and validation loss for medium (353M) and large (770M) models on FineWeb-Edu 100B.Source: §6.2 · p. 9, Figure 2 · p. 9, Figure 3 · p. 10

7 CONCLUSION

GRAPE unifies multiplicative rotations and additive unipotent mechanisms, recovering RoPE, ALiBi, and FoX as exact cases. The framework also includes contextual extensions and integrates with Transformer attention and streaming caches.

7 CONCLUSION

  • GRAPE unifies multiplicative SO(d) rotations and additive GL unipotent mechanisms, recovering RoPE and ALiBi exactly and establishing FoX as an exact instance.The conclusion presents the framework as an extensible design space for positional encoding in Transformer models.Source: §7 · p. 10

Appendix

  • The appendix includes a notation-and-definitions summary table.The supplied table caption identifies it as Table 3.Source: Table 3 · p. 16

A RELATED WORK

  • Prior work spans absolute and relative encodings, linear logit biases, rotary variants, forgetting mechanisms, and contextual position methods; GRAPE relates these families through group actions.GRAPE's contextual variants use cumulative token-adaptive phases while retaining headwise relativity and streaming caches.Source: A RELATED WORK · pp. 16–17

B APPLICATION IN MULTI-HEAD ATTENTION

  • In multi-head attention, GRAPE-M applies each head's orthogonal position map to queries and keys, while values remain unchanged.For one-parameter subgroups, logits depend only on the position offset; known-at-arrival key maps can be cached without rewriting.Source: B APPLICATION IN MULTI-HEAD ATTENTION · p. 17

C FORGETTING TRANSFORMER AS A SPECIAL ADDITIVE GRAPE

V. (FoX)

FoX is exactly represented by GRAPE-AP when its path edges are endpoint-independent, and constant gates recover ALiBi. GRAPE also composes additive biases with rotational encodings and offers a lower-cost rank-2 alternative to dense LieRE rotations.

V. (FoX)

  • FoX is an exact GRAPE-A/GRAPE-AP instance, realized by a rank-1 unipotent path with endpoint-independent edges.When the edges are endpoint-independent, the path-integral bias reduces to a difference of per-time potentials.Source: §V · p. 18

D NON-COMMUTING MULTIPLICATIVE GRAPE

  • The non-commuting thin compression preserves the spectrum, whose expressive power is captured by its mode angles.The ambient lifting preserves the spectrum of the compressed generator.Source: D NON-COMMUTING MULTIPLICATIVE GRAPE · p. 18

E COMPOSITION OF ADDITIVE GRAPE AND MULTIPLICATIVE GRAPE

  • GRAPE’s rotational and additive mechanisms compose as one group action while retaining exact relative scoring and efficient streaming caches.The joint lift uses a direct-sum rotational and unipotent structure; constant per-head FoX gates reduce to exact ALiBi.Source: §V · p. 18, E COMPOSITION OF ADDITIVE GRAPE AND… · p. 19

F COMPARISON WITH LIERE

  • GRAPE-M and LieRE both use skew-symmetric SO(d) generators, but GRAPE-M’s rank-2 closed form reduces per-token cost to O(d).The comparison gives LieRE’s direct exponential cost as O(d^3), with O(d^2) parameters and memory per head; GRAPE-M uses O(d) time and memory per head. Content-dependent generators require a new dense exponential in LieRE, whereas GRAPE-M supports fast closed-form evaluation.Source: F COMPARISON WITH LIERE · p. 19

G 2D AND 3D GRAPE FOR VISION AND MULTIMODAL POSITION ENCODING

GRAPE extends positional encoding to multidimensional coordinates by selecting group actions on coordinates. Its appendices give implementation pseudocode, rank-2 exponential differentiation and application details, and spectral-analysis setup.

G 2D AND 3D GRAPE FOR VISION AND MULTIMODAL POSITION ENCODING

  • GRAPE extends to 2D and 3D coordinates, with unipotent additive updates preserving the exact relative composition law.For vision and multimodal settings, the scalar offset can be a function of coordinate differences, allowing axis-aligned or radial recency biases.Source: G 2D AND 3D GRAPE FOR VISION AND… · p. 20

H ALGORITHMIC DETAILS AND PSEUDO CODE

  • The commuting multi-subspace pseudocode projects queries and keys, applies plane-wise rotations, then projects back.The algorithm loops over heads, positions, and coordinate pairs.Source: H ALGORITHMIC DETAILS AND PSEUDO CODE · p. 20

I DIFFERENTIATION AND FAST APPLICATION OF RANK-2 MATRIX EXPONENTIAL

  • Rank-2 exponential evaluation uses a few inner products for O(d) application, and the derivative formulas support mixed-precision backpropagation with small-s guards.The appendix derives the rank-2 update from the generator’s minimal polynomial.Source: I DIFFERENTIATION AND FAST APPLICATION… · p. 21

J SPECTRAL ANALYSIS OF GRAPE AND OTHER METHODS

J.1 RANK-2 PLANE: EXACT SPECTRUM AND GEOMETRIC INTERPRETATION

A rank-2 skew generator acts as a rotation on the plane spanned by its defining vectors, with its spectrum and phase determined by their geometry. Orthogonal commuting planes recover RoPE under the canonical coordinate-pair and log-uniform-spectrum choice.

J.1 RANK-2 PLANE: EXACT SPECTRUM AND GEOMETRIC INTERPRETATION

  • The rank-2 generator has eigenvalues {±is} and d−2 zeros, where s=∥a∥∥b∥sin ϕ; its exponential is orthogonal with spectral radius 1.The per-step angle is θ=ηs, bounded by η∥a∥∥b∥, with equality when a and b are orthogonal.Source: §J.1 · p. 21

J.2 MULTI-SUBSPACE GRAPE-M AND ROPE

  • Commuting orthogonal planes recover RoPE when they are canonical coordinate pairs with the canonical log-uniform spectrum.Source: §J.2 · p. 21

J.3 ADDITIVE GRAPE

Additive GRAPE uses square-zero nilpotent updates, making its transformations unipotent even though they need not preserve norms. Paired inverse-transpose scoring cancels their multiplicative distortion, leaving an exact relative additive bias.

J.3 ADDITIVE GRAPE

  • For A²=0, the additive lift is I+sA, with all eigenvalues equal to 1, determinant 1, and inverse I−sA.Its Jordan form has size-2 blocks for the 1-eigenspace, with the number of nontrivial blocks equal to rank(A).Source: §J.3 · p. 22
  • In the canonical rank-1 case, all but two singular values equal 1; general index-2 updates have dimension-free bounds 1−|s|∥A∥₂ and 1+|s|∥A∥₂.The latter bound is stated for the minimum and maximum singular values, respectively.Source: §J.3 · pp. 22–23
  • Paired inverse-transpose scoring cancels multiplicative anisotropy, preserving the exact relative law and leaving a pure additive term.This holds independently of the unipotent transformation’s singular values.Source: §J.3 · p. 23
  • GRAPE-AP’s path product inherits the unipotent spectrum, with row-dependent s≤0; paired scoring leaves its bounded additive bias.The two nontrivial singular values follow the rank-1 formula, while the rest equal 1.Source: §J.3 · p. 23
  • Across these lifts, eigenvalues do not grow with offset length, while the operator norm grows at most linearly in |s|.The determinant remains 1, and any expansion along one direction is balanced by contraction along its paired direction.Source: §J.3 · p. 23

J.4 COMPARISON TO PATH ATTENTION

PaTH Attention [Yang et al., 2025b] uses contextual products of identity-plus-rank-one factors, which are generally nonorthogonal. Repeated factors can contract energy and attenuate long-range logits, unlike norm-preserving GRAPE-M maps.

J.4 COMPARISON TO PATH ATTENTION

  • PaTH Attention [Yang et al., 2025b] applies contextual identity-plus-rank-one factors along the key-query path; unlike GRAPE-M factors, they are not generally orthogonal.The comparison concerns the cumulative product applied between key position j and query position i.Source: §J.4 · p. 23
  • Each H_t has eigenvalues 1 − β_t along w_t and 1 in the other directions, so it is norm-preserving only when β_t ∈ {0, 2}.For β_t ∈ (0, 2) \ {0, 2}, the component along w_t contracts; the operator norm remains 1.Source: §J.4 · p. 24
  • The smallest singular value of the path product can approach zero, especially when any β_s is near 1, and is exactly zero when β_s = 1.The path product’s smallest singular value has a worst-case exponential lower bound governed by the product of |1 − β_s|.Source: §J.4 · p. 24
  • With a time-invariant direction, PaTH factors commute and the eigenvalue along w is the product of (1 − β_s), producing exponential contraction except at β_s ∈ {0, 2}.In this aligned-plane case, P_t is an idempotent projector.Source: §J.4 · p. 24
  • Persistent β_t outside {0, 2} yields cumulative energy loss along a moving one-dimensional subspace, potentially flattening or attenuating long-range logits; GRAPE-M preserves norms.GRAPE-M maps have singular values of 1, preserving volumes and norms.Source: §J.4 · p. 24
Loading 2512.07805v6…