PaperlayerView exact revision on arXiv

Source-linked AI summary

Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

Eric Enouen, Sainyam Galhotra

arXiv:2609.38530v1cs.CLcs.LG

TL;DR

The paper asks how positional-encoding choices shape in-context retrieval, and studies this through mechanistic analysis across models and controlled pre-training ablations. It finds a shift from positional to semantic retrieval with task-specific long-context strengths and weaknesses, rather than a uniform improvement.

  • Problem

    How positional-encoding choices shape in-context retrieval remains unclear, despite reported long-context gains from hybrid architectures.

    Source: §1 · p. 1

  • Method

    The paper measures retrieval-mechanism allocation across models and controlled pre-training ablations, then relates those mechanisms to long-context retrieval behavior.

    Source: §1 · p. 1, §3 · p. 3, §5 · p. 7

  • Results

    RoPE models rely primarily on positional retrieval while PE hybrids shift toward semantic retrieval, with SWA NoPE improving multiple-target retrieval and QA but struggling to distinguish competing keys.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The results suggest that long-context performance differences reflect retrieval-strategy trade-offs, and mechanistic analysis can help developers shape models for desired downstream behavior.

    Source: §6 · p. 9

    Training models from scratch is costly, limiting the study to small-scale checkpoints and preventing larger-scale or exhaustive ablations and analysis of mechanism development during training.

    Source: LIMITATIONS AND FUTURE WORK · p. 9

Abstract

from arXiv · show

Language models increasingly use architectures that vary attention span and positional encoding across layers, such as applying RoPE with sliding-window attention and NoPE with global attention (SWA NoPE). However, how these choices shape in-context retrieval remains unclear. To study this question, we take a mechanistic view, tracing how positional encoding (PE) choice shapes the internal mechanisms models use for in-context retrieval. Across 22 open-weight models spanning eight families, we find that standard RoPE models rely primarily on positional retrieval, while PE hybrids shift toward semantic retrieval. We further show on a controlled pre-training ablation that confining positional encoding to local layers produces this semantic shift, degrading representations of positional information. Finally, we show that the reported long-context gains of PE hybrids mask a retrieval trade-off: SWA NoPE improves over RoPE on multiple-target retrieval and QA, but degrades when distinguishing competing keys. We show that these behavioral differences better track the mechanism shift from positional toward semantic mechanisms than a uniform improvement in long-context retrieval.

1 INTRODUCTION

The paper asks how positional-encoding choices shape retrieval mechanisms and long-context behavior. It finds that SWA NoPE shifts retrieval toward semantic mechanisms, with task-specific gains and weaknesses rather than a uniform improvement.

1 INTRODUCTION

  • The paper addresses the unclear explanation for how architectural PE choices shape long-context retrieval, including hybrids that vary attention span and positional encoding across layers.SWA NoPE combines local sliding-window RoPE layers with global NoPE layers.Source: §1 · p. 1
  • The authors use mechanistic analysis to test whether PE choice changes retrieval mechanisms and whether those differences explain long-context performance.They frame this analysis as an intermediate lens between architecture and retrieval behavior.Source: §1 · p. 1
  • Across 22 open-weight models from eight families, RoPE models rely primarily on positional retrieval, while PE hybrids rely primarily on semantic mechanisms.Controlled pre-training ablations show that SWA alone does not produce the shift; removing RoPE from global layers does, alongside degraded positional-information representations when PE is confined to local layers.Source: §1 · p. 1
  • SWA NoPE improves multiple-target NIAH and QA retrieval but struggles to distinguish semantically similar keys, indicating a retrieval trade-off rather than uniform gains.The authors link these different long-context profiles to a shift from positional toward semantic retrieval.Source: §1 · pp. 1–2, Figure 1 · p. 2

2 RELATED WORK

Prior work explores extending or removing positional encodings and reports gains from hybrid SWA NoPE architectures. This paper connects retrieval mechanisms to those architectural choices and finds that the reported gains are not uniform across retrieval settings.

2 RELATED WORK

3 MECHANISTIC LENS

The paper uses counterfactual activation patching to identify which retrieval mechanism drives a prediction, then estimates mechanism allocation across samples. It also compares positional retrieval with semantic retrieval by grouping lexical and reflexive mechanisms together.

3 MECHANISTIC LENS

  • Counterfactual patching uses paired original and counterfactual prompts and residual-stream activation patching to identify the mechanism dominating a prediction.The paper illustrates the approach with a Boxes task in which objects are assigned to boxes and the model is asked where an object is.Source: §3 · p. 3
  • Patched answers distinguish positional retrieval by location, lexical retrieval by the queried item's key, and reflexive retrieval by a self-referential pointer.In the example, these mechanisms yield B, C, and A, respectively.Source: §3 · p. 3
  • Patching outcomes also include no effect when the original prediction remains and unknown when the prediction matches none of the three mechanisms or the original answer.The study uses 20 entities, which allows additional predictions to fall into the unknown category.Source: §3 · p. 3
  • Mechanism allocation is estimated from outcome rates across synthetic samples, and the dominant mechanism is the most frequent among positional, lexical, and reflexive outcomes.The positional ratio compares positional rate P with the combined lexical and reflexive rates L+R; a higher ratio indicates more positional allocation.Source: §3 · p. 3

4 POSITIONAL ENCODING CHOICE SHAPES MECHANISM ALLOCATION

Across model families, PE hybrids shift in-context retrieval from positional toward semantic mechanisms. Controlled ablations link this shift to removing RoPE from global layers and weaker representations of ordering IDs when positional encoding is confined to local layers.

4.1 MODEL BATTERY

  • Every tested RoPE model is positionally dominant, while every PE hybrid is lexically or reflexively dominant across the 22-model battery.The models span eight families; Figure 3 shows the positional-ratio ranges separating the architecture classes.Source: §4.1 · pp. 4–5, Figure 3 · p. 5

4.2 ABLATING POSITIONAL ENCODING CHOICE

  • At both training lengths, SWA NoPE shifts allocation toward semantic retrieval, while SWA RoPE alone does not.For 16K, rpos falls from 0.30 to 0.14 under SWA NoPE but rises to 0.40 under SWA RoPE; for 32K, it falls from 0.26 to 0.13 under SWA NoPE and changes to 0.27 under SWA RoPE.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 CONFINING POSITIONAL ENCODING DEGRADES ORDERING ID CREATION

  • SWA NoPE lowers ordering-ID probe accuracy by 12.1 points at 16K and 11.0 at 32K versus RoPE, while semantic-key accuracy remains nearly unchanged.SWA RoPE’s ordering-ID accuracy decreases by 3.2 and 2.2 points, respectively; the ordering-ID versus semantic-key accuracy gap grows from roughly 4 points under RoPE to 16 under SWA NoPE.Source: §4.3 · p. 7, Table 3 · p. 7
  • The proposed window-size prediction—that smaller windows relative to context make ordering-ID formation harder and strengthen the mechanism shift—remains untested.The authors leave controlled checkpoints varying window size for future work.Source: §4.3 · p. 7

5 MECHANISM ALLOCATION SHAPES LONG-CONTEXT RETRIEVAL

The section examines whether mechanism differences predict long-context retrieval behavior across RULER subtasks and a competing-key task. SWA NoPE’s aggregate gains coexist with task-specific failures, consistent with a shift in retrieval strategy rather than uniform improvement.

5 MECHANISM ALLOCATION SHAPES LONG-CONTEXT RETRIEVAL

  • SWA NoPE improves overall RULER performance over RoPE and SWA RoPE, with gains concentrated in most NIAH and QA tasks rather than all subtasks.It does not improve over RoPE on the first NIAH multi-key task and does not consistently improve variable-tracking and aggregation tasks.Source: §5 · p. 8, Table 4 · p. 8
  • SWA NoPE performs particularly well on multi-value and multi-query NIAH, where all requested values are returned rather than selecting one key and rejecting distractors.The section uses a task with four needles—one queried key and three distractors—and varies shared words across keys from zero to three.Source: §5 · pp. 8–9
  • On competing-key retrieval, RoPE outperforms SWA NoPE even at 0/4 shared words; at 16K, their gap grows from 8.8 points at 0/4 to 27.2 at 2/4.The gap narrows at 3/4 as performance degrades across architectures; the model ordering reverses relative to multi-value and multi-query retrieval.Source: Table 5 · p. 8, §5 · p. 9

6 CONCLUSION

The conclusion links positional-encoding choice to retrieval mechanisms and to distinct long-context retrieval profiles, rather than a uniform change in retrieval ability. It presents mechanistic analysis as a potential aid to intentional architectural design.

6 CONCLUSION

  • Across controlled checkpoints and open-weight models, PE choice shifts reliance between positional and semantic retrieval, with degraded ordering-ID representations linked to confining positional encoding to local layers.The resulting long-context profiles differ rather than showing uniform improvement or degradation.Source: §6 · p. 9
  • Mechanistic analysis can help developers understand how architectural choices shape learned mechanisms and design for desired downstream behavior.The conclusion presents interpretability as a tool for intentional design, beyond explaining existing models.Source: §6 · p. 9
  • SWA NoPE keeps positional distance fixed as global context grows, but confining RoPE this way substantially changes learned retrieval mechanisms.The authors leave exploration of the benefits and drawbacks of these choices to future work.Source: §6 · p. 9

LIMITATIONS AND FUTURE WORK

The paper identifies limits in scaling its training-from-scratch ablations, in understanding how training data shapes mechanism allocation, and in studying linear-hybrid architectures.

Scaling

  • Training-from-scratch cost prevents larger-model tests, exhaustive design ablations, and analysis of how mechanisms develop during training.The authors leave these directions for future work.Source: LIMITATIONS AND FUTURE WORK · p. 9

Data

  • Mechanism allocation depends on which retrieval mechanisms the training objective reinforces and therefore on the training-data distribution.The authors propose studying how different distributions shape learned mechanisms and whether long-context training data can target particular mechanisms.Source: LIMITATIONS AND FUTURE WORK · p. 9

Linear hybrids

  • The paper's linear-hybrid analysis is preliminary, leaving a comprehensive study of their retrieval mechanisms for future work.The authors raise whether alternative efficient-attention strategies shift the same mechanisms or produce fundamentally different ones.Source: LIMITATIONS AND FUTURE WORK · p. 9

AI STATEMENT

The paper discloses its use of generative AI tools and outlines its experimental framework, mechanistic analysis, and model battery.

AI STATEMENT

  • Generative AI tools assisted with code implementation, methodology feedback, figures, and writing; the authors reviewed the work and accept responsibility for its final content.The authors state that AI was not used to generate datasets or develop theoretical models or mathematical claims.Source: AI STATEMENT · p. 10

A EXPERIMENTAL DETAILS

  • Experiments used bf16 on an RTX 6000 Ada, except models above 25B parameters, which ran on a B200.The RTX 6000 Ada has 48GB of memory.Source: A EXPERIMENTAL DETAILS · p. 14

A.1 MECHANISTIC LENS

  • The mechanistic framework uses paired original and counterfactual prompts, patches counterfactual activations at a selected layer, and counts which mechanism predicts the original prompt's token.The analysis uses three mechanisms across 29 dataset/query pairs; each model's patched token positions are selected by token role, with per-model choices reported in Table 6.Source: §A.1 · p. 14, Table 6 · p. 14

A.2 MODEL BATTERY

  • The model battery comprises 22 posttrained models across RoPE and PE-hybrid families, evaluated across the task battery with 20 entities and 1,000 samples per task.The models' patched positions are matched by role, and layer selection is conducted for each model.Source: §A.2 · p. 14

A.3 PRE-TRAINING ABLATION

The experimental details cover a controlled pre-training ablation, linear probes of positional and semantic information, and long-context retrieval tests. Layer scans validate the patching layers used for the main analysis.

A.3 PRE-TRAINING ABLATION

  • The pre-training ablation compares RoPE, SWA RoPE, and SWA NoPE 665M-parameter checkpoints on Boxes, using layer 15 for patching.The SWA variants use a 128-token sliding window and a 1:1 local-to-global layer ratio; SWA NoPE removes RoPE only from global layers. Layer scans identify layer 15 as the last layer before retrieval for each checkpoint.Source: §A.3 · p. 15, Figure 4 · p. 15

A.4 LINEAR PROBING

  • Linear probes test whether layer-15 residuals encode entity order or semantic keys, using randomized object-position assignments to separate the labels.The ordering probe has 20 classes (5% chance); the semantic-key control has 82 classes (about 1.2% chance).Source: §A.4 · p. 16

A.5 LONG-CONTEXT BEHAVIOR

  • Long-context evaluation tests RULER and a confusable-key task that varies how many of four competing keys share words.The confusable-key task uses four-word keys, 500 samples per similarity level and checkpoint, and 16K or 32K context lengths; its 0/4 condition controls for the changed key format.Source: §A.5 · p. 16

B LAYER SELECTION VALIDATION

  • Layer scans select the last layer before patching transfers an already-retrieved answer, yielding ℓ=23 for Gemma-3 4B, ℓ=29 for Gemma-4 4B, and ℓ=24 for Gemma-4 2B.The Gemma-4 2B choice is conservative because its scan is less clean; the three figures plot mechanism shares across layers for messiness-0 and messiness-4 counterfactuals.Source: B LAYER SELECTION VALIDATION · p. 17, Figure 5 · p. 17, Figure 6 · p. 17

C MECHANISM ALLOCATION EXTENDED RESULTS

The extended results report mechanism distributions and show that the RoPE–hybrid ordering in positional retrieval emerges in the aggregate, while target position shifts the balance between lexical and reflexive retrieval.

C.1 FULL DISTRIBUTION TABLE

  • Table 7 reports each model's positional, lexical, reflexive, unknown, and no-effect shares, along with rpos = P/(L+R).The no-effect rate is generally low but substantially higher for some models, particularly Gemma-4 12B, even when patching multiple tokens.Source: C MECHANISM ALLOCATION EXTENDED RESULTS · p. 18, Table 7 · p. 18

C MECHANISM ALLOCATION EXTENDED RESULTS

C.2 MECHANISM ALLOCATION BY TARGET ENTITY POSITION

  • Target position shifts the lexical–reflexive balance: reflexive retrieval dominates when the target precedes the query, while lexical retrieval dominates when it follows.The breakdown is reported to hold consistently across query types.Source: §C.2 · p. 20, Figure 9 · p. 20, Figure 10 · p. 20

D RULER EXTENDED RESULTS

Per-task RULER results show SWA NoPE’s gains are concentrated rather than uniform, and its 32K confusable-key errors usually retrieve a competing key.

D RULER EXTENDED RESULTS

E LINEAR HYBRIDS

The three tested linear hybrids do not show the PE-hybrid mechanism allocation pattern: Qwen3.5 falls within the RoPE range, while RecurrentGemma and Granite lie near the RoPE/PE boundary.

E LINEAR HYBRIDS

  • Linear attention hybrids are often motivated by computational long-context gains, but their effects on in-context retrieval mechanisms are an open question here.Source: E LINEAR HYBRIDS · p. 22
  • The tested hybrids interleave Gated DeltaNet with full p-RoPE attention in Qwen3.5 4B, gated linear recurrence with local p-RoPE in RecurrentGemma 2B, and Mamba2 with full NoPE in Granite-4.0-H 1B.Their interleaving ratios are 3:1, 2:1, and 9:1, respectively.Source: E LINEAR HYBRIDS · p. 22
  • Qwen3.5 falls within the RoPE range (rpos = 0.72), while RecurrentGemma (0.59) and Granite (0.60) lie near the boundary between Llama-3.1 8B (0.59) and Command R7B (0.53).Llama-3.1 8B is the lowest RoPE model and Command R7B the highest PE hybrid.Source: E LINEAR HYBRIDS · p. 22
  • RecurrentGemma and Granite have under 10% reflexive retrieval among legible predictions, and nearly half of their predictions fall outside the three-mechanism taxonomy.The authors leave the explanation for future work.Source: E LINEAR HYBRIDS · p. 22
Loading 2609.38530v1…