PaperlayerView exact revision on arXiv

Source-linked AI summary

How Local Mixing Encodes Relative Position in Global NoPE Attention

Cutter Dawes, Nick Alonso, Tom Figliolia, Beren Millidge

arXiv:2609.38109v1cs.CLcs.AIcs.LG

TL;DR

Hybrid architectures use local mixing layers alongside global NoPE attention, but how they represent position in those global layers has been unclear. This paper combines theoretical and empirical analyses to explain how local layers induce a recency bias that reaches global attention logits. The authors report that this bias can persist across sequence lengths in hybrid models.

  • Problem

    How hybrid models encode relative position in global NoPE layers remains unclear, despite evidence that such models do not encode absolute position; explicit encodings also add computational overhead and make length extension difficult.

    Source: §I · p. 1

  • Method

    The paper develops theoretical and empirical analyses of how SWA or gated linear layers create residual-stream recency bias that propagates through intervening layers and is selected by global NoPE projections.

    Source: §I · p. 1

  • Results

    Hybrid architectures develop relative position encodings in global NoPE layers, with recency bias in residual states and logits present at initialization and strengthening during training.

    Source: §I · p. 1

  • Takeaways & Limitations

    The proposed recency bias is invariant to absolute sequence length and may allow extrapolation beyond training distances without serious discontinuities.

    Source: §VII · p. 9

Abstract

from arXiv · show

The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings have been developed in transformer-based models, such as rotary position encoding (RoPE). Although explicit position encodings have long been assumed to be required, recent methods that interleave local mixing layers, such as sliding window attention (SWA) and gated linear attention, while not encoding position (NoPE) in global attention layers has recently been shown to be successful at scale. How and why this approach works is not well-understood. In this paper, we develop an explanation of how hybrid models of this sort can implicitly encode position at global NoPE layers. Supported by both theoretical and empirical evidence, our central argument is that SWA and gated linear attention induce a recency bias in the residual stream that propagates to, and is selected by, the global attention logits. Moreover, in contrast to the implicit position encodings found in models with only global NoPE attention, in which positional information arises solely from the causal mask, the recency bias in hybrid models can be maintained across long sequences. In addition to deepening our understanding of how hybrid models encode position, these findings may provide insights for how to encode position in a way that can extrapolate to longer sequence lengths indefinitely.

1 Zyphra Research

The paper asks whether hybrid architectures encode relative position in global NoPE layers, beyond prior evidence that they lack absolute-position encoding. It frames recency bias as a common way relative position encodings affect attention logits, then proposes that local mixing supplies this signal.

I. INTRODUCTION

  • Explicit position encodings such as RoPE add computational overhead and make length extension difficult, while NoPE has mixed performance and may extrapolate better beyond training lengths.The paper describes relative encodings as commonly creating recency bias by lowering logits as query-key distance increases.Source: §I · p. 1, §II · p. 2
  • Prior work found that hybrid NoPE models did not encode absolute position, leaving open whether they encode relative position.Earlier global-NoPE work showed recency bias only on sequences of approximately 30 tokens, while the conditions for attention to use residual-stream bias remained unexplained.Source: §I · p. 1, §II · p. 3
  • Hybrid models can develop relative position encodings in global NoPE layers through interactions between local layers and subsequent NoPE attention.The proposed mechanism is that local mixing induces a recency bias in the residual stream, which propagates to and is selected by global attention logits.Source: §I · p. 1
  • Empirical evidence shows recency bias in residual states and NoPE logits across depth, present at initialization and strengthening during training.The evidence covers hybrid models using SWA with RoPE or NoPE and KDA as the local layer.Source: §I · p. 1

II. BACKGROUND: RECENCY BIAS AS RELATIVE POSITION ENCODING

  • Relative position encodings include additive logit biases such as ALiBi and Fire, and multiplicative methods such as RoPE that alter query-key interactions.The cited methods commonly induce recency bias, with farther keys receiving lower logits.Source: §II · p. 2

III. THESIS: HYBRID ARCHITECTURES IMPLICITLY ENCODE RELATIVE POSITION AT GLOBAL NOPE LAYERS

The paper's thesis is that local layers create a recency bias in residual states that persists across sequence lengths and can be read out by global NoPE attention. Its theoretical account tracks that signal through intervening layers and into attention logits, while noting that the analysis does not capture all model computation.

III. THESIS: HYBRID ARCHITECTURES IMPLICITLY ENCODE RELATIVE POSITION AT GLOBAL NOPE LAYERS

  • Local layers such as SWA create a recency bias in residual states that persists across sequence lengths and can be transferred to global NoPE logits.The thesis states that trained global NoPE attention can select this bias, creating an implicit relative position encoding.Source: §III · p. 3
  • The theoretical analysis isolates SWA's structural effect but does not fully account for content-dependent attention or MLP computation.Empirical analysis complements the theory by checking residual-stream and logit recency biases across models, training stages, and data types, including KDA hybrids.Source: §III · p. 3

A. Theory

  • Lagged cross-moments provide matrix-valued similarity profiles whose slices include cosine similarity and alignment with the composed query-key projection.The NoPE logits capture the slice aligned with the query-key projection; the recency gap is the near-distance similarity minus the far-distance reference, whose value is the similarity floor.Source: A. Theory · p. 3

B. Empirics

The theory treats SWA as a local, data-dependent mixer that creates a distance-dependent output profile, while the experiments examine hybrid models across window sizes and scales.

B. Empirics

  • Across 120M and 350M models, the experiments vary SWA window size against a global NoPE baseline, using 350M models for main-text results.Unless otherwise noted, SWA uses RoPE and is interleaved with global NoPE at a 1:1 ratio.Source: B. Empirics · p. 3

A. Theory

  • SWA acts like a data-dependent convolution, with its fixed window bounding the scale of the recency bias.After training, the mean attention profile can amplify or dampen the effect within that window, depending on its alignment with the relevant projections.Source: A. Theory · p. 4
  • With roughly uniform attention over uncorrelated projected values, SWA approximates a moving average whose output cosine decreases with distance as windows overlap less.For a window of 64, outputs at lags one and two share 63 and 62 projected values, respectively; similarity approaches zero once windows no longer overlap.Source: A. Theory · p. 4
  • At 1,000 tokens, global NoPE outputs at fixed nearby lags share nearly all their averaged terms, unlike SWA’s persistent window-scale recency profile.The example compares outputs ten positions apart, which share 99% of their terms under uniform global attention.Source: A. Theory · p. 4
  • At initialization, SWA’s recency bias weakens as its window grows, while global attention’s positional signal dilutes as sequence length grows.SWA’s locality scale is independent of sequence length; global attention’s is not.Source: A. Theory · p. 4
  • For trained SWA, similarity structure mixes incoming lagged structure through the mean attention autocorrelation and channel projections; the window bounds the scale, while the attention profile can amplify or dampen the bias.The effect on each similarity metric depends on alignment with the relevant projections.Source: A. Theory · p. 4

B. Empirics

The empirical results show that smaller SWA windows strengthen residual-stream recency bias and correspond to lower validation loss, while the bias also strengthens during training and depth.

B. Empirics

  • At initialization, SWA retains a strong distance-dependent profile within its window as sequence length increases, while global NoPE becomes too correlated to resolve most distances.The comparison probes query positions along an 8k-token sequence.Source: Figure 2 · p. 5, B. Empirics · p. 5
  • Residual-stream recency bias strengthens during training and with depth, stabilizes early in training, then attenuates from its peak while remaining above initialization.Profiles are shown at the first, middle, and last SWA layers.Source: B. Empirics · p. 5, Figure 3 · p. 5
  • With or without RoPE in SWA, smaller windows produce stronger recency biases across depth, with a more pronounced window-size effect in NoPE-in-SWA.The comparison uses distance profiles and layer-wise recency gaps.Source: B. Empirics · p. 5, Figure 4 · p. 6
  • Smaller windows achieve lower late-training validation loss with or without RoPE, and their advantage is stronger than shown because compute was not matched across window sizes.The authors describe this as preliminary evidence that recency-bias strength matters to hybrid-model success.Source: B. Empirics · pp. 5–6, Figure 5 · p. 6

V. STEP 2: PRESERVATION BY INTERVENING TRANSFORMATIONS

Residual addition carries branch similarity into the residual stream, where it interacts with the existing input structure. Normalization and MLPs can change the similarity gap and floor, but a wide MLP preserves a nonnegative recency-bias ordering at random initialization.

Residual addition

  • Residual addition carries the branch similarity structure into the residual stream, where it interacts with the input’s existing similarity structure.For uncorrelated white inputs, the cross terms vanish and the output cross-moment is GZ(d) = adAΣ + ga(d)AΣA⊤.Source: §V · p. 6

Effect of normalization

  • Normalization has little effect on similarity structure itself, while its learned gain can alter the similarity floor and gap.RMSNorm exactly preserves unit-normalized cross-moment structure; the gain’s effect depends on how it reweights shared-mean and fluctuation contributions.Source: §V · pp. 6–7

Effect of the MLP

  • At random initialization, a wide MLP preserves the ordering of a nonnegative recency-bias profile, though it may change its floor, magnitude, and shape.The kernel induced by the nonlinearity is nondecreasing for nonnegative cosine similarity.Source: §V · p. 7
  • The initialization result does not guarantee that trained MLPs preserve the incoming similarity structure.Training may lead the MLP to suppress, amplify, or reorient it.Source: §V · p. 7

B. Empirics

Module-level measurements broadly match the theory, while cumulative block changes are smaller. SWA gives the strongest cosine-gap increase, and full-attention and MLP blocks largely preserve the incoming gap overall.

Module changes

  • SWA provides the strongest cosine-gap boost and also reduces the cosine floor; global NoPE attention increases the gap less.Figure 6 reports after-minus-before changes averaged across module appearances at w = 128.Source: Figure 6 · p. 7, B. Empirics · p. 8
  • Normalization, including its learned affine transformation, reduces the cosine floor and increases the gap.The plotted changes compare module outputs with their inputs; residual-addition steps instead compare with the isolated branch.Source: Figure 6 · p. 7, B. Empirics · p. 8
  • Residual addition and MLPs each reduce the cosine gap and increase the floor, while complete full-attention and MLP blocks largely preserve the incoming gap.The cumulative block effect is much smaller than the individual module effects.Source: Figure 6 · p. 7, B. Empirics · p. 8

VI. STEP 3: GLOBAL NOPE INHERITS THE RESIDUAL STREAM RECENCY BIAS

Global NoPE logits can inherit residual-stream recency bias when query and key projections align with the input’s lagged cross-moment differences. The alignment determines the sign and magnitude of the logit recency gap.

Readout by global NoPE

  • The logit recency gap is positive precisely when the query-key projection matrix aligns positively with the input cross-moment difference between near and far distances.Its magnitude depends on the extent of that alignment.Source: A. Theory · p. 8
  • Unlike cosine similarity, the logit similarity profile can depend on the skew component of the full cross-moment structure.Depending on slice orientation, projections can preserve, distort, or reverse the residual-similarity ordering.Source: A. Theory · p. 8
  • Independent zero-mean query and key initializations give an expected projection matrix of zero, so the recency bias need not appear in logits at initialization.Training can align the projections, allowing the logits to inherit the residual-stream bias.Source: A. Theory · p. 8

B. Empirics

Across training, global NoPE logits develop a recency-biased profile, consistent with learning to select residual-stream bias. The discussion connects this mechanism to length extension while noting that its quantitative relationship to generalization remains unresolved.

B. Empirics

  • As training progresses, global NoPE logits develop a recency-biased profile across depth, strengthening and stabilizing early in training.At initialization, distance dependence is weak; the logit bias stabilizes sooner across layers than the residual-stream bias does.Source: B. Empirics · p. 8
  • For RoPE-in-SWA, logit gaps vary little by window size, whereas NoPE-in-SWA gaps at larger windows swing strongly positive and negative.Across depth, both variants’ logit gaps are fairly stable after initially strengthening.Source: B. Empirics · p. 8, Figure 8 · p. 9
  • When the window is too large and residual-stream recency bias too weak, NoPE-in-SWA fails to develop logit recency bias and performance collapses.The proposed mechanism assigns local layers the role of producing a residual cue for global NoPE logits to select and maintain.Source: B. Empirics · pp. 8–9

VII. DISCUSSION

  • The proposed mechanism is that local or linear layers induce residual-stream recency bias, which intervening layers preserve and global projections select into attention logits.The discussion frames this bias as an implicit relative position encoding with effects similar to explicit relative encodings.Source: §VII · p. 9
  • The discussion proposes that a finite local window supplies a positional scale that does not dilute with sequence length and is invariant to absolute sequence length.It also argues that the learned encoding does not change radically beyond training distances, avoiding serious extrapolation discontinuities.Source: §VII · p. 9
  • The paper leaves open a quantitative account of bias propagation and attenuation across depth and sequence lengths, and of how these relate to length generalizability.These relationships are identified as future work.Source: §VII · p. 9

Appendix

The appendix specifies how similarity and logit profiles are estimated in the SWA experiments, then derives the output cross-moment under a fixed mean-attention profile. For uncorrelated white inputs, the output cross-moment follows the triangular overlap profile, with uniform attention as a special case.

A. Similarity measures and estimation

B. Models, training, and evaluation inputs

  • The SWA study compares 120M and 350M models across windows of 64–4096, with sequence length 8192 and training on 12B and 24B tokens, respectively.The main-text figures use 350M models and TextbookChapters unless captions specify otherwise; KDA controls interleave KDA and global NoPE layers at both scales.Source: B. Models, training, and evaluation… · p. 12
  • Recency measurements use 48 sequences of length 8192, with queries at positions 4352–8191 and lags 1–4096; validation-loss plots use DCLM sequences of length 4096.Random-token controls sample uniformly from the padded vocabulary with seed 0; validation-loss trajectories cover the final 30% of training.Source: B. Models, training, and evaluation… · p. 12

APPENDIX B SUPPLEMENT TO STEP 1: SWA PRODUCES A RECENCY BIAS IN THE RESIDUAL STREAM

  • The appendix supplies additional theoretical and empirical results for 350M models trained on TextbookChapters to corroborate the main SWA recency-bias analysis.The matrix plots use averaged outer products because they capture shared structure, although sampling noise can contribute.Source: A. Similarity measures and estimation · p. 12, Appendix B · p. 13

A. Theory

  • For uncorrelated white inputs, the output cross-moment is proportional to the triangular overlap profile; uniform attention gives the text’s triangular-profile special case.The derivation assumes concentrated vector norms and moments that are approximately stationary away from the causal boundary; learned attention is represented by a fixed mean profile.Source: A. Theory · p. 13

B. Empirics

The appendix reports that SWA-induced residual recency bias increases smoothly through depth and is consistent across model scales. Global NoPE logits show a different pattern: their recency bias stabilizes across layers and varies less with window size.

B. Empirics

  • Residual-stream recency bias increases smoothly through SWA layers, with a similar ordering and magnitude at 120M and 350M scales.Figure 9 shows the layerwise profile at the final 350M checkpoint; its 120M-scale companion supports the cross-scale comparison.Source: B. Empirics · p. 13, Figure 9 · p. 13
  • Changing SWA window size predictably changes both residual cosine profiles and language-modeling behavior, but the window comparison does not isolate causality.Figure 9b shows cosine profiles at first, middle, and last SWA layers across window sizes; the validation-loss curves are in Figure 5.Source: B. Empirics · p. 13, Figure 9 · p. 13

A. Theory

  • The cross-moment analysis separates magnitude from alignment with the isotropic component or composed query-key projection to interpret cosine and logit similarities.Figure 10 plots cross-moment magnitude, alignment at SWA layers, and alignment at model-scale post-LN1 inputs to global heads.Source: A. Theory · p. 14, Figure 10 · p. 14
  • Global NoPE logit recency bias rises initially, then remains largely consistent across layers and varies less with window size than residual-stream bias.The relative stability suggests the model develops a consistent logit-based recency bias to serve as its attention position encoding.Source: A. Theory · p. 14, Figure 11 · p. 15

APPENDIX D CONTROLS

The appendix finds that the main SWA recency-bias patterns hold at 120M scale and under random-token controls, though transfer to logits is less consistent outside natural language. It also derives an exponential recency profile for simplified KDA mixing, with a scale that remains resolved as sequence length grows.

A. 120M-scale robustness

  • At 120M scale, SWA recency bias is present at initialization, strengthens with training and depth, sharpens with smaller windows, and propagates to global NoPE logits.Figure 12 also compares RoPE- and NoPE-in-SWA behavior and validation loss.Source: A. 120M-scale robustness · p. 15, Figure 12 · p. 16
  • With smaller windows, validation loss decreases despite fewer training flops, while NoPE-in-SWA loss collapses for larger windows.The authors suggest that larger-window NoPE-in-SWA models may lack a sufficiently strong mechanism to resolve recency bias and encode position.Source: A. 120M-scale robustness · p. 15, Figure 12 · p. 16

B. Random-token controls

  • Random-token controls reproduce initialization and depthwise recency trends, but RoPE window-size trends are inconclusive and transfer to logits is less consistent than for natural language.For NoPE-in-SWA, smaller windows strengthen recency bias at both model scales; overall, the results are consistent with a mechanism untuned for data outside its training distribution.Source: B. Random-token controls · p. 15, Figure 13 · p. 17, Figure 14 · p. 18

A. Theory

  • In simplified KDA mixing, shared values yield an exponential residual recency profile, with smaller retention factors producing narrower, steeper profiles.The approximation assumes zero-mean, similar-magnitude, uncorrelated projected values and a fixed per-step decay factor; the profile scale is independent of sequence length.Source: A. Theory · p. 21
  • For correlated inputs, KDA’s smoothly decaying overlap profile mixes incoming structure across distances, which later transformations determine whether global NoPE projections read out as larger recent-key logits.This is an approximate relation under concentrated norms and exponential filter weights.Source: A. Theory · p. 21

B. Empirics

KDA-interleaved models at both 350M and 120M scales show increasing residual-stream recency bias across depth and recency-biased logits in later global NoPE layers. The appendix reports this pattern for natural-language and random-token inputs.

B. Empirics

  • Across both model scales, KDA layers primarily increase residual-stream recency bias across depth, while later global NoPE layers learn recency-biased logits.Figures 17 and 18 compare natural-language and random-token inputs, respectively; the authors conclude the mechanism can apply beyond SWA to local sequence mixers such as gated linear attention.Source: B. Empirics · p. 21, Figure 17 · p. 22, Figure 18 · p. 23
Loading 2609.38109v1…