PaperlayerView exact revision on arXiv

Source-linked AI summary

Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

Runlin Shi, Bojian Yin, Guoqi Li

arXiv:2609.02986v2cs.LG

TL;DR

Hybrid design lacks an evidence-grounded basis for deciding which roles FA and LA should perform and how to allocate them. The paper analyzes RoPE-based head functions, derives design principles, and instantiates them in HwH. HwH improves retrieval and zero-shot extrapolation over Transformer, pure-LA, and layer-wise hybrid baselines while retaining strong language modeling and commonsense reasoning.

  • Problem

    Hybrid architectures lack a grounded basis for choosing which roles FA or LA should serve and how to allocate them, while prior retrieval-head methods do not characterize the complementary function needed for design from scratch.

    Source: §1 · pp. 1–2

  • Method

    The paper uses RFIS and RPD to characterize RoPE-frequency dependence and derive head-level retrieval and positional functions, then applies the resulting principles in HwH.

    Source: §1 · pp. 2–3

  • Results

    HwH retains strong language modeling and commonsense reasoning while improving retrieval and zero-shot extrapolation over Transformer, pure-LA, and layer-wise hybrid baselines.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The analysis supports local positional modeling and position-independent global retrieval, with the functions assigned at head granularity and by layer.

    Source: §6 · p. 9

    The paper identifies future extensions to function-aligned GQA, other modalities, larger scales, improved components, and designs beyond multi-head Transformers.

    Source: §6 · p. 9

Abstract

from arXiv · show

Hybrid architectures combining Full Attention (FA) and Linear Attention (LA) are increasingly prominent, yet their allocation remains heuristic. We seek an evidence-grounded basis in head-level functional organization learned by RoPE-based Transformers. Behavioral probes do not yield a complete taxonomy, so we propose two intervention metrics: RoPE Frequency Importance Score (RFIS), measuring how each frequency affects a head's attention distribution, and RoPE Positional Dependence (RPD), isolating dependence on rotary positional modulation. On Qwen3-series models and Llama3.1, RFIS suggests and RPD verifies a complete taxonomy of retrieval and positional heads separated by a salient mid-low-frequency band. Controlled Transformers show that this boundary follows the training-length positional scale; we term it the Global Positional Band (GPBand). The analysis suggests a potential cause of zero-shot length-extrapolation failure and yields two principles: positional modeling should operate only locally, with global access through position-independent retrieval; and both functions should be assigned at head granularity with layer-specific allocation. We instantiate them in Head-wise Hybrid Architecture (HwH), using NoPE FA for global retrieval and LA for local positional modeling. With an FA-to-LA ratio below 1:3, HwH retains strong language modeling and commonsense reasoning while improving retrieval and substantially strengthening zero-shot long-context extrapolation over Transformer, LA, and a layer-wise hybrid baseline. Ablations validate both principles and component roles, highlighting principled hybrid architecture design as a promising route toward future foundation models.

1 INTRODUCTION

The paper addresses how to assign FA and LA roles in hybrids, using RoPE-based head functions as a design reference. It develops a retrieval–positional taxonomy and derives principles instantiated in HwH, which improves retrieval and zero-shot extrapolation over the cited baselines while retaining strong language modeling and commonsense reasoning.

1 INTRODUCTION

  • FA offers exact token access but has quadratic sequence complexity and a growing decoding KV cache, whereas fixed-state LA is efficient but can struggle with exact long-range retrieval.Existing hybrids rely on fixed allocations or empirically selected choices, leaving the role, granularity, and allocation strategy unresolved.Source: §1 · p. 1
  • Prior retrieval-head studies and methods based on that specialization do not characterize the complementary function needed to ground architecture design from scratch.The paper frames modern RoPE-based Transformers’ functional differentiation as a design reference for hybrids.Source: §1 · pp. 1–2
  • RFIS and RPD establish a complete taxonomy of RoPE-based Transformer heads as retrieval or positional, separated by a salient mid-low-frequency band.RFIS measures how removing a frequency changes the complete attention distribution; RPD isolates whether the effect arises from rotary positional modulation.Source: §1 · p. 2
  • Controlled Transformers show that the functional boundary follows the training-length positional scale, termed the Global Positional Band (GPBand).The analysis suggests global positional fitting as a potential failure mode beyond training length and motivates local positional modeling with position-independent global retrieval.Source: §1 · p. 2
  • The paper derives two hybrid-design principles: assign positional modeling locally and global access through position-independent retrieval, with functions allocated per head and by layer.HwH instantiates these principles using NoPE FA for global retrieval and LA for local positional modeling.Source: §1 · p. 2
  • HwH retains strong language modeling and commonsense reasoning while improving retrieval and zero-shot extrapolation over Transformer, pure-LA, and layer-wise hybrid baselines.Its component and allocation ablations further validate both design principles and the assigned roles of FA and LA.Source: §1 · p. 3

2 PRELIMINARIES

The preliminaries describe the attention mechanisms and positional encoding used in the analysis. FA computes token interactions, LA uses recurrent state, and RoPE rotates query and key dimensions by frequency.

2 PRELIMINARIES

  • FA forms query, key, and value projections and computes each head’s output; it requires O(T^2) pairwise interactions and a KV cache that grows linearly with context.GQA shares each of G KV heads among a group of H query heads; G=H and G=1 recover MHA and MQA, respectively.Source: §2 · p. 3
  • LA replaces pairwise full attention with a fixed-size recurrent state, yielding linear sequence complexity and constant-size recurrent memory for fixed dimensions.The paper uses Gated DeltaNet, which adds data-dependent decay and a delta-rule update.Source: §2 · p. 3
  • RoPE partitions query and key heads into two-dimensional groups rotated by frequency, yielding exact frequency contributions to their dot product.The passage identifies θ as the RoPE base and r=0 as the highest frequency.Source: §2 · p. 3

3 FUNCTIONAL DIFFERENTIATION AND PRINCIPLES FOR HYBRID DESIGN

Behavioral probes reveal head-level tendencies but do not provide a complete mechanism-level taxonomy. RFIS and RPD identify retrieval and positional heads separated by a training-length-related frequency band, motivating two principles for hybrid design.

3.1 BEHAVIORAL PROBES OF ATTENTION-HEAD TENDENCIES

  • Behavioral probes provide retrieval and local-streaming tendencies, but these are not globally complementary and cannot classify all heads mechanistically.Qwen3-4B is the primary analysis model, with Qwen3-1.7B, Qwen3-8B, and Llama3.1-8B-Instruct used for cross-scale and cross-family checks.Source: §3.1 · p. 4

3.2 RFIS REVEALS RETRIEVAL–POSITIONAL FUNCTIONAL DIFFERENTIATION

  • RFIS removes one frequency contribution while holding others fixed, then measures the change in the complete attention distribution; larger values indicate stronger dependence on that frequency.RFIS is bounded in [0, 1], and its nested expectation weights samples equally.Source: §3.2 · p. 4
  • Retrieval heads depend more on frequencies below the salient mid-low band, while non-retrieval heads emphasize the band and higher frequencies.LDscore has no comparably clear relation, and RFIS alone cannot establish whether the frequency effects depend on position.Source: §3.2 · p. 4

3.3 RPD VERIFIES THE POSITIONAL INTERPRETATION OF RFIS

  • RPD replaces one frequency’s relative rotation with the identity while preserving learned Q/K coordinates, confirming that band-and-above dependence is positional.Below-band RPD is nearly zero; together, RFIS and RPD establish a two-type taxonomy of retrieval and positional heads.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

3.4 GLOBAL POSITIONAL BAND

  • Changing the RoPE base shifts the salient band’s frequency groups while preserving periods at the 2048-token training-context scale, defining the Global Positional Band (GPBand).The matched Transformers use θ = 10^4 and θ = 10^6; the band shifts from groups 21–22 to 14–15.Source: §3.4 · pp. 5–6, Figure 3 · p. 6
  • The analysis motivates local-only positional modeling with position-independent global retrieval, assigning both functions at head granularity with layer-specific allocation.The authors suggest global positional fitting may contribute to extrapolation failure beyond the training length.Source: §3.4 · p. 6

4 EMPIRICAL VALIDATION WITH THE HEAD-WISE HYBRID ARCHITECTURE

HwH assigns NoPE FA to position-independent global retrieval and LA/GDN to local positional modeling, with layer-specific allocation. From-scratch comparisons and ablations report strong task performance and support the component roles and allocation strategy.

4.1 HEAD-WISE HYBRID ARCHITECTURE

  • HwH uses NoPE FA for global retrieval and LA/GDN for local positional modeling, with FA occupying at most one quarter of a layer.HwH-std uses GDN alone in the first and last layers, a 1:3 NoPE-FA:GDN ratio in the middle half, and 1:7 in the remaining layers.Source: §4.1 · p. 6

4.2 MAIN COMPARISON

  • The matched 380M and 1.4B models were pretrained on FineWeb-Edu for 15B and 100B tokens, respectively, with a 2K context length.Comparators were a RoPE Transformer, pure GDN, and Inter, which alternates three GDN layers with one NoPE FA layer.Source: §4.2 · p. 7
  • 380M: HwH-std matches or exceeds all baselines; at 1.4B it remains competitive and achieves the best overall retrieval result.The comparison covers language modeling and commonsense reasoning as well as real-world retrieval and RULER NIAH.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7
  • 4K: HwH-std extrapolates nearly perfectly through twice the training sequence length and performs strongest beyond 4K.GDN extrapolates above 2K only when the target lies in a training-length-sized local window near the sequence end; Inter's extrapolation nearly vanishes beyond 5K.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 FUNCTION-SPECIFIC COMPONENTS

  • SWA slightly improves retrieval and substantially improves extrapolation over GDN; replacing NoPE FA with RoPE FA sharply degrades retrieval and extrapolation.GDN is stronger than SWA in commonsense reasoning; RoPE FA replacement does not improve general modeling ability.Source: Figure 5 · p. 8, §4.3 · p. 8

4.4 LAYER-SPECIFIC ALLOCATION

  • Removing NoPE FA from the first and last layers improves extrapolation, especially through 4K; reducing their ratio from 1:3 to 1:7 further improves beyond 4K and strengthens retrieval.All tested allocation variants retain comparable commonsense-reasoning ability.Source: §4.4 · p. 8, Figure 6 · p. 9

5 RELATED WORKS

Prior work combines full and linear attention at layer, within-layer, or head granularity, while mechanistic studies of attention heads and RoPE leave the salient middle frequencies insufficiently explained.

Hybrid Architectures

  • Hybrid designs allocate attention through interleaved layers, within-layer branches or head types, and retention of retrieval-critical heads during conversion.Layer-wise allocation studies recommend LA-to-FA ratios of 3:1–6:1 (Wang et al., 2025); Tan et al. (2026) retain FA at retrieval-critical heads when converting pretrained Transformers.Source: §5 · p. 9

Mechanistic Analysis of Attention Heads and RoPE

  • Earlier frequency accounts associate low frequencies with semantic or symbolic behavior and high frequencies with positional behavior, leaving the middle frequencies without a sufficient mechanism-level explanation.Oka et al. (2026) observe the salient band and relate its location theoretically to the RoPE base and training length.Source: §5 · p. 9

6 CONCLUSION

The paper presents a head-level taxonomy and GPBand as a basis for two hybrid-design principles, then reports from-scratch HwH experiments supporting those principles and the assigned roles of LA and NoPE FA.

6 CONCLUSION

  • The analysis identifies the training-length-related GPBand as the functional boundary between retrieval and positional modeling and motivates local positional modeling, position-independent global retrieval, and layer-specific head allocation.From-scratch HwH experiments validate the principles and the respective roles of LA and NoPE FA.Source: §6 · p. 9
  • Future work may extend the analyses and principles to function-aligned GQA, other modalities, larger scales, improved components, and hybrids beyond multi-head Transformers.Source: §6 · p. 9

AI USE STATEMENT

  • Generative AI assisted code implementation, language polishing, LaTeX formatting, and figure preparation; the authors reviewed outputs, tested code, and take responsibility for the manuscript.Source: AI USE STATEMENT · p. 10

A DETAILED BEHAVIORAL ANALYSIS

Behavioral probes identify retrieval and local-streaming tendencies but do not classify every head mechanistically. Frequency interventions and cross-model analyses reveal a retrieval–positional distinction, with band structure varying across layers in pretrained models.

A.1 BEHAVIORAL PROBE DEFINITIONS AND MEASUREMENT SETTINGS

  • QRscore measures query-to-evidence attention, while LDscore measures the fraction of non-sink attention within a recent window; measurements use truncated, model-specific long-context samples.The analyses evaluate the final 128 retained query positions; QRscore preserves the gold evidence's relative position after truncation.Source: §A.1 · pp. 13–14

A.2 QWEN3-4B BEHAVIORAL PATTERNS

  • In Qwen3-4B, retrieval tendency is sparse and concentrated in middle layers, but early and later layers show cases where retrieval and local-streaming tendencies are both weak or not complementary.The probes therefore provide behavioral anchors rather than a complete taxonomy.Source: §A.2 · p. 14, Figure 7 · p. 15

A.3 CROSS-SCALE AND CROSS-FAMILY BEHAVIORAL CONSISTENCY

  • Retrieval and local-streaming tendencies show similar layer-specific organization across Qwen3 scales, Llama3.1, and Gemma, though shared Qwen3 architecture and training may partly explain within-family agreement.The source of the close head-level agreement remains open.Source: §A.3 · p. 14, Figure 8 · p. 15

B.1 MEASUREMENT SETTINGS

  • Pretrained-model RFIS/RPD measurements use 100 long-context samples and the final 128 retained query positions; controlled Transformers use 2,048-token sequences.RFIS and RPD interventions are performed separately for each layer, Query head, and frequency, with the visible-key set unchanged; panels use separate raw scales and a display-only color mapping.Source: §B.1 · p. 16

B.2 LAYER-MEAN FREQUENCY STRUCTURE IN QWEN3-4B

  • Qwen3-4B layer-mean RFIS/RPD show multiple separated GPBand-like structures rather than the single layer-invariant band observed in controlled Transformers.A multi-stage pretraining history is a plausible explanation, but no unique effective training length is known for every layer.Source: Figure 9 · p. 16, §B.2 · p. 16

B.3 CROSS-FAMILY MECHANISM-LEVEL CONSISTENCY

  • In Llama3.1 layers 13 and 14, retrieval-associated heads emphasize lower frequencies with weak positional dependence, while positional heads show similar RFIS and RPD at the salient mid-low band and above.The LD strips do not provide an alternative complete taxonomy.Source: §B.3 · p. 17, Figure 10 · p. 17
  • Shared GQA Key frequency coordinates can entangle retrieval and positional signatures within individual heads, and the performance effects of this interference remain to be quantified.The dominant two functions remain identifiable despite the entanglement.Source: §B.3 · p. 17

B.4 CONTROLLED ANALYSIS OF GPBAND

Controlled models link the observed GPBand to the positional scale set by RoPE base and training length; removing QKNorm shifts its location.

B.4 CONTROLLED ANALYSIS OF GPBAND

  • Changing θ from 10^4 to 10^6 at length 2048 moves the band from groups 21–22 to 14–15; doubling training length moves it to approximately group 24.The matched models vary only RoPE base or training length; Figure 11 shows the comparisons.Source: §B.4 · p. 17, Figure 11 · p. 18
  • At length 2048, predicted GPBand indices are 21.27 for θ = 10^4 and 14.18 for θ = 10^6, with period 2864 in both cases.For length 4096 and θ = 10^4, the predicted index is 23.68 and period 5727, close to observed group 24 with period 6283.Source: §B.4 · pp. 18–19, Table 3 · p. 18
  • RFIS recovers the band through its effect on the attention distribution, while RPD verifies that the band is position dependent.The same mid-low band is associated with the training-length positional scale.Source: §B.4 · pp. 18–19
  • Without QKNorm, the θ = 10^4 band shifts toward group 23 and the θ = 10^6 band toward groups 15–16; norm compensation is proposed only as a hypothesis.Direct norm trajectories would be needed to verify the suggested training dynamics.Source: §B.4 · p. 19, Figure 12 · p. 19

B.5 GEMMA-7B AND THE LIMITS OF NORM-BASED ANALYSIS

Gemma-7B shows an unusually extended low-frequency positional band, complicating norm-based interpretation without establishing a third dependence pattern. The section also explains the intervention metrics and bounded normalized readout.

B.5 GEMMA-7B AND THE LIMITS OF NORM-BASED ANALYSIS

  • Gemma-7B’s dominant position-dependent structure extends through approximately group 120, beyond the predicted training-length band near group 104.A weaker band near group 96 is also visible, but its role is unclear; the extension may reflect absent QKNorm or an undocumented training pipeline.Source: §B.5 · p. 19, Figure 13 · p. 20
  • Frequency location and Q/K norm magnitude alone cannot establish whether a channel depends on positional modulation; RPD is needed for that distinction.Norm-based estimates omit directional alignment, while RFIS removes one frequency’s exact contribution and RPD replaces only its relative RoPE rotation with identity.Source: §B.5 · p. 20, C THEORETICAL DETAILS OF RFIS AND RPD · p. 20
  • In Gemma-7B layers 14 and 16, retrieval-associated heads emphasize lower frequencies with weak RPD, while positional heads show matched RFIS/RPD activity near and above the shifted band.The 128 frequency groups dilute individual contributions, making the pattern less distinct; the shifted boundary remains unresolved but yields no stable third pattern.Source: §B.5 · p. 20, Figure 14 · p. 21

C.1 BOUNDEDNESS OF THE NORMALIZED JS READOUT

  • The normalized RFIS and RPD readouts are bounded by 1; finite-logit softmax distributions generally approach, but do not attain, that upper bound.The ln 2 bound follows from the chosen logarithm base, and changing the base leaves normalized scores unchanged.Source: §C.1 · p. 22

D ADDITIONAL EXPERIMENTAL DETAILS

The appendix specifies matched from-scratch pretraining, baselines, evaluation protocols, and configuration details for the experiments. It also documents a scale- and configuration-dependent short-length instability in two 380M models.

D ADDITIONAL EXPERIMENTAL DETAILS

  • Models are pretrained on FineWeb-Edu at 2048-token context, with 380M and 1.4B scales trained for 15B and 100B tokens, respectively.Within each scale, architecture comparisons share data order, optimizer schedule, and evaluation protocol; Q/K/V widths are matched for capacity comparisons.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 22
  • The baselines represent RoPE full attention throughout, linear GDN throughout, and a layer-wise hybrid alternating three GDN layers with one NoPE full-attention layer.Evaluation covers language modeling, commonsense reasoning, real-world retrieval, and synthetic retrieval; the dense NIAH grid spans contexts up to 8192 tokens and 11 insertion depths.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 22
  • At 1K, the 380M Transformer and HwH-std-r show anomalously poor NIAH-Single-2 responses, whereas the 1.4B Transformer remains stable at 1K and 2K.The authors characterize this as scale- and configuration-dependent short-sequence robustness, distinct from length-extrapolation behavior.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 23

E ADDITIONAL HWH CONFIGURATION DETAILS

  • The 380M ablation configurations distinguish outer layers from the middle half, and HwH-swa uses sink-free RoPE sliding-window attention with window 128 as its local component.Table 4 records the ablation distinctions; its ratios denote global- versus local-component Q/K/V dimensions.Source: E ADDITIONAL HWH CONFIGURATION DETAILS · p. 23, Table 4 · p. 23

E.1 DETAILED ABLATION RESULTS

The appendix provides task-level ablation tables for functional components and layer-specific allocation, alongside complete 380M NIAH-Single-2 length–depth grids.

E.1 DETAILED ABLATION RESULTS

  • Tables 5–8 report language-modeling and real-world-retrieval task-level results for functional-component and layer-specific-allocation ablations.The supplied table captions identify the task and ablation scope but do not state cell values or a winner.Source: Table 5 · p. 23, Table 6 · p. 23, Table 7 · p. 24
  • Figure 15 consolidates complete 380M NIAH-Single-2 results for baselines and component and allocation ablations, with HwH-std repeated as a reference.Rows run from 100% depth at the top to 0% at the bottom; white curves show depth averages on the same scale.Source: §E.1 · p. 23, Figure 15 · p. 24
Loading 2609.02986v2…