PaperlayerView exact revision on arXiv

Source-linked AI summary

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

Zhuokun Chen, Xi Lin, Xiyu Wu, Jiahao He, Jianfei Cai, Bohan Zhuang

arXiv:2610.05842v1cs.CL

TL;DR

Linear attention compresses history for efficient long-context decoding, but selective access to sparse, distant evidence remains difficult. HLA uses query-dependent gates to compose GDN chunk transitions, improving long-context results across pretrained Qwen3.5 models and controlled from-scratch tests.

  • Problem

    Compressed linear-attention histories can be difficult to search sharply for sparse, distant evidence, and fixed chunk-mixing weights cannot adapt to each query.

    Source: §1 · pp. 1–2

  • Method

    HLA augments GDN with query-dependent gates that interpolate exact historical chunk transitions with identity, controlling both chunk memory and its transformation of earlier states.

    Source: §1 · pp. 2–3

  • Results

    Across pretrained Qwen3.5 scales, HLA improves over Native GDN on LongBench-V2 and RULER, including a 5.57-percentage-point overall gain on LongBench-V2 at 0.8B.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    Across pretrained and controlled from-scratch evaluations, results support query-dependent composition of compact recurrent-memory summaries for long-context modeling beyond the training context.

    Source: §1 · p. 3, §5 · p. 10

    Pooling is bidirectional within each window, but representatives are used only after chunk completion; current-chunk queries use only the causal prefix transition.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Linear attention enables efficient long-context autoregressive decoding by compressing history into recurrent states, but this compression can make selective access to sparse and distant information difficult. Existing chunk-based extensions increase memory capacity, yet learned chunk-mixing coefficients may remain fixed with respect to input content and therefore cannot adapt historical access to each query. We introduce \emph{Hybrid Linear Attention} (HLA), a query-dependent chunk-level attention mechanism for Gated DeltaNet (GDN). HLA represents each completed chunk as an exact affine state transition and computes content-dependent routing gates from compact, self-attentively pooled representatives. Each gate interpolates the corresponding historical transition with the identity map, controlling both the chunk's additive memory and its transformation of earlier states. Effective-support regularization further encourages concentrated routing for sparse inference. We evaluate HLA under both pretrained adaptation and from-scratch training. Across Qwen3.5 models from 0.8B to 9B, HLA consistently improves over native GDN and fixed chunk mixing, with gains of up to 5.57 percentage points on LongBench-V2 and 3.97 points on RULER. In a controlled from-scratch 1.3B setting trained for 100B tokens with a 4K context, HLA also improves RULER performance from 4K to 32K, with gains increasing from 0.83 points at 4K to 4.22 points at 32K. These results demonstrate that query-dependent composition of recurrent memory improves long-context modeling and remains effective beyond the training context while using compact per-chunk affine summaries. Project page: https://caesarhhh.github.io/hla/

1 Introduction

Linear attention compresses history efficiently but can struggle to selectively retrieve sparse, distant evidence. HLA adds query-dependent chunk routing to GDN by gating exact affine transitions.

1 Introduction

  • Linear attention’s feature-dimension-bounded rank can limit sharp selection, especially when sparse relevant evidence must be found among long histories.Softmax attention can produce more concentrated distributions, while linear-attention variants often distribute scores more evenly.Source: §1 · pp. 1–2
  • Early information can be attenuated or overwritten through later GLA or GDN transitions, while fixed MHLA chunk weights cannot adapt to input- and query-dependent relevance.Figure 1 contrasts recurrent retention, attenuation of an early needle-containing chunk, and cross-input chunk relevance.Source: §1 · p. 2
  • HLA is evaluated on retrieval-intensive long-context tasks and requires only lightweight adaptation of pretrained GDN-based models.The evaluation focuses on accessing sparse and distant information over long sequences.Source: §1 · p. 3
  • HLA gates each historical chunk’s complete affine transition, controlling both its additive memory and its transformation of earlier states.Routing gates are predicted from self-attentively pooled representatives and interpolate transitions with the identity map.Source: §1 · pp. 2–3
  • Across LongBench-V2 and RULER, HLA improves long-context performance over native GDN and fixed chunk mixing under pretrained adaptation and from-scratch training.The evaluation includes generalization beyond the training context.Source: §1 · p. 3

2 Related Work

Prior efficient sequence models compress history into recurrent states, while hybrid architectures combine recurrent layers with a smaller number of softmax-attention layers. HLA instead makes the composition of exact chunk-level transitions query-dependent within GDN layers.

2 Related Work

  • Linear and recurrent models offer bounded-memory or linear-time sequence processing, but compressed history can remain difficult to access sharply when relevant evidence is sparse and distant.GDN’s past effects are set before future queries, and MHLA aggregates chunk states with shared fixed weights.Source: §2 · p. 3
  • Unlike layerwise hybrids, HLA adds lightweight query-dependent chunk attention within GDN layers by composing exact chunk-level state transitions.Each completed chunk is represented as an affine transition; query gates interpolate historical transitions with identity.Source: §2 · p. 4, Figure 2 · p. 4
  • Hybrid architectures pair predominantly linear or recurrent layers with a smaller number of full-attention layers to balance memory efficiency and modeling quality.The related work cites Kimi Linear’s 3:1 KDA-to-MLA interleaving and Ring-linear’s study of layerwise ratios.Source: §2 · p. 4

3 Methodology

HLA expresses GDN recurrence as exact chunk-wise affine transitions, then uses query-dependent chunk routing to compose historical transitions. Its training objective encourages selective historical access for sparse inference.

3.1 Gated DeltaNet as Chunk-Wise Affine Transitions

  • Each GDN chunk is summarized exactly as an affine transition whose additive term contributes memory and whose transformation term acts on earlier state.The transitions compose because affine maps are closed under composition.Source: §3.1 · pp. 4–5

3.2 Query-Dependent Chunk-Level Attention

  • HLA scores historical chunks using representatives formed by self-attention pooling, then assigns independent sigmoid routing weights rather than sum-to-one weights.Representatives are used only after their chunk is complete; tokens in the current chunk use its causal-prefix transition.Source: §3.2 · p. 5

3.3 Query-Conditioned Historical States

  • For each query, HLA interpolates every historical chunk transition with the identity, so its gate controls both additive memory and the transformation of earlier memory.A gate of one applies the original transition; a gate of zero leaves the incoming state unchanged.Source: §3.3 · p. 6

3.4 Training Objective

  • The training objective penalizes excessive historical routing mass and encourages concentrated routing without imposing a hard limit on accessed chunks.The effective-support quantity equals K for a uniform distribution over K chunks and decreases as routing becomes more concentrated.Source: §3.4 · pp. 6–7
  • At inference, routing weights below θ = 0.1 are zeroed without renormalization, approximating continuous routing while allowing skipped identity transitions to be omitted exactly.The thresholding reduces affine-state memory traffic and computation.Source: §3.2 · p. 6, §3.4 · p. 7

4 Experiments

HLA improves long-context benchmark results across pretrained Qwen3.5 scales and controlled from-scratch RULER evaluations. The experiments also examine implementation settings, retrieval and storage trade-offs, and inference latency.

4.1 Implementation Details

  • Pretrained adaptation freezes Qwen3.5-Base backbones at 0.8B–9B, while from-scratch 1.3B HLA and GDN models share a 100B-token budget and 4K training context.The reported pretrained training sequences use 4096 tokens; from-scratch models are evaluated on RULER from 4K to 32K.Source: §4.1 · p. 7

4.2 Main Results

  • With matched 100B-token training and a 4K context, HLA improves GDN's RULER macro average by 0.83, 2.67, 3.07, and 4.22 points from 4K to 32K.S1 improves from 56.6 to 100.0 at 8K, 23.8 to 97.0 at 16K, and 11.2 to 67.4 at 32K; results remain task-dependent.Source: §4.2 · pp. 7–8
  • Across Qwen3.5 models from 0.8B to 9B, HLA improves Native GDN by 5.57, 1.99, 1.39, and 1.19 points on LongBench-V2 and by 3.974, 1.250, 1.401, and 1.013 on RULER.HLA also achieves the highest RULER average at every scale compared with MHLA.Source: §4.2 · pp. 8–9, Table 2 · p. 8, Table 3 · p. 8

4.3 Ablation Study

  • LongBench-V2 accuracy rises from 25.25% to 29.62% when chunk size increases from 128 to 256, then falls to 26.64% at 512 and 26.24% at 1024.The study adopts 256 as the best evaluated setting, balancing retrieval granularity and contextual completeness.Source: §4.3 · p. 9, Figure 3 · p. 10
  • LongBench-V2 accuracy increases from 26.24% to 29.62% as the pooling window grows from 8 to 32, then falls to 28.43% at 64.A window of 32 is used as the default, balancing compactness with discriminative routing information.Source: §4.3 · p. 9, Figure 3 · p. 10
  • At chunk sizes 256, 512, and 1024, affine-state memory is 50%, 25%, and 12.5% of matched KV-cache memory, respectively.At the default chunk size of 256, storage is halved while achieving the highest LongBench-V2 accuracy among the evaluated chunk sizes.Source: §4.3 · p. 9, Figure 3 · p. 10
  • HLA adds 9.8–14.9% prefill latency and 13.9–18.8% steady-state decoding latency over Native GDN across model scales.The relative decoding overhead declines from 1.188× at 0.8B to 1.139× at 9B.Source: Table 4 · p. 9, §4.3 · p. 10

5 Conclusion

HLA composes exact affine transitions for completed chunks according to the current query. The paper reports gains in long-context benchmarks and identifies long-horizon agentic workloads as an untested evaluation setting.

5 Conclusion

  • HLA represents completed chunks as affine state transitions and dynamically modulates their composition according to the query, controlling new memory and transformations of earlier history.The conclusion reports improvements over native GDN and fixed chunk mixing, plus controlled RULER gains from 4K to 32K.Source: §5 · p. 10
  • Across pretrained Qwen3.5 and controlled from-scratch experiments, HLA improves long-context performance, including at contexts beyond the 4K training length.The conclusion attributes the supported scope to compact recurrent-memory summaries composed according to the query.Source: §5 · p. 10
  • The evaluation covers standard long-context language-understanding benchmarks, not interactive settings with continuously accumulated and selectively reused memory.The conclusion identifies multi-turn tool use and persistent memory in long-running agents as future evaluation directions.Source: §5 · p. 10

A.1 Algorithmic Sketches

The references describe HLA’s training/prefill and dynamic-decoding computations, from exact chunk summaries and pooled routing representatives to reverse historical readout. They also distinguish a mathematical forward reference from optimized implementations.

A.1 Algorithmic Sketches

  • Historical chunks are scanned newest to oldest, while the current chunk contributes through its causal prefix; zero-weight transitions may be skipped without deleting cached summaries.The algorithm sketches cover training/prefill and dynamic autoregressive decoding, with convolution cache preserved across chunk boundaries.Source: §A.1 · p. 14

B PyTorch-Style Pseudocode

  • The PyTorch-style listing is a mathematical forward reference for one sequence, layer, and value head, not the optimized training or inference implementation.It assumes common floating-point dtype; production uses required mixed-precision accumulation.Source: B PyTorch-Style Pseudocode · p. 14

B.1 Affine Updates and Prefix Readouts

  • Each chunk’s exact rank-one token updates produce affine summaries and local prefix readouts, computed by augmenting the native GDN state with values and an identity matrix.The augmented state yields the final pair [B_j | A_j].Source: §A.1 · p. 14, §B.1 · p. 15

B.2 Self-Attentive Pooling and Chunk Attention

  • Pooling applies self-attention within each window and averages token outputs; query-dependent chunk weights use log-mean-exp sigmoid scores over representatives.Historical weights below θ are zeroed at inference without renormalization.Source: §A.1 · p. 14, §B.2 · p. 16

B.3 Reverse Historical Readout

  • Historical readout traverses chunks in reverse order, adding each weighted memory readout and blending the readout vector with the chunk’s affine transform.Zero-weight transitions can be skipped when inference thresholding is active.Source: §B.3 · p. 17

B.4 Training and Prefill Reference

The training and prefill reference partitions the sequence into chunks, computes each chunk’s affine summary and local readouts, then routes each query over completed historical chunks. It preserves the real active prefix in the returned cache.

B.4 Training and Prefill Reference

  • Chunks are processed in parallel to produce affine summaries and local readouts, while pooled representatives are created only for complete chunks.A partial final chunk is processed at its true length and is not treated as historical in the forward pass.Source: §B.4 · p. 17
  • For each query, prefill scores only earlier chunks and combines their transitions by a newest-to-oldest historical scan after the local readout.Historical weights are thresholded only at inference; the reference returns pre-normalization readouts and an inference cache.Source: §B.4 · p. 17
  • The inference cache retains completed summaries and representatives alongside the active chunk’s exact affine and pooling state, preserving only real tokens for continuation.The cache also retains keys from the active chunk’s incomplete pooling window.Source: §B.4 · pp. 17–18

B.5 Dynamic Decode Reference

The dynamic decode reference updates the active chunk state for each new token, recomputes query-dependent weights over completed chunks, and then forms the readout. It closes and caches a chunk only after computing the current token’s output.

B.5 Dynamic Decode Reference

  • Each decode call receives one new token, updates the active affine cache, and recomputes that token’s historical weights.The native convolution cache is maintained separately and is not reset when a chunk is completed.Source: §B.5 · p. 18
  • The token’s output starts with the active-chunk readout, then adds selected historical contributions through a newest-to-oldest scan.The current chunk has weight one; inference thresholding is disabled during training.Source: §B.5 · p. 18
  • After the current token’s output is computed, a full active chunk is appended to the completed cache and its affine and pooling state is reset.The decode reference maintains the active chunk’s pooling-window keys until a representative is finalized.Source: §B.5 · p. 18

B.6 Complexity Analysis

The complexity analysis compares per-layer, per-head inference costs for HLA with softmax attention, recurrent linear attention, GDN, and MHLA. HLA’s accessed-transition costs depend on the number of selected chunks, while its cache stores affine summaries and routing representatives.

B.6 Complexity Analysis

  • Softmax attention has O(Td) decoding work and KV-cache storage, while recurrent linear attention and GDN have O(d2) decoding and cache costs; MHLA adds O(N2d2) prefill combination work.The MHLA mixture is shared within each query chunk, with O(Nd2) cache.Source: §B.6 · p. 19
  • HLA prefill requires O(Td2) affine and local-prefix work, O(Tpd) pooling, O(TNPd) representative scoring, and O(TMd2) selected-transition work.M is the average number of accessed chunks per query, including the current chunk.Source: §B.6 · pp. 18–19
  • HLA decoding costs O(d2) to update the active affine summary, O(d2+pd) amortized for pooling, O(NPd) to score representatives, and O(Md2) to apply selected transitions.Common backbone projections are omitted.Source: §B.6 · pp. 18–19
  • Each completed HLA chunk stores two d×d affine matrices and P representatives, for cache storage O(N(d2 + Pd) + Ld) including the active buffer.All chunk summaries remain cached because a later query may select a chunk skipped by an earlier query.Source: §B.6 · p. 19

B.7 Analysis of Long-Range Historical Contributions

Across the analyzed contexts, HLA shifts historical contributions toward distant chunks and retains stronger retrieval of far-away evidence than native GDN. These changes are concentrated in selected middle and upper layers.

B.7 Analysis of Long-Range Historical Contributions

  • Contributions measure RMS additive historical-chunk influence on pre-normalization output, not softmax attention or token-level causal attribution.The aggregate analyses use 64 paired examples across 4K, 8K, 16K, and 32K contexts.Source: §B.7 · p. 19
  • HLA raises distant-half contribution from 24.64% to 42.48% at 4K and 16.87% to 33.09% at 8K.The comparison uses the aggregate contribution analysis in Figure 4.Source: §B.7 · p. 19, Figure 4 · p. 20
  • At 16K and 32K, HLA raises distant-half contribution from 13.58% to 36.75% and 6.22% to 26.44%, respectively.At 32K, contribution-weighted mean distance is 2,876 tokens for GDN versus 9,591 for HLA.Source: §B.7 · pp. 19–20, Figure 4 · p. 20
  • At 16K, HLA retrieval remains above 90% in the farthest evidence-distance bin while GDN approaches zero; the same pattern persists at 32K.Both models face greater difficulty at the largest distances at 32K.Source: Figure 5 · p. 20, §B.7 · p. 20
  • HLA’s contribution changes are concentrated in selected middle and upper layers, where it extends source distance and favors distant, evidence-containing chunks.The pattern is layer-dependent rather than a uniform amplification of distant context.Source: §B.7 · p. 20

B.7.1 Qualitative Contribution Maps

Contribution maps show HLA distributing historical influence over a broader range of source positions than native GDN, including more distant history near target evidence in some examples. The visualizations are qualitative, not token-level causal attributions.

B.7.1 Qualitative Contribution Maps

  • The maps plot source position horizontally and model depth vertically, with a marker indicating target-evidence position.A shared logarithmic scale exposes weak but non-zero historical contributions.Source: §B.7.1 · p. 21
  • Across several middle and upper layers, HLA shows stronger distant-history contributions, with the difference becoming more pronounced at longer contexts.The layer-wise panels cover 4K, 8K, 16K, and 32K contexts.Source: Figure 7 · p. 22
  • HLA increases contribution-weighted source distance and assigns greater relative contribution to evidence-containing chunks in many middle and upper layers, especially at longer contexts.Source: Figure 8 · p. 22, Figure 9 · p. 22
  • At longer contexts, native GDN’s contributions concentrate near recent positions, while HLA retains visible contributions from more distant history, sometimes near target evidence.The visualizations are qualitative and do not establish token-level causal attribution.Source: §B.7.1 · p. 23, Figure 10 · p. 24
Loading 2610.05842v1…