Dynamic Linear Attention

Xin Wang; Hui Shen; Boyuan Zheng; Xueshen Liu; Minkyoung Cho; Zhongwei Wan; Zesen Zhao; Zhuoqing Mao; Shen Yan; Mi Zhang

Source-linked AI summary

Dynamic Linear Attention

Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang

arXiv:2606.10650v1Jun 9, 2026cs.CLcs.AI

TL;DR

Long-context modeling with linear attention lacks adaptive memory granularity because fixed state-merging policies can obscure important tokens. DLA dynamically merges states based on token information variation while bounding memory capacity, and consistently outperforms state-of-the-art baselines across 16 datasets.

Problem
Existing multi-state linear attention methods use fixed or hard state boundaries that cannot adapt to token-level information variation during long-context modeling.
Method
DLA dynamically constructs states around information variation and selectively merges adjacent low-information states within a fixed-size chronological cache.
Results
DLA consistently outperforms state-of-the-art baselines across 16 datasets spanning commonsense reasoning, in-context retrieval, and long-context modeling.
Takeaways & Limitations
DLA provides an adaptive multi-state linear-attention framework that improves long-context modeling while preserving predictable inference cost.
Takeaways & Limitations
Fixed hierarchical multi-state schedules determine memory granularity independently of token-level representation variation, limiting fine-grained preservation around salient tokens.

Abstract

from arXiv · show

The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner. However, existing multi-state linear attention methods rely on fixed state merging policies that cannot adapt to dynamically varying token importance, irreversibly obscuring critical tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic memory modeling framework for multi-state linear attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control memory growth with minimal information loss. We pre-train DLA on two different linear attention models and evaluate on 16 datasets across three categories. Experimental results demonstrate the superiority of DLA over state-of-the-art.

1 Introduction

The introduction motivates dynamic memory modeling for long-context linear attention, arguing that fixed state construction cannot track non-uniform information and proposing DLA to adapt state resolution while bounding memory. DLA is evaluated across 16 datasets and is reported to outperform Log-Linear Attention across all tasks.

Motivation: Standard self-attention has quadratic computational and memory complexity, motivating efficient attention mechanisms for long-sequence modeling.Linear attention is presented as an efficient direction for scaling LLMs to long contexts.
Background: Multi-state linear attention partitions long token histories into chunks and summarizes them into compact memory states to improve long-sequence representation capacity.Log-Linear Attention is identified as a representative method improving efficiency and practicality for long-context inference.
Problem: Existing multi-state methods degrade as context length increases because fixed block sizes and rule-based merging schedules mismatch dynamically varying information density.These policies can obscure important tokens and accumulate errors over long sequences.
Method: DLA combines information-aware state construction with capacity-controlled memory, preserving higher resolution in semantically volatile regions while summarizing stable spans.The framework determines state boundaries from token-level information variation and explicitly bounds the total number of memory states.
Evaluation: 16 datasets cover eight commonsense reasoning benchmarks, six in-context retrieval datasets, and two long-context modeling datasets.DLA is pre-trained on the Mamba-2-780M and Gated DeltaNet-1.3B linear-attention backbones.
Results: DLA consistently outperforms Log-Linear Attention across all evaluated tasks and, with Mamba-2, reaches performance comparable to full-attention Transformers at similar parameter budgets.These are the introduction’s highlighted findings across the three evaluation categories.

2 Preliminary

This section introduces standard attention, linear attention, and multi-state variants as the foundation for DLA. It emphasizes that linear attention reduces computational cost but fixed or single-state memory limits fine-grained information preservation over long sequences.

Standard Attention: Standard self-attention has quadratic time and memory complexity in sequence length T, motivating sub-quadratic attention mechanisms.The formulation uses query, key, and value matrices with a causal mask.
Linear Attention: Linear attention removes softmax normalization and reorders computation through associativity, enabling a recurrent single-state formulation.The recurrent state matrix St ∈ R^d×d summarizes all past tokens.
Linear Attention: Linear attention enables linear-time inference with constant memory but compresses the entire history into one state, limiting representation capacity under long contexts.The feature map ϕ(·) is typically an identity mapping or learnable linear projection unless otherwise specified.
DeltaNet: DeltaNet adds a data-dependent step size for controlled forgetting, yet its single global state cannot selectively preserve fine-grained information over long sequences.This improves adaptivity over a pure accumulator without resolving the single-state limitation.
Multi-State Linear Attention: Multi-state linear attention partitions historical context into segments and summarizes each separately, while log-linear attention uses logarithmically many Fenwick-tree states at multiple scales.Log-linear attention achieves O(T log T) training complexity and O(log T) time and memory per decoding step, but its fixed schedule can absorb salient tokens prematurely.

3 Dynamic Linear Attention (DLA)

DLA dynamically constructs memory states using token-level information variation, preserving finer representations near semantic transitions while summarizing stable regions. It further bounds memory capacity by merging adjacent low-information states while maintaining chronological order and stable inference cost.

Information-Aware Dynamic State Merging: DLA adaptively determines state granularity from token-level information variation instead of fixed schedules or predefined block boundaries.This enables high-resolution representations around semantic transitions and aggressive summarization of stable regions.
Capacity-Bounded Memory Modeling: When the cache reaches capacity, DLA merges adjacent states with the lowest information density, preserving temporal order while minimizing information loss.Each state tracks token counts and aggregated information scores, with information density computed as the score sum divided by the number of summarized tokens.
Capacity-Bounded Memory Modeling: DLA reads from a temporally ordered, capacity-bounded memory using linear attention, enabling stable inference cost while emphasizing informative states.The maintained memory contains at most K chronologically ordered summary states.
Information-Aware Dynamic State Merging: Fixed blocking can incur larger summarization deviation on non-stationary sequences because it may mix tokens with distinct representation variation or semantic regimes.The deviation bound is controlled by within-block heterogeneity, and adaptive boundaries aligned with change points can avoid cross-segment terms.
Information-Aware Dynamic State Merging: During pre-training, DLA uses soft gating for differentiable boundary learning, then switches to hard segmentation during inference for discrete states aligned with semantic boundaries.The same information-aware criterion is retained across training and inference.

4 Experiments

DLA is evaluated against vanilla and multi-state linear-attention baselines across 16 datasets spanning commonsense reasoning, in-context retrieval, and long-context modeling. It consistently improves task performance, efficiency over log-linear attention, and robustness across its components and hyperparameters.

Experimental Setup: DLA is evaluated on 16 datasets spanning eight commonsense reasoning, six in-context retrieval, and two long-context modeling benchmarks.Training uses 50B tokens with a 16K sequence length and a DLA state-cache capacity of 30.
Commonsense Reasoning: DLA consistently outperforms vanilla and log-linear linear-attention variants across eight commonsense reasoning tasks, with up to 52% and 22% relative accuracy gains over log-linear Mamba-2 and Gated DeltaNet, respectively.
In-Context Retrieval: DLA consistently outperforms baselines on six in-context retrieval tasks, achieving at most 49% performance improvement.
Long-Context Modeling: On RULER, DLA improves long-context retrieval across single-needle and multi-needle settings, reaching up to 350% relative improvement on S-NIAH-3 and 67% on MQ-NIAH versus log-linear attention.
Long-Context Modeling: On LongBench, DLA consistently outperforms vanilla and log-linear variants across narrative QA, multi-field QA, summarization, and few-shot learning tasks.
Efficiency: DLA achieves higher throughput and lower, more stable memory consumption than log-linear attention under varying batch sizes and context lengths, while both multi-state variants are less efficient than vanilla Mamba-2.
Ablation and Sensitivity: Both DLA components contribute to performance: DLA(I) and DLA outperform Log-Linear variants, while DLA consistently outperforms DLA(I); varying capacity k and boundary τ has only marginal effects.

5 Related Work

Linear attention and state space models address softmax attention’s quadratic long-context bottleneck with O(T) computation, but representative methods compress history into a single recurrent state using fixed merging policies. This motivates adaptive state construction that preserves fine-grained information under long contexts.

Table 5 reports ablations of Mamba-2 and Gated DeltaNet variants, including DLA(I), which uses only information-aware dynamic state merging.
Table 6 reports ablations of the Mamba-2 DLA variant across memory budget k and merge boundary τ.
Linear attention and state space models reformulate attention computation to achieve O(T) complexity on long sequences.
DeltaNet and Mamba compress the entire history into a single recurrent state by continuously merging incoming tokens into a fixed-size summary.
Fixed merging policies ignore token-level information variation, leaving adaptive state construction necessary for preserving fine-grained information under long contexts.

6 Conclusion

DLA is a multi-state linear-attention framework that replaces fixed state merging with information-aware dynamic construction and capacity-bounded memory modeling. It allocates memory resolution according to token-level information variation while keeping inference cost predictable, and is evaluated across two backbones and 16 datasets.

6 Conclusion: DLA replaces fixed merging with information-aware dynamic state construction for multi-state linear attention.Its memory resolution follows token-level information variation.
6 Conclusion: Capacity-bounded memory modeling keeps DLA’s inference cost predictable while improving representation quality and preserving efficiency.The framework controls memory through adaptive allocation of representation resolution.
6 Conclusion: DLA is pre-trained on two linear-attention backbones and evaluated on 16 datasets across three aspects.The supplied conclusion passage states that DLA performs consistently across this evaluation, but does not provide the completed result.