PaperlayerView exact revision on arXiv

Source-linked AI summary

MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution

Prasoon Dev, Anirudh Sankar, Vasudeva Varma

arXiv:2609.35664v2cs.CLcs.AI

TL;DR

GLA’s single-resolution memory must cover both local syntax and long-range semantics, a representational bottleneck that gating alone does not resolve. MS-GLA assigns heads to multiple pooled temporal resolutions and fuses their outputs. Across evaluation settings, the paper reports stronger overall results than GLA, including lower language-modeling perplexity and higher recall-task F1 for its best-performing configuration.

  • Problem

    GLA’s single-resolution memory must represent local syntactic details and long-range semantic structure, creating a representational bottleneck that gating alone does not resolve.

    Source: §1 · pp. 1–2

  • Method

    MS-GLA assigns dedicated GLA head groups to multiple temporal resolutions, keeping the total attention-head and recurrent-state budget equal to the baseline.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    MS-GLA {1, 2, 4} achieves the best overall results across the three evaluation settings, with roughly 9.5% lower average perplexity and roughly 18.9% higher average F1 than GLA.

    Source: §5.1 · p. 6, §5.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The findings establish multi-resolution decomposition as an effective extension of GLA across language modeling, recall accuracy, and long-context extrapolation.

    Source: §8 · p. 10

    Experiments are limited to the 340M-parameter, 7B-token scale, so behavior at larger model scales has not been empirically validated.

    Source: A Limitations · p. 13

Abstract

from arXiv · show

Gated Linear Attention (GLA) Transformers advance linear recurrent models through data-dependent gating, but face a core limitation: the fixed-capacity memory matrices across all heads operate at a single temporal resolution, where each token is processed individually, forcing them to simultaneously encode local syntactic patterns and long-range semantic structure, creating a representational bottleneck that gating alone is insufficient to resolve. We introduce Multi-Scale Gated Linear Attention (MS-GLA), which addresses this by distributing attention heads across multiple temporal resolutions. Coarser resolutions pool longer token spans naturally specializing toward long-range dependencies, while finer head groups retain sensitivity to local syntactic structure. A learnable, input-dependent fusion layer dynamically recombines head group outputs at each timestep, expanding effective memory capacity without increasing per-head state size. This multi-resolution decomposition draws on principles from Multi-Scale State-Space Models (MS-SSM), adapting them to the gated linear attention setting. We evaluate MS-GLA on language modeling, recall-intensive tasks, and long-context generalization. Across all settings, MS-GLA consistently achieves higher accuracy and lower perplexity than GLA at matched parameter counts, with up to 18.9% improvement on recall-intensive tasks and 9.5% lower average perplexity on language modeling benchmarks, validating multi-temporal resolution decomposition as a principled and effective extension of Gated Linear Attention.

1 Introduction

GLA’s single-resolution memory must cover both local syntax and long-range semantics, a representational bottleneck that gating alone does not remove. MS-GLA distributes the existing head budget across temporal scales so finer and coarser groups can specialize, then dynamically fuses their outputs.

1 Introduction

  • GLA adds data-dependent gates to choose what context to retain or forget while preserving efficient linear-recurrent sequence modeling.The introduction describes linear recurrent models as enabling O(1) inference memory and parallelized training.Source: §1 · p. 1
  • A single-resolution GLA memory must summarize the sequence while representing both rapidly changing syntax and slowly evolving semantics, which gating alone cannot resolve.The bottleneck is how a limited memory budget is stretched across different timescales.Source: §1 · pp. 1–2
  • MS-GLA assigns dedicated groups of GLA heads to multiple temporal resolutions, with coarser pooled sequences targeting long-range dependencies and finer heads retaining local syntactic sensitivity.The design adapts multi-scale decomposition principles from MS-SSM to gated linear attention.Source: §1 · p. 2
  • MS-GLA redistributes the baseline’s existing attention-head budget rather than adding full-sized models or increasing the parameter count.Specialized group outputs are causally upsampled and dynamically recombined by a learnable fusion layer.Source: §1 · p. 2

2 Background and Related Work

Linear attention replaces softmax attention with a linear recurrent formulation, and GLA adds data-dependent forgetting to improve what information is retained. Related work explores distinct ways to represent multiple memories or temporal scales.

2.1 Linear Attention and Gated Linear Attention (GLA)

  • Linear attention reformulates sequence mixing as a recurrent computation to obtain linear-time complexity, while chunkwise training divides sequences into chunks with sequential inter-chunk updates and parallel intra-chunk computation.GLA adds a data-dependent forget gate to the recurrent update; its gate is parametrized for row-wise scaling and supports chunkwise parallel training.Source: §2.1 · pp. 2–3

2.2 Mixture-of-Memories (MoM)

  • Mixture-of-Memories maintains independent memory states and uses a learned top-k router to direct each token to a sparse subset, whereas MS-GLA processes the sequence at multiple pooled temporal resolutions.The paper describes the mechanisms as complementary and proposes combining them as future work.Source: §2.2 · p. 3

2.3 Multi-Scale Sequence Modeling and MS-SSM

  • MS-SSM decomposes sequences into scales using causal convolutions, feeds each scale to a separate SSM, and combines branch outputs with an input-dependent scale mixer.Coarser branches are initialized for slower forgetting and long-range dependencies, while finer branches favor local dynamics.Source: §2.3 · pp. 3–4

3 Multi-Scale Gated Linear Attention (MS-GLA)

MS-GLA distributes a fixed attention-head and recurrent-state budget across pooled sequence resolutions, then causally aligns and fuses the branch outputs. Its branches retain compatibility with chunkwise GLA training.

3 Multi-Scale Gated Linear Attention (MS-GLA)

  • MS-GLA uses multiple GLA pathways at different temporal scales to represent local patterns and long-range structure within the same layer.The design is presented as an architectural bias toward multi-timescale modeling.Source: §3 · p. 4

3.1 Multi-Scale Decomposition via Temporal Pooling

  • A scale s forms a lower-resolution sequence by non-overlapping average pooling blocks of s tokens; s = 1 preserves token-level resolution, while larger scales summarize larger temporal blocks.Sequences are right-padded when needed, and pooling can mask padding tokens; pooled length is approximately ⌈L/s⌉.Source: §3.1 · p. 4

3.2 Scale-Specific GLA Heads

  • Each scale receives dedicated, independently parameterized GLA heads while the total attention-head and recurrent-state budget remains equal to the baseline.Branches can specialize their gates and projections to their respective scales without parameter sharing.Source: §3.2 · p. 4

3.3 Causal Upsampling and Learnable Scale Fusion

  • Branch outputs are repeated and causally shifted to align them with the original token resolution, then input-dependent weights computed from the unpooled hidden state fuse the aligned outputs.The causal hold delays each pooled block’s output until all tokens in that block have been observed.Source: §3.3 · p. 5

3.4 Compatibility with Chunkwise GLA Training

  • Static, non-overlapping pooling keeps each branch contiguous, allowing chunkwise GLA computations on shorter sequences and preserving the baseline training structure.Multiple branches add computation, but their shorter sequences mitigate part of that overhead.Source: §3.4 · p. 5

4 Experimental Setup

The experiments compare MS-GLA variants with GLA under matched model and training budgets, focusing on language modeling and long-context evaluation. Models share a backbone and training recipe, with the head budget allocated differently across scales.

4 Experimental Setup

  • The experiments compare MS-GLA variants with a GLA baseline under matched model and training budgets, focusing on autoregressive language modeling and long-context evaluation.The shared implementation and optimization recipe are intended to isolate multi-scale temporal decomposition.Source: §4 · p. 5

4.1 Model variants.

  • Five MS-GLA variants use scales {1, 2}, {1, 4}, {2, 4}, {1, 2, 4}, or {1, 2, 4, 8}, with the same parameter count as baseline GLA.The variants partition the fixed total head and state budget across scales.Source: §4.1 · p. 6

4.2 Training details.

  • Models are trained from scratch on FineWeb-Edu as 340M-parameter models for 7B tokens, with matched training tokens and an identical optimization recipe.Training uses the same tokenizer and AdamW; configuration details are provided in Appendix B.Source: §4.2 · p. 6

5 Results

Across language modeling, recall, and long-context tests, MS-GLA {1, 2, 4} performs strongly, while the coarse-only {2, 4} configuration repeatedly underperforms. The results also report stable fusion routing and modest efficiency costs.

5 Results

  • Evaluation spans language modeling, recall-intensive retrieval, long-context generalization up to 30,720 tokens, routing stability, and computational efficiency.Source: §5 · p. 6

5.1 Language Modeling

  • Roughly 9.5% lower average perplexity and improved mean downstream accuracy across LAMBADA, PIQA, and HellaSwag are reported for MS-GLA {1, 2, 4} versus GLA.MS-GLA {1, 4} leads on Wikitext perplexity, while the four-scale model falls slightly short of {1, 2, 4} overall.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5.2 Recall-Intensive Tasks

  • Roughly 18.9% higher average F1 than GLA is reported for MS-GLA {1, 2, 4}, which leads on SWDE and SQuAD.MS-GLA {1, 4} has the best FDA score; the four-scale variant remains above baseline overall but trails {1, 2, 4}.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Long-Context Generalization

  • At positions extending to 30,720 tokens, fine-resolution configurations stay below GLA perplexity on both datasets, while {1, 2, 4} and {1, 2, 4, 8} have the flattest profiles.On SlimPajama, better MS-GLA configurations reduce the mid-sequence document-boundary perplexity spike; coarse-only {2, 4} diverges most severely on both datasets.Source: §5.3 · pp. 7–8, Figure 1 · p. 8

5.4 Fusion Layer Routing Stability

  • Fusion routing weights shift only trivially across 32K-token sequences, and the finest-resolution branch receives the majority of routing weight.For {1, 2, 4}, the fine-to-coarse split changes from [52.64%, 18.41%, 28.95%] in the first 10% to [52.57%, 18.41%, 29.03%] in the final 10%.Source: Table 3 · p. 8, §5.4 · p. 8

5.5 Computational Efficiency

  • MS-GLA {1, 2, 4} retains roughly 92% of GLA throughput with ∼14% higher active memory; coarse-only {2, 4} exceeds baseline throughput.Coarser branches process proportionally shorter pooled sequences, offsetting much of the added branch computation.Source: Table 4 · p. 9, §5.5 · p. 9

6 Discussion

MS-GLA {1, 2, 4} achieves the best overall results across the three evaluation settings. The discussion highlights the importance of retaining a fine-resolution branch, balancing scale coverage against branch capacity, and the long-context behavior of multi-scale configurations.

6 Discussion

  • MS-GLA {1, 2, 4} consistently achieves the best overall results across language modeling, recall tasks, and long-context generalization.The authors suggest three scales, including the finest resolution, balance multi-resolution coverage with per-branch representational capacity.Source: §6 · p. 9

6.1 The Necessity of Fine-Resolution Branches

  • Configurations retaining a native-resolution branch outperform GLA in aggregate, while {2, 4} degrades across all settings.Without the fine branch, LAMBADA perplexity reaches 31.47, recall F1 falls below GLA on SWDE and SQuAD, and long-context perplexity diverges most beyond 10K tokens.Source: §6.1 · p. 9

6.2 Scale Coverage vs. Branch Capacity

  • Adding scale s = 8 to {1, 2, 4} brings marginal or no language-modeling and recall benefit, and regresses on SWDE and FDA.The four-scale model remains competitive on long-context tasks, where the added coarse scale may modestly help with very distant dependencies.Source: §6.2 · pp. 9–10

6.3 Long-Context Behavior and Temporal Scale

  • On PG19, GLA's perplexity drifts upward beyond 5K tokens, while {1, 2, 4} and {1, 2, 4, 8} maintain flatter profiles.Better MS-GLA configurations also reduce SlimPajama's 12–13K-token boundary spike; this benefit depends on retaining a fine-resolution branch, whose router remains favored even at 15× the training context.Source: §6.3 · p. 10

7 Future Work

The authors identify evaluation beyond 340M parameters and longer training horizons as next steps. They also propose dynamic pooling resolutions, layer-specific scale sets, and hybrid architectures combining MS-GLA with periodic softmax attention.

7 Future Work

  • Future work includes testing multi-billion-parameter models over longer training horizons and exploring dynamic resolutions, layer-varying scales, and hybrid linear–softmax architectures.Dynamic resolutions could aggregate tokens by linguistic constituents or information density, while scale sets could vary across layers.Source: §7 · p. 10

8 Conclusion

The paper introduces MS-GLA to address GLA's single-resolution memory bottleneck by distributing recurrent state updates across temporal scales. It reports consistent gains over GLA in language modeling, recall accuracy, and long-context extrapolation.

8 Conclusion

  • MS-GLA addresses GLA's single-resolution memory bottleneck by decoupling recurrent state updates across temporal scales.The paper reports consistent improvements over GLA in language modeling, recall accuracy, and long-context extrapolation.Source: §8 · p. 10

A Limitations

The authors identify compute constraints behind several study limitations. The reported evidence is bounded by model scale, baseline scope, single-run results, and incomplete ablations.

A Limitations

  • Compute constraints, including a single GPU and hardware issues during the research window, contributed to the study's limitations.The paper directs readers to Appendix B for full hardware details.Source: A Limitations · p. 13
  • All experiments use 340M-parameter models trained on 7B tokens, so behavior beyond 340M parameters has not been empirically validated.The authors state that the added operations scale linearly with sequence length but have not been tested at larger scales.Source: A Limitations · p. 13
  • Comparisons cover the GLA family, not freshly trained RetNet, Mamba, or Transformer baselines under the same setup.Competitiveness with those broader baselines has not been directly measured.Source: A Limitations · p. 13
  • Each configuration has one reported training run (seed 42), with no multiple seeds or significance testing.Source: A Limitations · p. 13
  • The ablations vary the number and choice of scales, but not pooling operators or alternative fusion mechanisms.Examples left for future work include average versus max or last-token pooling, and uniform averaging instead of the learned softmax router.Source: A Limitations · p. 13

B Training Details

Appendix B documents the shared training setup for the GLA baseline and MS-GLA variants, including data, optimization, hardware, reproducibility, and implementation details.

Optimization

  • AdamW used a 3 × 10−4 peak learning rate, cosine decay, 3,400 warmup steps, 0.1 weight decay, and gradient clipping at 1.0.The minimum learning-rate ratio was 0.1; non-finite gradient steps were skipped, and ϵ was 1 × 10−15.Source: B Training Details · p. 13

Data and batching

  • All models were trained from scratch on FineWeb-Edu for 106,813 steps, matching a 7B-token budget, with 2,048-token sequences and batch size 32.Training used the fla-hub/transformer-1.3B-100B tokenizer and one gradient-accumulation step.Source: B Training Details · p. 13

Reproducibility

  • Runs used random seed 42, with checkpoints saved every 8,096 steps and training metrics logged every 10 steps.Source: B Training Details · p. 13

Hardware and parallelism

  • Training used one NVIDIA RTX PRO 6000 Max-Q GPU, and a 340M-parameter run took approximately 2–3 days depending on configuration.The workstation’s inadequate power supply constrained the overall compute budget and informed project scoping; tensor and loss parallelism were disabled, and models were compiled before training.Source: B Training Details · pp. 13–14

Framework and code

  • Models were implemented with flash-linear-attention and flame on torchtitan, and the modular implementation, per-model configurations, and train.sh launch script were released.The appendix states that the full configuration supplements Section 4.2; exact launch commands are included in train.sh.Source: B Training Details · pp. 13–14

Related papers

Loading 2609.35664v2…