PaperlayerView exact revision on arXiv

Source-linked AI summary

On Subquadratic Architectures: From Applications to Principles

Anamaria-Roberta Hartl, Levente Zólyomi, David Stap, Pieter-Jan Hoedt, Niklas Schmidinger, Lukas Hauzenberger, Sebastian Böck, Günter Klambauer, Sepp Hochreiter

arXiv:2606.12364v1cs.LG

TL;DR

It remains unclear which subquadratic sequence architectures best handle complex dependencies while reducing Transformers’ quadratic cost. This paper compares xLSTM, Mamba-2, and Gated DeltaNet across code and time-series settings, finding that xLSTM performs best overall through robust state tracking and accumulation.

  • Problem

    It remains unclear which subquadratic sequence architectures best model complex dependencies despite their scalability advantage over Transformers.

  • Method

    The paper compares xLSTM, Mamba-2, and Gated DeltaNet across code, time-series, distillation, and controlled synthetic length-generalization tasks using a unified memory-dynamics framework.

  • Results

    xLSTM-family backbones outperform the other subquadratic operators in nearly all comparisons across code pre-training, code distillation, and time-series pre-training.

  • Takeaways & Limitations

    The findings indicate that xLSTM’s gains on complex tasks stem from combining robust state tracking with counting-like accumulation.

  • Takeaways & Limitations

    Code language modeling uses 400M-parameter models, distillation uses a single teacher, and broader scales, teachers, domains, and operator families remain untested.

Abstract

from arXiv · show

Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from large language models, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.

1 Introduction

The paper compares xLSTM, Mamba-2, and Gated DeltaNet on code and time-series tasks with complex dependencies, finding that xLSTM leads across most settings. A unified architectural analysis and synthetic length-generalization tasks attribute its gains to robust state tracking and counting-like accumulation.

  • 1 Introduction: A unified formulation compares how xLSTM, Mamba-2, and Gated DeltaNet write, forget, overwrite, and read from state.This formulation yields a hypothesis about the architectural mechanisms underlying their empirical differences.
  • 1 Introduction: Synthetic counting and state-tracking tasks test accumulation beyond training length and ordered finite-state updates, respectively.The tasks are designed to isolate counting-like accumulation and state tracking as distinct capabilities.
  • 1 Introduction: Together, practical comparisons, unified analysis, and synthetic tasks indicate that xLSTM’s gains stem from robust state tracking combined with counting-like accumulation.xLSTM solves both synthetic task families well beyond its training length.
  • 1 Introduction: The study provides a head-to-head comparison of xLSTM, Mamba-2, and Gated DeltaNet on tasks with complex dependencies beyond standard English-web pre-training.The evaluated domains are code and time series, whose structure requires tracking interacting states across long contexts.
  • 1 Introduction: xLSTM backbones lead across most empirical evaluation settings, motivating analysis of why their advantages emerge on complex data.The paper contrasts these settings with standard language-modeling and commonsense benchmarks, where architectural differences are small.

2 Experiments with Complex Dependencies

Across code pre-training, code distillation, and time-series foundation-model pre-training, xLSTM-family backbones outperform competing subquadratic operators in nearly all matched comparisons. The strongest and most consistent advantages appear on code generation and small-to-midscale forecasting, while broader benchmarks preserve the same ordering with smaller margins.

  • Overall comparison: Together, the experiments show a consistent xLSTM-family advantage across nearly all matched practical comparisons, with narrow exceptions in distillation and larger-scale time-series models.Gated DeltaNet leads MBPP+ in distillation, while the time-series gap narrows at 80M parameters.
  • 2.1 Code-focused language-model pre-training: xLSTM [7:1] leads code generation at every pass@k and in every training configuration.At pass@64, its margin over the next-best backbone is 1.43 points at 20B code tokens, 0.90 points at 100B, and 1.81 points on the mixed corpus.
  • 2.1 Code-focused language-model pre-training: xLSTM [7:1] has the best aggregate reasoning and commonsense score in all three training configurations, with smaller margins than on HumanEval.It leads the closest non-xLSTM backbone by under 0.1 points at 20B and 100B code tokens, and by roughly half a point on the mixed corpus.
  • 2.2 Code distillation: xLSTM [1:0] is the stronger code-distillation student on average, scoring 0.768 versus 0.755 for default Gated DeltaNet and 0.756 for Gated DeltaNet [−1, 1].It matches or exceeds default Gated DeltaNet on three of four pass@1 code metrics and trails only on MBPP+ by 0.014.
  • 2.3 Time-series foundation-model pre-training: xLSTM [3:1] achieves the best MASE and CRPS at every scale from 1M to 40M parameters.At 10M parameters, it reaches 0.733 MASE and 0.508 CRPS, compared with 0.767 and 0.525 for Mamba-2; at 80M, the models nearly converge.

3 Analysis of Leading Subquadratic Attention Architectures

The section unifies xLSTM, Mamba-2, and Gated DeltaNet as gated linear-attention architectures and analyzes their recurrent state dynamics. It attributes xLSTM’s advantage to its more flexible memory correction, supporting accumulation and state tracking.

  • Common linear-attention formulation: Linear attention exposes an explicit matrix state and enables O(T) complexity instead of regular attention’s O(T^2) complexity.Chunk-wise computation preserves linear complexity while processing within-chunk steps in parallel for efficient hardware use.
  • xLSTM: xLSTM combines linear-attention mLSTM layers with nonlinear recurrent sLSTM layers, using exponential input gating to down-weight or overwrite previous values.When correctly normalized, the exponential input gate behaves like a softmax over time, allowing flexible correction of stored information.
  • Mamba-2: Mamba-2 can be interpreted as xLSTM[1:0] with tied input and forget gates, making it less expressive and potentially vulnerable to counting problems.Its tied gates resemble a GRU, whose known counting issues motivate this expectation.
  • Gated DeltaNet: Gated DeltaNet combines Delta-Net fast weights with Mamba-2 transition dynamics and explicitly removes state components along the current key direction.This replaces matching old values with new ones, improving overwriting for retrieval but causing problems for counting.
  • Comparative analysis: Across the variants, xLSTM and Gated DeltaNet use independent gates, whereas Mamba-2 ties its gates; xLSTM provides the most flexible weighting correction.Mamba-2 has limited correction of previous weights, while Gated DeltaNet explicitly overwrites old values and xLSTM can down-weight them through a softmax-like input gate.

4 Experiments on Accumulation and State Tracking

Synthetic length-generalization experiments show that xLSTM[1:1] is the only tested configuration combining accumulation and state tracking across extrapolated sequence lengths. Its recurrent component enables perfect state tracking while retaining useful counting extrapolation, unlike competing designs’ tradeoffs.

  • Length Generalization: xLSTM[1:1] is the only configuration that length-generalizes on both Majority counting and parity.It achieves the highest counting accuracy at every tested length and solves parity perfectly across the evaluated lengths.
  • Experimental Setup: Models trained at length 128 were evaluated at lengths 128, 512, and 2048 on counting and state-tracking tasks.Tasks included AnBn, AnBnCn, Majority, Parity, Modular Arithmetic (Z5), and word-problem evaluation in S3.
  • Comparative Results: Mamba-2 collapses on every task, with AnBn accuracy dropping from 1.000 at length 128 to 0.241 at 2048.Its Parity accuracy never exceeds 0.352 even in-distribution.
  • Comparative Results: Default Gated DeltaNet solves the easiest counting variant at moderate length but degrades to 0.268 on Majority at 2048 and does not solve state tracking.The results match the expected accumulation-versus-state-tracking tradeoff.
  • Discussion: Adding xLSTM[0:1] to form xLSTM[1:1] yields perfect state tracking at all tested lengths while preserving weaker counting extrapolation.xLSTM[1:0] is the strongest counting model but fails on state-tracking tasks, separating the two primitives.

5 Conclusion · Appendix · A Notation table

The paper’s comparison finds that xLSTM generally outperforms Mamba-2 and Gated DeltaNet across complex data settings. A unified memory-dynamics formulation motivates analysis of gates, normalization, and overwriting mechanisms, while the study remains limited in scale, teacher diversity, and architectural coverage.

  • 5 Conclusion: The study provides the first comparison of xLSTM, Mamba-2, and Gated DeltaNet across complex data domains.
  • 5 Conclusion: xLSTM backbones outperform other subquadratic operators in nearly all comparisons across the studied settings.
  • 5 Conclusion: The empirical results motivate analysis of why xLSTM performs better under the proposed settings.
  • 5 Conclusion: A common formulation makes the architectures directly comparable at the level of memory dynamics.
  • 5 Conclusion: The formulation examines how gates, normalization, and overwriting mechanisms contribute to architectural behavior.
  • 5 Conclusion: Code-focused language modeling is conducted at a 400M-parameter scale, and distillation uses a single teacher.
  • 5 Conclusion: Only the time-series experiments include a scaling sweep, leaving larger model scales, additional teachers, and further data domains for future work.
  • 5 Conclusion: The comparison focuses on recent leading subquadratic architectures and excludes families already compared in Beck et al. (2024).

B Code-focused Language Model Pre-training Results … E Synthetic Task Results

The supplied passages identify two subsections within the merged results block, both under C Distillation Results: full Pass@k on HumanEval and results on math data. No substantive passages for sections B, D, or E are provided here.

  • C Distillation Results: The merged block covers code-focused pre-training, distillation, time-series foundation-model pre-training, and synthetic-task results, but the supplied evidence contains only distillation subsection entries.The available passages are both table-of-contents-style entries under C Distillation Results.
  • C Distillation Results: The C.1 subsection concerns HumanEval evaluation using the Full Pass@k metric.No numerical HumanEval result is included in the supplied passage.
  • C Distillation Results: The C.2 subsection evaluates distillation on math data.No metric values, baselines, or comparative findings are included in the supplied passage.

F Experimental & Implementation Details · G Related Linearization Work · A Notation table

The section identifies four experimental components—language-model pretraining, distillation, time-series foundation-model pretraining, and synthetic counting/state tracking—and defines notation used throughout the paper. Its notation includes layer counts, gating and transition parameters, operators, hybrid xLSTM notation, and time-series metrics.

  • F Experimental & Implementation Details: The experimental details include language-model pretraining as subsection F.1.The subsection is listed on page 22.
  • F Experimental & Implementation Details: The experiments also cover linearization via distillation in subsection F.2.This subsection is listed on page 22.
  • F Experimental & Implementation Details: Time-series foundation-model pretraining is presented in subsection F.3.This subsection is listed on page 24.
  • F Experimental & Implementation Details: Synthetic counting and state-tracking experiments are presented in subsection F.4.This subsection is listed on page 25.
  • A Notation table: The notation defines m and s as the numbers of linear-attention and nonlinear recurrent layers, respectively.It identifies m with xLSTM[1:0] blocks and s with xLSTM[0:1] blocks.
  • A Notation table: The notation includes parameters for discretization, transitions, forgetting, and write strength.These are w∆, a, wα, and wβ, respectively, with a specified as non-negative.
  • A Notation table: The table defines activation functions and operators, including tanh, exp, softplus, softmax, element-wise products, outer products, norms, and diagonal matrices.It also specifies the associated roles of tanh, exp, softplus, and softmax in the model functions.
  • A Notation table: The notation defines xLSTM[m:s](·) for hybrid models and identifies MASE and CRPS as time-series evaluation metrics.MASE denotes Mean Absolute Scaled Error, while CRPS denotes Continuous Ranked Probability Score.

B Code-focused Language Model Pre-training Results · C Distillation Results · C.1 Distillation: Full Pass@k on HumanEval

Across code-focused pre-training and code-model distillation, xLSTM [7:1] or xLSTM [1:0] is consistently strongest on code-generation evaluations. The results also show that xLSTM layer ratios matter, while negative eigenvalues help selectively beyond code pre-training.

  • B Code-focused Language Model Pre-training Results: The xLSTM layer ratio affects performance: xLSTM [1:0] and xLSTM [11:1] differ from xLSTM [7:1] across the evaluated configurations.The appendix introduces these variants specifically to examine the linear-attention-to-recurrent-layer ratio within xLSTM.
  • B Code-focused Language Model Pre-training Results: xLSTM [7:1] is best at every reported HumanEval pass@k across all three code-focused pre-training configurations.The compared models are 400M-parameter inter-layer hybrids trained with the same recipe, differing only in sequence operator configuration.
  • B Code-focused Language Model Pre-training Results: Gated DeltaNet is second-best in the 20B-token and 100B-token code-only settings, whereas Mamba-2 is second-best on the mixed corpus.The mixed corpus is Nemotron-CC-Code-v1 + FineWeb-Edu.
  • B Code-focused Language Model Pre-training Results: Gated DeltaNet [−1:1] does not substantially improve code hybrid pre-training, although negative-eigenvalue state tracking yields clear improvements in other reported tasks.The passage attributes those task-specific gains to state-tracking capabilities enabled by negative eigenvalues.
  • B Code-focused Language Model Pre-training Results: xLSTM [7:1] has the best aggregate reasoning and commonsense score in all three training configurations, but the margins are smaller than on HumanEval.This suggests those broad evaluations are less sensitive to backbone differences than code generation.
  • B Code-focused Language Model Pre-training Results: xLSTM [7:1] achieves the best aggregate score on pure-code pre-training at both 20B and 100B tokens, while the mixed-corpus setting has a different best ratio.The supplied passage identifies xLSTM [7:1] as best for pure code and notes a distinct result in the 20B-token mixed-corpus setting.
  • C.1 Distillation: Full Pass@k on HumanEval: xLSTM [1:0] is strongest at every k on both HumanEval and HumanEval+ for distilled code-model students.The full pass@k spread covers k ∈{1, 2, 8, 16, 32, 64}.
  • C Distillation Results: Gated DeltaNet [−1, 1] improves over default Gated DeltaNet at every k on both distillation benchmarks, but does not close the gap to xLSTM [1:0].All students follow the recipe described in Appendix F.2.

C.2 Distillation Results on Math Data

On math distillation, xLSTM [1:0] leads Gated DeltaNet overall, winning on GSM8K and AIME 2024 pass@8 while achieving the higher average across the reported benchmarks. The results are based on students distilled from Qwen3-4B-Instruct using a matched recipe.

  • xLSTM [1:0] leads on GSM8K and AIME 2024 pass@8, while Gated DeltaNet is slightly stronger on another reported metric.
  • 0.645 is xLSTM [1:0]’s average across GSM8K, MATH-500, and AIME pass@8, versus 0.625 for Gated DeltaNet.
  • The math-distillation comparison uses students distilled from Qwen3-4B-Instruct with a matched recipe, reporting exact match for GSM8K and MATH-500 and pass@8 for AIME 2024.

D Time-series Foundation-Model Pre-training Results

Time-series foundation models were pre-trained under a shared protocol differing only in the sequence mixer and evaluated zero-shot on GIFT-Eval using geometric-mean MASE and CRPS. xLSTM [3:1] performs best across both metrics from 1M to 40M parameters, while results nearly converge at 80M.

  • Evaluation setup: All models use the same time-series pre-training protocol, with only the sequence mixer changed, and are evaluated zero-shot on GIFT-Eval.Performance is reported using MASE and CRPS aggregated by geometric mean; lower values are better.
  • Scaling comparison: xLSTM [3:1] achieves the best result on both metrics from 1M to 40M parameters.Table 12 compares five parameter scales and also includes Gated DeltaNet trained with negative eigenvalues enabled in the state transition matrix.
  • Scaling comparison: At 80M parameters, xLSTM [3:1] and Mamba-2 match on MASE, while Mamba-2 is best on CRPS.The architectures nearly converge at this scale.

E Synthetic Task Results … G Related Linearization Work

The paper details synthetic length-generalization evaluations, implementation settings for language, distillation, and time-series models, and related linearization approaches. Synthetic tasks train at length 128 and evaluate extrapolation to lengths 512 and 2048.

  • E Synthetic Task Results: Synthetic sequence mixers are evaluated at lengths 128, 512, and 2048, with models trained at length 128.Table 13 reports accuracy on synthetic counting and state-tracking tasks across these three lengths.
  • F.1 Language Model Pretraining: Language models use Nemotron-CC-Code-v1 and the 100B-subset FineWeb Edu corpus, tokenized with the GPT-NeoX tokenizer.All models are trained on 8xH100 GPUs with bfloat16 and PyTorch Distributed Data Parallel.
  • F.1 Language Model Pretraining: The 400M-parameter language-model comparison matches model sizes and uses pre-norm architectures with RMSNorm.Defaults come from lm-engine for Gated DeltaNet, Mamba 2, and Attention, while xLSTM follows Beck et al. (2026).
  • F.2 Linearization via Distillation: Distillation replaces each Transformer softmax-attention block with an intra-layer hybrid combining linear attention and sliding-window attention.The linear-attention branch uses xLSTM [1:0], default Gated DeltaNet, or Gated DeltaNet [−1, 1] for code distillation.
  • F.2 Linearization via Distillation: The hybrid uses a per-head sigmoid gate to combine linear attention with sliding-window attention using window size 512 and four sink tokens.The xLSTM [1:0] branch applies head-wise softmax feature maps and per-head scalar output gates.
  • F.2 Linearization via Distillation: The distillation recipe performs hidden-state alignment followed by sparse knowledge distillation, with sequence length 4,096 and 10,000 Stage II optimization steps per domain.The recipe uses γ = 0.9, β = 0.1, and k = 256, enabling precomputed teacher targets during Stage II.
  • F.3 Pretraining Time Series Foundation Models: Time-series foundation models are pretrained on approximately 47.5M series from Chronos, GIFT-Eval, and synthetic KernelSynth data.The corpus is cleaned for zero overlap with the GIFT-Eval training corpus, and models use 4xA100 GPUs with bfloat16 and DDP.
  • F.4 Synthetic Counting and State-tracking Experiments: Synthetic experiments cover counting and state-tracking tasks, train from scratch at length 128, and report held-out target-position accuracy across 128, 512, and 2048.The tasks include AnBn, AnBnCn, Majority, Parity, modular arithmetic over Z5, and permutation composition in S3; results use five seeds and report the maximum.
Loading 2606.12364v1…