PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Mamba-3 Through Oscillatory Dynamics

Shutian Liang, Yue Song

arXiv:2610.13341v1cs.LG

TL;DR

Mamba-3 adds learned input-dependent rotations to recurrent updates, but their contribution beyond state retention is unclear. The paper interprets these rotations as underdamped dynamics and finds benefits for state tracking and long-context language modeling, with phase interventions highlighting the role of learned rotations.

  • Problem

    What do Mamba-3’s learned rotations contribute beyond state retention, particularly over long contexts?

    Source: §1 · p. 1

  • Method

    The paper interprets Mamba-3’s decay–rotation transition as an underdamped second-order system and compares damping regimes and phase interventions.

    Source: §1 · p. 1, §5.3 · p. 8

  • Results

    Underdamped dynamics outperform non-oscillatory alternatives on state tracking, while oscillatory models improve long-context prediction and length generalization; phase interventions degrade long-context prediction.

    Source: §1 · p. 2, §4.3 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    Decay controls how much information persists, while phase shapes how it combines with new inputs and is accessed over time.

    Source: §6 · p. 9

    Non-oscillatory models trained from scratch also maintain stable perplexity beyond training context, and how they achieve this remains an open question.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Mamba-3 introduces input-dependent rotations into recurrent state updates, but their contribution beyond state retention remains unclear. We interpret these ro- tations as underdamped second-order dynamics, providing a framework for in- vestigating the role of oscillation in sequence modeling. Controlled comparisons show that underdamped dynamics outperform non-oscillatory damping regimes on state-tracking tasks, while language-model pretraining demonstrates benefits for long-context prediction and length generalization. Interventions in pretrained models further establish the importance of learned phase, and state analysis sug- gests that rotation regulates how retained information combines with incoming up- dates. Taken together, these findings identify underdamped dynamics as a useful mechanism for recurrent computation and suggest that oscillation supports long- context processing by regulating how information accumulates.

1 INTRODUCTION

Mamba-3 adds input-dependent learned rotations to recurrent state updates, raising the question of what they contribute beyond retention. The paper interprets these rotations as underdamped dynamics and reports benefits for state tracking and long-context language modeling.

1 INTRODUCTION

  • Selective SSMs use input-dependent updates to control what enters and leaves the state, while multiplicative decay controls retention.SSMs support linear-time sequence processing, constant-memory autoregressive inference, and parallel training algorithms.Source: §1 · p. 1
  • Mamba-3 learns input-dependent phase increments, unlike Transformer rotary positional embeddings with prescribed positional schedules.The rotations change the direction of the retained state, producing spiraling evolution under fixed coefficients.Source: §1 · p. 1
  • A second-order interpretation connects decay to damping and rotation to oscillation, enabling comparisons across underdamped, critically damped, and overdamped regimes.Controlled state-tracking experiments show an advantage for underdamped dynamics while holding the coupled second-order state structure common across variants.Source: §1 · p. 1
  • Removing, reversing, or shuffling learned phase substantially degrades long-context prediction, while state analysis suggests rotation moderates constructive accumulation.Phase removal also increases alignment between retained history and incoming updates, together with larger state norms.Source: §1 · p. 2
  • Underdamped dynamics improve state tracking; oscillatory pretraining improves length generalization at both evaluated scales, with downstream LongBench-E gains at the larger scale.The evaluated model scales are 187M and 444M.Source: §1 · pp. 1–2
  • The findings suggest recurrent memory depends both on how much information is retained and on how it is transformed as new inputs arrive.The paper proposes oscillatory dynamics as a design dimension for sequence architectures.Source: §1 · p. 2

2 BACKGROUND

Selective SSMs use recurrent hidden-state updates, with input-dependent controls for writing and reading and transition structures that distinguish Mamba variants. A damped second-order system provides a common framework for distinguishing oscillatory underdamped dynamics from non-oscillatory regimes.

2.1 SELECTIVE STATE-SPACE MODELS

  • Selective SSMs make recurrence components input-dependent, letting each token influence retention, writing, and readout; Mamba-1, Mamba-2, and Mamba-3 differ in their transition structures.Mamba-1 uses selective diagonal decay, Mamba-2 shares decay within each head, and Mamba-3 adds input-dependent rotation to decay.Source: §2.1 · p. 3
  • Mamba-2’s scalar retention changes state magnitude while preserving direction; Mamba-3’s rotary state pairs add a phase degree of freedom that changes direction as well as decaying.SISO and MIMO variants share the decay–rotation structure but differ in their input and readout parameterizations.Source: §2.1 · p. 3

2.2 DAMPED SECOND-ORDER DYNAMICS

  • A damped second-order system is parameterized by natural frequency, damping ratio, and external forcing, with its eigenvalues determining intrinsic response.The paper uses this system as a common framework for comparing damping regimes.Source: §2.2 · p. 3
  • Underdamped systems have complex-conjugate eigenvalues and decaying oscillations, whereas critically damped and overdamped systems have real eigenvalues and non-oscillatory intrinsic dynamics.Table 1 summarizes these three damping regimes.Source: §2.2 · p. 3, Table 1 · p. 3

3 AN OSCILLATORY INTERPRETATION OF MAMBA-3

The paper converts Mamba-3’s discrete decay–rotation update into continuous dynamics with coefficients fixed over each interval. Rewriting that evolution as a second-order system shows that nonzero learned rotation yields underdamped oscillation, with decay setting the damping envelope and phase setting frequency and direction.

From discrete rotation to continuous dynamics

  • Each discrete decay–rotation update is the exact flow of a continuous-time generator when coefficients are held fixed over the interval.The analysis considers homogeneous propagation, separating intrinsic transition dynamics from input injection.Source: §3 · p. 4

An underdamped second-order representation

  • Differentiating the component equations yields a second-order representation, and the auxiliary coordinate can be recovered when ν ≠ 0, making the representations equivalent within the interval.The derivation starts from the continuous dynamics of a rotary state pair.Source: §3 · p. 4
  • For a > 0, any nonzero ν gives 0 < ζ < 1; a sets the damping envelope, |ν| the oscillation frequency, and ν’s sign the rotation direction.Thus Mamba-3’s learned phase controls the oscillatory component of state evolution.Source: §3 · p. 4

4 OSCILLATION AND STATE TRACKING

Controlled comparisons isolate oscillation from second-order state coupling, while mode analysis characterizes how damping regimes differ. On state-tracking tasks, underdamped dynamics outperform critically damped and overdamped alternatives.

4.1 CONTROLLED COMPARISONS ACROSS DAMPING REGIMES

  • The variants share a coupled two-dimensional state structure, input-update rule, and state dimension, isolating the benefit of oscillatory modes beyond coupling alone.The damping parameter selects underdamped, critically damped, or overdamped alternatives around a shared spectral center.Source: §4.1 · p. 5

4.2 MODE PERSISTENCE AND STATE GEOMETRY

  • At fixed natural frequency, critical damping gives the fastest asymptotic attenuation and directional degeneracy, while underdamping preserves relative state geometry and decays more slowly.Overdamped propagation instead becomes dominated by the slow eigendirection, indicating asymptotic mode collapse.Source: §4.2 · p. 5

4.3 STATE-TRACKING TASKS

  • 95.68% scaled accuracy for the underdamped variant versus 81.60% for the overdamped variant on Arithmetic without Brackets; underdamped dynamics also lead across all three tasks.The advantage is particularly pronounced on Parity and Mod 3; Mamba-3 also outperforms variants with fixed or removed rotations.Source: Table 2 · p. 6, §4.3 · p. 6

5 OSCILLATION IN LANGUAGE MODELING

Across language tasks, phase removal particularly harms context-dependent prediction, while oscillatory pretraining improves long-context perplexity at both evaluated scales. Interventions and state analyses indicate that learned phase is tied to how retained information combines with incoming updates and is accessed.

5.1 OSCILLATION ACROSS LANGUAGE TASKS

  • Removing phase rotation harms LAMBADA accuracy substantially more than Relation accuracy, with larger layer windows generally increasing LAMBADA degradation.Figure 2 reports accuracy changes after phase removal in contiguous layer windows.Source: §5.1 · pp. 6–7, Figure 2 · p. 6

5.2 PRETRAINING AND LONG-CONTEXT EVALUATION

  • Oscillatory models achieve lower PG-19 perplexity at both scales, with a more pronounced advantage at longer contexts for the larger model.Figure 3 compares perplexity across context lengths; its dotted line marks the 2K training context.Source: §5.2 · p. 7, Figure 3 · p. 7
  • LongBench-E scores are comparable at 187M (9.53 with oscillation, 9.76 without) but favor oscillation at 444M (13.22 vs. 10.61).The reported downstream gains at the larger scale contrast with comparable scores at 187M.Source: §5.2 · p. 7, Table 3 · p. 7
  • Standard downstream accuracy is comparable at 187M (43.43 vs. 43.63) and modestly higher with oscillation at 444M (51.08 vs. 50.18).Oscillatory models also achieve lower FineWeb-Edu and LAMBADA perplexity at both scales.Source: §5.2 · p. 7, Table 4 · p. 8

5.3 THE ROLE OF PHASE IN LONG-CONTEXT LANGUAGE MODELING

  • Pretrained phase channels show oscillations at different timescales during language processing.Figure 4 presents representative phase-channel trajectories.Source: §5.3 · p. 8, Figure 4 · p. 8
  • All three phase interventions increase PG-19 perplexity; in SISO at 64K, removal and reversal raise it to 41.33× and 74.10× the native value.The interventions shuffle, zero, or reverse phase increments while leaving decay factors unchanged.Source: §5.3 · p. 8, Table 5 · p. 8
  • Removing phase increases history–update alignment and state magnitude, consistent with learned rotations moderating constructive accumulation.The increases occur at 16K and 64K in both SISO and MIMO models.Source: §5.3 · p. 9, Table 6 · p. 9
  • Removing rotation from writing or readout alone raises perplexity substantially, while removing both partially restores performance, supporting phase consistency between them.The remaining gap from the native model may reflect mismatch with the state accumulated under native rotations.Source: §5.3 · p. 9, Table 7 · p. 9

6 CONCLUSION

The conclusion links Mamba-3’s learned rotations to underdamped dynamics and argues that decay and phase play complementary roles in recurrent memory. The appendix extends this dynamical perspective to linear-attention and recurrent models, then describes their homogeneous propagation, including state preservation in linear attention.

6 CONCLUSION

  • Decay controls how much information persists, while phase shapes how retained information combines with new inputs and is accessed over time.The conclusion connects this account to underdamped dynamics and oscillatory computation.Source: §6 · p. 9

A EXTENSION TO LINEAR ATTENTION AND RECURRENT MODELS

  • The dynamical perspective motivates input-dependent oscillation as a design dimension for linear-attention and other recurrent architectures.The discussion notes that controlling underdamped modes in linear-attention updates remains future work.Source: §6 · p. 9, A EXTENSION TO LINEAR ATTENTION AND… · p. 15

A.1 HOMOGENEOUS STATE PROPAGATION

  • For homogeneous propagation, the appendix freezes coefficients within an interval and omits the additive input term to analyze the transition.For Momentum DeltaNet, it analyzes the augmented state containing both the recurrent state and momentum state.Source: §A.1 · p. 15

A.2 LINEAR ATTENTION

  • Linear attention preserves its state within the interval, while its full recurrence accumulates input contributions.Source: §A.2 · p. 16

A.3 DELTANET

The appendix represents several linear-attention updates through homogeneous dynamics and selected second-order subsystems. DeltaNet and Gated DeltaNet have shared-direction modal pairings, KDA has distinct decay modes, and Complex KDA and Momentum DeltaNet admit underdamped modes under stated conditions.

A.3 DELTANET

  • DeltaNet’s second-order plane representation has a decaying mode and a preserved mode, with all such planes sharing the same decaying key-direction mode.The planes are overlapping representations rather than independent two-dimensional subsystems.Source: §A.3 · p. 17

A.4 GATED DELTANET

  • Gated DeltaNet is overdamped on each plane, with key-direction decay rate γ + b and orthogonal-coordinate decay rate γ.Its two-dimensional plane representations overlap because they share the key-direction mode.Source: §A.4 · pp. 17–18

A.5 KIMI DELTA ATTENTION

  • KDA’s distinct modal decay rates yield overdamped pairs that can form independent, nonoverlapping invariant subspaces.Source: §A.5 · p. 19

A.6 COMPLEX KDA

  • Under the stated conditions, a Complex KDA mode admits an underdamped second-order realization.Source: §A.6 · p. 19

A.7 MOMENTUM DELTANET

  • When its discriminant is negative and 0 < α, µ < 1, Momentum DeltaNet’s frozen key-direction subsystem admits an underdamped second-order realization.Source: §A.7 · p. 20

B RELATED WORK

  • The related-work discussion situates this interpretation among oscillator-based recurrent models and input-dependent rotary recurrences.It identifies the connection as enabling damping-regime comparisons and analysis of learned phase and state accumulation.Source: B RELATED WORK · p. 20

C.1 MATRIX EXPONENTIAL OF THE ROTATION GENERATOR

  • The matrix-exponential derivation establishes the decay–rotation transition used in the main text.Source: §C.1 · p. 21

C.2 MODE PERSISTENCE ACROSS DAMPING REGIMES

The appendix contrasts modal persistence across damping regimes and analyzes how phase can regulate accumulation under persistent forcing. It also outlines constructions and evaluation protocols for state-tracking, language-model, and pretrained-model phase-intervention experiments.

C.2 MODE PERSISTENCE ACROSS DAMPING REGIMES

  • Underdamped propagation preserves two equally decaying modes, whereas overdamped propagation suppresses the fast mode and critical propagation becomes asymptotically rank one.At fixed natural frequency, critical damping has the fastest asymptotic exponential attenuation.Source: §C.2 · p. 23

C.3 STATE ACCUMULATION UNDER PERSISTENT FORCING

  • With rotation, accumulation under persistent forcing depends on the complex gap |1−ρe^iθ| rather than only the radial gap 1−ρ.Thus, near-unit retention does not necessarily imply large state growth; rotation can modulate constructive accumulation without changing radial retention.Source: §C.3 · p. 23

C.4 HOUSEHOLDER REFLECTIONS FOR ROTATION

  • In the two-dimensional construction, two reflections compose to a rotation.Source: §C.4 · pp. 23–24

D.1 STATE TRACKING TASKS

  • The state-tracking comparison uses underdamped, critically damped, and overdamped variants with otherwise identical architecture and task-specific protocols.The maximum training sequence length increases from 40 to 160, and selected configurations are evaluated at length 224.Source: §D.1 · p. 24

D.2 OSCILLATION ACROSS LANGUAGE TASKS

  • The pretrained-model language-task intervention sets rotation angles to zero within swept contiguous layer windows and reports accuracy differences in percentage points.Evaluations cover factual relations and LAMBADA using pretrained 1.5B SISO and MIMO models.Source: §D.2 · p. 24

D.3 FROM-SCRATCH PRETRAINING

  • From-scratch Mamba-3 MIMO models are trained on 100B FineWeb-Edu tokens at a 2K context length and evaluated for length extrapolation and beyond-context performance.The protocols include PG-19 and LongBench-E, plus additional datasets evaluated in 4096-token sequences.Source: §D.3 · pp. 24–25

D.4 PRETRAINED MODEL PHASE INTERVENTION

  • Pretrained-model interventions alter phase structure, including shuffling phase increments across rotary pairs while preserving their set and leaving time steps and radial decay unchanged.Write/read phase interventions are restricted to the final 100 prediction steps after native context processing.Source: §D.4 · p. 26

D.5 LONGBENCH-E TASK ABBREVIATIONS

The section introduces abbreviations for LongBench-E tasks.

D.5 LONGBENCH-E TASK ABBREVIATIONS

E.1 PHASE STRUCTURE INDUCED BY OSCILLATION

Selected state pairs show input-dependent rotations close to modular-counting phase increments: input 0 leaves phase nearly fixed, while input 1 advances parity and mod-3 phases. The pairs are consistent with conditional rotations representing modular state, though not every state pair is shown to share this geometry.

E.1 PHASE STRUCTURE INDUCED BY OSCILLATION

  • Parity and mod-3 state pairs rotate close to the phase increments for counting ones modulo 2 and 3.Input 0 changes phase by 0.50° for parity and 0.21° for mod-3; input 1 changes it by 179.51° and 119.78°, respectively.Source: Figure 5 · p. 27, §E.1 · p. 27
  • The observed pairs are consistent with conditional rotations encoding modular state, but do not establish that every state pair follows this geometry.A fixed rotation on every step would also advance phase on input 0, unlike the particular count-of-ones code described.Source: §E.1 · p. 27

E.2 LENGTH GENERALIZATION ON OTHER DATASETS

Across the evaluated context lengths, oscillatory models generally achieve lower perplexity on four datasets, with more consistent improvements at 444M parameters. The advantage varies by dataset and scale, and is especially apparent beyond the 2K training context on CodeParrot and TriviaQA.

E.2 LENGTH GENERALIZATION ON OTHER DATASETS

  • Beyond the 2K training context, differences are most apparent on CodeParrot and TriviaQA, while SlimPajama improvements remain small.The figure compares CodeParrot, MATH-Hard, TriviaQA, and SlimPajama at 187M and 444M model scales.Source: §E.2 · p. 27, Figure 6 · p. 28
  • At 444M, the oscillatory model has lower perplexity on all four datasets across evaluated context lengths; at 187M, it is higher on MATH-Hard.At 187M, lower perplexity occurs on CodeParrot, TriviaQA, and SlimPajama. The comparisons cover evaluated lengths up to 4K.Source: §E.2 · p. 27, Figure 6 · p. 28

E.3 TRAINING DYNAMICS

Oscillatory variants reach lower validation perplexity toward the end of pretraining at both model scales. On PG-19, they also achieve lower perplexity across evaluated context lengths, with a larger separation at longer contexts, especially for the 444M model.

E.3 TRAINING DYNAMICS

  • At both 187M and 444M, the oscillatory variant reaches lower validation perplexity toward the end of training.Figure 7 shows validation-perplexity trajectories throughout pretraining, with insets for the final 20,000 steps.Source: §E.3 · p. 28, Figure 7 · p. 28
  • On PG-19, both model sizes finish with lower perplexity at every evaluated context length from 2K to 64K, with a larger gap at 16K–64K, particularly at 444M.The variants remain close at short contexts; the separation grows at longer contexts.Source: §E.3 · p. 28, Figure 8 · p. 29, Figure 9 · p. 29

Related papers

Loading 2610.13341v1…