PaperlayerView exact revision on arXiv

Source-linked AI summary

MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models

Syed Ibrahim Omer, Ginny Y. Wong, Xiangyu Zhao

arXiv:2610.09092v2cs.LGcs.AI

TL;DR

Pre-trained SSMs are often conditioned outside their recurrent operator, leaving their temporal dynamics fixed. MaRK maps context into bounded modulations of a frozen SSM’s parameters, producing distinct timestep-conditioned memory profiles. The reported conclusion is that these profiles remain stable and recoverable under stated assumptions.

  • Problem

    Existing conditioning for iterative SSM generation uses input injection or activation modulation without directly parameterizing the underlying temporal dynamics.

    Source: §1 · pp. 1–2

  • Method

    MaRK maps context vectors to bounded modulations of a frozen SSM’s recurrence, read-in, read-out, skip, and discretization parameters.

    Source: §1 · p. 2

  • Results

    MaRK produces distinct timestep-conditioned memory profiles while preserving stability and recoverability under the stated assumptions.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results support viewing iterative-task adaptation as a controlled deformation of an SSM’s underlying Markov parameters.

    Source: §5 · p. 10

    Systematic comparisons with established conditioning frameworks on downstream benchmarks and evaluation of MaRK under scaling laws remain future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

State Space Models (SSMs) offer an efficient alternative to Transformers for sequence modeling, yet conditioning pre-trained SSMs for iterative generation typically operates outside the recurrent operator, through input injection or activation modulation. While such mechanisms expose the model to conditioning information, they leave the underlying temporal dynamics fixed. We introduce MaRK (Markov-adapted Recurrent Kernels), a dynamic operator-conditioning framework that maps context vectors directly into bounded modulations of a frozen SSM's recurrence ($A$), read-in ($B$), read-out ($C$), skip ($D$), and discretization ($\Delta$) parameters. Viewed through the lens of LPV-SSM systems, MaRK induces a context-indexed family of Markov parameter sequences, allowing each diffusion timestep to reshape the model's input-output memory kernel. We instantiate MaRK on a frozen 111M-parameter Hydra SSM backbone and study three adapter geometries: Hypernet, Chebyshev polynomial, and Discrete Cosine Transform kernels. Since these adapters modify the Markov parameter sequence through low-rank auxiliary maps on the frozen backbone, parameter-efficient fine-tuning arises as a structural consequence of the adaptation mechanism itself, requiring only 6.3--11M trainable auxiliary parameters to transition from a bidirectional objective to an iterative diffusion regime. The bounded recurrence parameterization further yields an analytic Affine Quadratic Stability certificate for the modulated recurrence. Through synthetic LPV recovery experiments and Markov-operator diagnostics, we show that MaRK recovers coordinate-invariant temporal operators under matched assumptions and produces distinct, stable timestep-conditioned memory profiles. Empirically, the Chebyshev variant yields the strongest performance, achieving an average validation loss of 2.55, followed by the DCT (2.59) and Hypernet (3.77) geometries.

1 Introduction

Existing SSM conditioning exposes models to context through inputs or activations without directly parameterizing the temporal operator. MaRK instead modulates a frozen SSM’s recurrent parameters, with bounded adapters and evidence of distinct timestep-conditioned memory profiles.

1 Introduction

  • Existing input injection and AdaLN condition computation without directly parameterizing the Markov parameter sequence that characterizes an SSM’s input-output temporal behavior.Input injection can require hidden states to carry context across the sequence, while AdaLN acts outside the core continuous-time operator.Source: §1 · pp. 1–2
  • MaRK directly modulates a frozen SSM’s recurrence, read-in, read-out, skip, and discretization parameters as functions of conditioning context.The framework is presented as operator-level conditioning rather than input additions or activation normalization.Source: §1 · p. 2
  • MaRK uses Hypernet, Chebyshev, and DCT kernel geometries with bounded low-rank modulations, requiring 6.3–11M trainable parameters for a frozen 111M-parameter Hydra base.The bounded parameterization also supports analytic Affine Quadratic Stability certificates for the modulated recurrence.Source: §1 · p. 2
  • Timestep-specific lag-norm profiles reveal distinct temporal memory signatures, and the evaluation includes synthetic operator recovery and Markov-operator diagnostics.The stated evaluation also includes stability certificates and adapter ablations.Source: §1 · p. 2

2 Linear Parameter Varying (LPV) Foundation

The LPV-SSM framing makes the SSM parameters functions of a diffusion-context vector, so context can alter the Markov sequence that characterizes input-output behavior. Bounded log-space modulation yields a constructive stability certificate for the diagonal recurrence.

2 Linear Parameter Varying (LPV) Foundation

  • MaRK views each modulated SSM layer as a continuous-time LPV system whose parameters depend on the diffusion-timestep context vector.The context vector is obtained from a timestep encoding and a two-layer MLP.Source: §2 · pp. 2–3
  • MaRK modulates A, B, C, D, and Δ; diagonal parameters receive bounded shifts, while B and C receive bounded scales and shifts through rank-r factorized projections.The factorization addresses the dimensionality of large B and C matrices.Source: §2 · p. 3
  • Modulating A and Δ rather than applying AdaLN-style scalar affine transforms directly parameterizes the system’s eigenvalue distribution and frequency response.This makes context explicit in the temporal kernel’s decay properties.Source: §2 · p. 3
  • The Markov parameter sequence characterizes the LPV-SSM’s complete input-output behavior, and conditioning its factors adapts the temporal operator at each generation step.Systems with different internal coordinates but identical Markov sequences are behaviorally indistinguishable.Source: §2 · p. 3

2.1 Constructive Stability via Log-Space Parameterization.

  • Bounded log-space shifts make each discretized diagonal recurrence eigenvalue lie in (0, 1), yielding an identity-matrix common Lyapunov function and an AQS stability certificate.The stated certificate applies to all admissible conditioning contexts.Source: §2.1 · pp. 3–4

3 Method

The method freezes a bidirectional Hydra BERT SSM backbone and trains MaRK adapters to condition its operator on diffusion timesteps. The setup uses masked discrete diffusion with a context-adaptive token-level objective and compares three kernel geometries.

3.1 Pretrained Hydra BERT SSM backbone

  • MaRK is applied to a frozen 111M-parameter, 23-layer Hydra BERT backbone, whose bidirectional mixer suits token refinement using left and right context.Only MaRK adapters are learned, conditioning the backbone’s A, B, C, D, and Δ parameters on diffusion context.Source: §3.1 · p. 4

3.2 Discrete Diffusion Language Modeling

  • The reverse masked-diffusion model predicts all masked positions in parallel using full bidirectional context, with the timestep embedding driving MaRK’s operator modulation.The forward process masks each token independently at the sampled timestep.Source: §3.2 · p. 4

3.3 Training Objective

  • CART weights each masked-token loss by the usable unmasked context surrounding that token, using a distance-weighted sum in which nearer visible tokens contribute more.The token-wise weights multiply masked-token cross-entropy terms before averaging.Source: §3.3 · p. 5, §3.2 · p. 5

3.4 MaRK Kernel Adapters

  • The adapter mapping acts as a scheduling function from semantic context to stable dynamical parameters, with adapter geometries compared head-to-head.The certified setting uses the bounded modulation structure.Source: §3.4 · p. 5
  • The Hypernet maps context to shifts and low-rank factors, the Chebyshev adapter uses polynomial-basis shifts, and the DCT kernel uses frequency-damped spectral weights across diffusion steps.The described parameter counts are approximately 6.3M for Hypernet, 11M for Chebyshev, and 7.1M for DCT.Source: §3.4 · p. 5

4 Experiments and Analysis

The experiments assess MaRK’s recurrence stability, synthetic operator recovery, and timestep-conditioned memory profiles, then compare adapter geometries, parameter ablations, and inference overhead.

4.1 AQS Certificate and Diagnostics

  • All 23 Hydra layers have positive stability margins under the stated modulation and discretization assumptions.The AQS check uses P = I and evaluates the worst-case discrete eigenvalue at δ = −βA with Δmin = 0.001.Source: Figure 1 · p. 6, §4.1 · p. 6

4.2 Synthetic LPV Identification

  • Under stated excitation and mixing assumptions, the Markov operator is identifiable up to structural equivalence, and squared-loss estimation error converges at O(1/N).Empirical MOE decays with fitted rates N^-0.54 for Hypernet, N^-0.66 for Chebyshev, and N^-0.64 for DCT; Chebyshev has the lowest structured-basis error after the smallest sample regime.Source: Figure 2 · p. 7, §4.2 · pp. 7–8

4.3 Dynamics Analysis

  • All three adapters produce distinct timestep-conditioned Markov sequences while preserving stable decay over thousands of lags.Hypernet shifts memory upward most broadly; Chebyshev yields smoothly ordered decay curves, while DCT selectively redistributes energy across lags and timesteps.Source: Figure 3 · p. 8, §4.3 · pp. 8–9

4.4 Ablations

  • 2.55 average loss for Chebyshev versus 2.59 for DCT; both structured bases beat Hypernet on every completed validation dataset.Chebyshev leads DCT on four datasets and ties on WikiText; their differences on WikiText and AG News are within seed noise.Source: Table 1 · p. 9, §4.4 · p. 9
  • Freezing A raises validation loss by +0.18 to +0.25 points, while B,C-only modulation degrades it by +0.34 to +0.51 points.A-Only beats Δ-Only by up to 0.75 points, but single-parameter configurations trail the Full baseline by up to +0.50 points.Source: Table 2 · p. 9, §4.4 · pp. 9–10
  • At sequence length 4096, MaRK adds 14% to 16% latency and less than 1% memory overhead relative to frozen Hydra.The conditioning cost is about 37 ms to 42 ms independent of sequence length, so its relative latency overhead falls as sequences grow.Source: Table 3 · p. 10, §4.4 · p. 10

5 Conclusion

MaRK frames SSM adaptation as bounded, context-dependent reconfiguration of the recurrent operator, with stability and recoverability supported in the studied setting. Its kernel geometries impose different structural priors, and CART weights the masked diffusion training loss by token.

5 Conclusion

  • MaRK makes SSM adaptation a controlled deformation of the frozen model’s recurrent operator, yielding context-indexed stable input-output dynamics in the studied setting.The conclusion presents operator-level conditioning as the central contribution and notes that systematic downstream comparisons and scaling behavior remain open.Source: §5 · p. 10

A.2 The MaRK Kernel Classes

  • The three kernels parameterize the same bounded modulation map: Hypernet is least constrained, while Chebyshev favors smooth variation and DCT favors bandlimited variation.Chebyshev coefficients represent polynomial components; DCT coefficients separate frequency content, while Hypernet is less directly interpretable.Source: §A.2 · p. 14

A.2.1 Hypernet Adapter

  • Hypernet maps the conditioning vector through an MLP and linear projections to factorized shifts, serving as the flexible baseline before structured basis restrictions.Its comparatively unconstrained mapping makes it the most expressive baseline in the study.Source: §A.2.1 · p. 15

A.2.2 Chebyshev Polynomial Adapter

  • Chebyshev maps context to a coordinate in [−1, 1] and polynomial coefficients, giving a low-degree global approximation suited to smoothly varying dynamics.Because the basis has global support, changing one coefficient can affect the modulation across the full coordinate range; degree K sets a bias-variance tradeoff.Source: §A.2.2 · p. 15

A.2.3 Discrete Cosine Transform (DCT) Kernel

  • DCT represents modulations with bandlimited real-valued basis transformations and learned attenuation of higher frequencies, but can limit sharp or non-bandlimited changes.Too few frequencies or overly strong decay can underfit non-bandlimited operator paths; all three variants still use bounded shifts.Source: §A.2.3 · pp. 15–16

A.3 CART Objective

  • CART replaces a sequence-level weight with per-token weights built from geometric kernels over visible tokens, then substitutes those weights into the masked diffusion loss.The kernel sharpness p controls whether influence spreads broadly or concentrates on nearby masked tokens; the paper uses p = 0.45.Source: §A.3 · p. 16

A.4 Scope of Contribution

The paper’s contribution is an initial, theoretically grounded formulation of operator-level SSM conditioning, not a claim of empirical superiority over external conditioning methods. Its scope is a continuous, ordered diffusion-timestep coordinate, and matched-budget quality comparisons and scaling behavior remain open.

A.4 Scope of Contribution

  • MaRK’s comparison with input injection and AdaLN motivates direct operator conditioning structurally; the paper does not claim it dominates those methods across empirical settings.The paper characterizes its contribution as the formulation and initial validation of an operator-level conditioning paradigm.Source: §A.4 · p. 17
  • The conditioning signal is scoped to a continuous, scalar, ordered diffusion timestep, so the paper does not claim transfer to autoregressive modeling or high-dimensional prompt conditioning.The smooth-operator prior relies on an ordered coordinate whose adjacent values index nearly identical denoising tasks.Source: §A.4 · p. 17
  • Dynamic Operator Modulation maps context ct to continuous-time system parameters (A, B, C, D, Δ), raising instability, identifiability-collapse, and optimization-burden risks; MaRK addresses these risks with a constrained architecture rather than unrestricted recurrent weights.Source: §A.4 · p. 17
  • The evidence is not a matched-budget quality comparison against external conditioning methods, and systematic downstream comparisons and scaling behavior remain future directions.The reported ablation, overhead comparison, and Markov-operator diagnostics support mechanistic and systems-level claims about the conditioning signal reaching the recurrent operator.Source: §A.4 · p. 17

A.5 Limitations and Future Directions

The experiments support internal MaRK design choices but do not establish broad comparative performance, scaling behavior, or downstream generation quality. The theoretical guarantees and conditioning regime also have explicit boundaries.

A.5 Limitations and Future Directions

  • The study does not establish that operator modulation outperforms AdaLN or input-stream injection on reconstruction quality.The comparison is limited to inference latency, memory, and parameter count, without a controlled head-to-head baseline under matched training compute, objective, and frozen-backbone constraints.Source: §A.5 · p. 18
  • Reported language-model experiments use a 111M-parameter Hydra-BERT backbone with 6.3–11M trainable adapter parameters, so scaling laws remain unestablished.The results do not determine behavior as model depth, hidden width, adapter rank, sequence length, training tokens, or pretrained SSM family vary.Source: §A.5 · p. 18
  • The evaluation covers AQS certificates, synthetic LPV recovery, Markov-operator diagnostics, and CART-weighted validation loss, but not generation quality or downstream task accuracy.The authors also identify robustness across long-context language benchmarks as untested and suggest future controlled long-range evaluations reporting quality and systems metrics.Source: §A.5 · p. 18
  • The identifiability and generalization arguments assume a bounded, well-specified LPV-SSM class, stable Markov operators, sufficiently exciting inputs, sub-Gaussian noise, and mixing conditions.The authors caution that these guarantees should not be read as unconditional for arbitrary pretrained sequence models or conditioning distributions.Source: §A.5 · p. 18
  • Bounded modulation may constrain expressivity, and the tested conditioning coordinate is the diffusion timestep.The effects of modulation bounds, low-rank dimension, Chebyshev degree, DCT frequency count, and diffusion weighting remain open for systematic robustness study; applicability to non-continuous, unordered conditioning signals is also unresolved.Source: §A.5 · p. 18

B Proofs

The appendix gives theoretical arguments for strictly bounded MaRK kernels, covering stability and operator identifiability with statistical convergence.

B Proofs

  • The appendix first presents an AQS stability argument for bounded affine parameter variations.It then proves coordinate-invariant operator identifiability and the statistical convergence guarantee for Proposition 4.1.Source: B Proofs · p. 18
  • The identifiability result concerns the coordinate-invariant input-output operator rather than a particular internal state-coordinate representation.This is the stated target of the proof of Proposition 4.1.Source: B Proofs · p. 18
  • The appendix also proves a statistical convergence guarantee for Proposition 4.1.The section introduction identifies this as the second theoretical argument after the AQS proof.Source: B Proofs · p. 18

B.1 Proof of Affine Quadratic Stability (AQS)

The AQS proof uses bounded affine variations and a vertex condition to establish stability, with the diagonal Hydra ZOH case admitting the identity matrix as its certificate.

B.1 Proof of Affine Quadratic Stability (AQS)

  • For bounded affine variations over a hyper-rectangle, AQS requires a positive-definite P and positive ε satisfying the Lyapunov inequality at every vertex.The stated condition is A(vk)⊤PA(vk) − P ⪯ −εI for all vertices vk.Source: §B.1 · p. 18
  • Strictly bounded tanh modulation places the evaluated systems within the convex hull of the vertices; the diagonal Hydra ZOH case uses P = I.The proof applies the vertex argument to affine dependence on bounded modulation variables.Source: §B.1 · p. 18
  • If the Lyapunov inequality holds uniformly at all finite vertices, every interpolated system inherits the decrease and V(x) = x⊤Px decays exponentially.This is the proof's conclusion for the bounded affine family.Source: §B.1 · p. 19

B.2 Proof of Operator Identifiability (Proposition 4.1)

The proof identifies the observable Markov operator up to state-coordinate equivalence and derives convergence under stability, excitation, boundedness, and mixing assumptions.

B.2 Proof of Operator Identifiability (Proposition 4.1)

  • The identifiable object is the extended input-output Markov operator, including the direct D term, while internal matrices are identifiable only up to structural state-coordinate equivalence.If D is fixed by the model class, its term drops out; the local Markov operator assumes context is frozen, while varying contexts identify a time-varying product.Source: §B.2 · pp. 19–20
  • Persistent excitation is required at the operator level to identify the Markov parameters from the stochastic experiment.The proof states that impulse-response identification extends to the stochastic setting in L2 by persistent excitation.Source: §B.2 · pp. 19–20
  • AQS yields geometric decay of Markov operators, including time-varying products, which supports the mixing argument.The proof combines this decay with bounded-density innovations and exponentially mixing exogenous inputs to obtain exponential β-mixing.Source: §B.2 · p. 20

Then the empirical Rademacher complexity of the squared-loss class satisfies

  • The squared-loss complexity argument bounds the predictor class and transfers that bound using Lipschitz loss and β-mixing sequence techniques.It invokes covering-number and entropy-integral steps, contraction for squared loss, and independent-block coupling for dependent observations.Source: Then the empirical Rademacher… · p. 21
  • The convergence proof uses bounded losses and β-mixing, then connects excess risk to operator error through persistent excitation.The proof applies a uniform convergence theorem for absolutely regular sequences and uses the centered, independent noise and well-specified model to cancel noise variance in excess risk.Source: Then the empirical Rademacher… · p. 22
  • Under the stated assumptions, excess prediction risk and squared extended Markov-operator error are bounded by C/N with high probability.The constant depends on stability, representation, excitation, tail, parameterization, mixing, and confidence-level quantities, but not on N.Source: Then the empirical Rademacher… · p. 22

C.1 Training Details

Training uses a Hydra corpus with separate training and validation splits, while updates are restricted to MaRK adapters and timestep embeddings. The modulation settings bound the recurrence shift to support a uniform stability margin.

C.1 Training Details

  • The corpus contains 4,295,110,656 tokens: 3,537,149,952 for training and 757,960,704 for validation.Validation is used for monitoring and held-out loss reporting; model updates use only the training split.Source: §C.1 · p. 22
  • Only MaRK adapters and timestep embeddings are trainable; the Hydra backbone and masked-LM classification head remain fixed.Training uses masked-token diffusion negative log-likelihood with CART weighting at p = 0.45.Source: §C.1 · p. 23
  • All three kernels share a 14,400-step schedule with a 6.0 × 10−4 base learning rate, while Chebyshev uses degree 5 and DCT retains 8 frequencies on a 256-point grid.The rate warms up for 720 steps and then cosine-decays; the shared rank, adapter MLP dimension, and timestep embedding dimension are 2, 256, and 128.Source: §C.1 · p. 23, Table 7 · p. 24

C.2 Modulation Bounds and CART Settings

  • The A-shift is capped at βA ≤ 0.5, while the remaining modulation magnitudes are learned; finite bounds provide a uniform stability margin.The A-shift bound is the only modulation constraint entering the stability certificate, and increasing βA shrinks the margin toward zero.Source: §C.2 · p. 24

C.3 Evaluation Protocol and Statistics

The evaluation reports CART-weighted validation loss with seed-based uncertainty intervals. Validation random-number handling is controlled across datasets, with a batch cap for the largest split.

C.3 Evaluation Protocol and Statistics

  • Every language-model result is CART-weighted validation loss, reported as weighted_nll; perplexity is its exponential, not the reported metric.Accuracy-table cells average 10 seeds, while inference-table cells average 15 seeds.Source: §C.3 · p. 24
  • Tables 1 and 2 use mean ± 1.96×SEM over 10 seeds, while Table 3 uses a Student-t interval with 14 degrees of freedom over 15 seeds.Synthetic LPV curves use the same seed and interval convention as the tables.Source: §C.3 · p. 24
  • The harness restores the post-model-build RNG state before each dataset, and caps arXiv evaluation at 500 batches while evaluating smaller datasets in full.The RNG handling reproduces leave-one-out draws despite validation and model construction consuming the global random stream.Source: §C.3 · p. 24

C.4 AQS Certificate Evaluation

The AQS evaluation checks worst-case discrete-time stability margins using an identity Lyapunov function. Synthetic LPV experiments separately test recovery of frozen-time Markov operators under matched parameterizations.

C.4 AQS Certificate Evaluation

  • The AQS procedure uses P = I to obtain a closed-form certificate for the pretrained diagonal recurrence without solving an SDP.It evaluates the worst-case parameter vertex under the bounded A-shift using the smallest discretization, ∆min = 0.001.Source: §C.4 · pp. 24–25
  • The certificate is conservative: it fixes P = I, evaluates the least-stable vertex, and uses ∆min = 0.001; its reported margin is the worst-case eigenvalue's distance from the unit circle.Because βA is shared across bounded kernels, the certificate depends on pretrained Alog rather than adapter geometry, allowing one backbone pass to cover all three variants.Source: §C.4 · p. 25

C.5 Synthetic LPV Identifiability Algorithms

  • Synthetic identifiability tests use diagonal LPV-SSMs with state dimension 4, rank 2, length 20, six dataset sizes from 50 to 1,600, and 15 seeds.Inputs are standard normal, diffusion coordinates uniform on [0, 1], and outputs receive Gaussian noise with σ = 0.1; Chebyshev and DCT fix C = 1 to remove readout scaling ambiguity.Source: §C.5 · p. 25
  • Recovery is measured by frozen-time Markov operator error over a 200-point τ grid and lags k = 0 through 14.For Chebyshev and DCT, the readout is fixed at C(τ) = 1.Source: §C.5 · p. 25
  • Hypernet maps a sinusoidal timestep embedding through a four-layer SiLU MLP to bounded shifts of recurrence, input, and time-varying readout parameters.It is the only synthetic generator with a learned time-varying readout, so its recovery error reflects that path as well as recurrence and input maps.Source: §C.5 · pp. 25–26
  • Chebyshev varies recurrence and low-rank input factors through bounded polynomial shifts, while DCT uses cosine modes with learned spectral decay; both fix C = 1.The synthetic Chebyshev basis has degree 3, whereas the language-model adapter uses degree 5; the synthetic DCT matches the language-model adapter's basis length and frequency count.Source: §C.5 · pp. 26–27

Related papers

Loading 2610.09092v2…