PaperlayerView exact revision on arXiv

Source-linked AI summary

The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models

Yuheng Song

arXiv:2610.08816v1cs.LG

TL;DR

The paper asks how much state, context, or dynamical criticality is needed to forecast a specified long-memory process. It analyzes these resources in forecast risk and derives target-specific laws. For algebraic memory, matching bounds give a square-log state-complexity law; fractional memory changes the predictive geometry, while uniformly contractive nonlinear recurrences face a necessary accuracy-to-criticality condition.

  • Problem

    For a specified predictive-memory law, the paper asks what forecasting error is achievable with finite state, context, or stability margin.

    Source: §1 · pp. 1–2

  • Method

    The paper studies state, context, and stability resources through excess forecast risk for algebraic and fractional targets and for uniformly contractive nonlinear recurrences.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    For algebraic memory, best r-mode error decays as e^-Θ(√r), while fractional memory has a sharp 1/L context law and fixed-d square-log state complexity.

    Source: §1 · p. 2, §4.2 · p. 5

  • Takeaways & Limitations

    The results give target-specific resource laws, with vanishing error on an algebraic target requiring a uniformly contractive recurrence to move quantitatively toward criticality.

    Source: §1 · p. 2

    The sharp state-complexity theorem is limited to its specified finite-state and exponential architecture class, and the nonlinear stability theorem assumes uniform contraction and derivative budgets.

    Source: §8 · p. 9

Abstract

from arXiv · show

Long-range temporal dependence poses a resource question for sequence models: for a specified predictive-memory law, how much state, context, or dynamical criticality is required in order to forecast accurately? We study this question directly in forecasting risk. For algebraically decaying predictive memory, we prove matching upper and lower approximation bounds for exponential and finite-state modes. The best $r$-mode forecast error decays as $e^{-\Theta(\sqrt r)}$, so reaching forecast error $\tau$ needs $r=\Theta(\log^2(1/\tau))$ states or modes. Earlier curse-of-memory results establish broad limitations of stable recurrent models under different approximation notions; here both sides match for one canonical predictive target in forecast risk, which fixes the optimal resource exponent for that target. We then show that genuine fractional long memory changes the geometry itself. In particular, forecast error is measured after fractional integration, prediction from a finite context of length $L$ has an exact $1/L$ leading order, and a fixed fractional strength $d$ keeps the square-log state-complexity law. Near the short-memory boundary, we identify the relevant $d^2$ and $d^4$ scales and give a uniform constructive law in the intermediate regime. For nonlinear contextual recurrences with uniformly contractive state dynamics, we derive an exponential first-chaos envelope and an explicit necessary condition that relates forecast accuracy to the contraction margin. Vanishing forecasting error on an algebraic target forces the recurrence quantitatively toward criticality, a condition that is necessary and not by itself sufficient. Finite-sample Kullback--Leibler calculations further connect the predictive geometry to statistical information. Theorem-matched experiments with contractive state-space, gated recurrent, and attention models reproduce the state and stability predictions.

1 INTRODUCTION

The paper frames long-memory forecasting as a resource-allocation problem: for a specified predictive-memory law, it asks how state, context, and stability constraints determine forecasting error. It establishes target-specific resource laws and distinguishes its forecast-risk analysis from earlier work using other approximation notions.

1 INTRODUCTION

  • For a specified predictive-memory law, the paper asks how much state, context, or stability margin is needed to achieve a given forecasting error.It argues that long memory is not a binary property, but a target-specific forecasting question.Source: §1 · pp. 1–2
  • For algebraically decaying memory, matching approximation bounds yield a root-exponential best r-mode error and a square-log state or head complexity law.The result settles both sides of the resource question for this canonical target within the stated architecture class.Source: §1 · p. 2
  • For fractional long memory, the predictive norm applies fractional integration, finite-context prediction has a sharp 1/L leading order, and fixed fractional strength retains the square-log state-complexity law.Near the short-memory boundary, the paper identifies d^2 and d^4 scales and gives a uniform constructive law in the intermediate regime.Source: §1 · p. 2
  • For uniformly contractive nonlinear contextual recurrences, the first-chaos memory coefficients have an exponential envelope, yielding a necessary accuracy-to-criticality condition on algebraic targets.The paper also connects trajectory-level conditional KL divergence to accumulated forecast risk in Gaussian causal models.Source: §1 · p. 2

1.1 RELATED WORKS

  • Earlier stable-recurrence limitations use different approximation notions, while this paper matches both bounds for one canonical predictive target in forecast risk.Related work also studies long-memory parameterizations, state resources, power-law architectures, and fractional prediction; the paper positions its contribution as a sequence-model resource interpretation.Source: §1.1 · pp. 2–3

2 PREDICTIVE MEMORY AS A RESOURCE PROBLEM

The paper measures approximation by excess forecast risk, whose geometry depends on the process spectrum. It then defines state, context, and stability resources and the regimes in which their laws are studied.

2.1 FORECAST-RISK GEOMETRY

  • Excess forecast risk is a quadratic form in coefficient error, and Parseval expresses it as a spectral quadratic form determined by the process spectrum.With a regular spectrum, the predictive norm is equivalent to coefficient ℓ2 error; under genuine fractional long memory, that equivalence fails.Source: §2.1 · p. 3

2.2 RESOURCES, TARGETS, AND REGIMES

  • Table 1 summarizes three resource laws, distinguishing a two-sided state law, an exact leading-order context law, and a necessary-but-not-sufficient stability law.The introduction relates each law to its nearest existing line of work.Source: Table 1 · p. 4
  • The resource questions concern state dimension, finite context length, and contraction margin, evaluated on algebraic and fractional target-memory families.The stability resource is the contraction margin δ = −log ρ.Source: §2.2 · p. 4
  • The analysis uses different spectral regimes: regular spectra for state and stability, and a single spectral pole with a bounded regular factor for fractional context and state.The nonlinear stability analysis also assumes derivative bounds on a forward-invariant domain containing the stationary solution.Source: §2.2 · p. 4

3 SHARP STATE COMPLEXITY OF ALGEBRAIC MEMORY

For algebraic predictive memory under a stable regular-spectrum model, the paper obtains matching upper and lower approximation rates within a specified finite-state architecture class. The construction and obstruction meet at the same resource exponent.

3 SHARP STATE COMPLEXITY OF ALGEBRAIC MEMORY

  • The upper bound uses positive exponential quadrature of the Laplace representation, while the lower bound uses the rank limit on candidate Hankel matrices and the target's root-exponential Hankel spectrum.A corrected Ky Fan comparison transfers the spectral obstruction into forecast risk.Source: §3 · pp. 4–5
  • For target memory a⋆_j = cj^-α with α > 1, the stated finite-state class has matching upper and lower forecast-error rates, giving root-exponential decay and square-log state complexity.The class unions finite-context r-dimensional linear-state kernels and includes r-mode exponential kernels as a diagonal subclass.Source: §3 · pp. 4–5
  • Unlike broader curse-of-memory results under other approximation notions, this theorem matches both sides for one canonical algebraic predictive target and fixes its resource exponent.The claim is limited to the linear-state class described in the section.Source: §3 · p. 5

4 SINGULAR LONG MEMORY AND CONTEXT COMPLEXITY

Fractional long memory changes the geometry used to evaluate forecasting error, and the paper derives consequences for finite context and state resources. It also treats the short-memory boundary, where its constructive result leaves part of the lower-bound picture open.

4.1 FRACTIONAL BENCHMARK AND PREDICTIVE GEOMETRY

  • Under fractional long memory, forecast-risk geometry applies fractional integration, amplifying low-frequency coefficient perturbations rather than measuring ordinary coefficient error.For pure fractional noise, the stated identity is exact up to the factor σ².Source: §4.1 · p. 5

4.2 CONSEQUENCES FOR THE TWO RESOURCES

  • Finite-context prediction for the fractional model has a sharp 1/L leading order, which differs from an unweighted coefficient-tail calculation because the predictive metric is singular.For a fixed finite set of separated fractional poles with a sufficiently smooth positive short-memory envelope, the leading coefficient is the sum of squared pole strengths.Source: §4.2 · p. 5
  • For every fixed fractional strength d, Corollary 4 gives a state-complexity result in fractional forecast risk, with a positive exponential-quadrature upper construction and a fractional Hankel-spectrum lower bound.The construction is attained within the diagonal subclass.Source: §4.2 · p. 6

4.3 THE SHORT-MEMORY BOUNDARY

  • As d approaches zero, the fixed-d constants cannot remain fixed because memory’s contribution disappears in absolute forecast risk; O(1) memory suffices once τ ≳ σ²d².Source: §4.3 · p. 6
  • In the intermediate small-d window, the paper gives a uniform constructive upper bound using a truncated positive exponential kernel.The construction linearizes fractional coefficients as a_j(d) = d/j + O(d²) and truncates a positive harmonic exponential kernel.Source: §4.3 · p. 6
  • A matching uniform lower law in the small-d boundary layer remains open, as does the deeper regime τ ≪ σ²d⁴.The construction is an upper bound, not a complete characterization of those regimes.Source: §4.3 · p. 6

5 QUANTITATIVE STABILITY BARRIERS FOR RECURRENT MEMORY

The section distinguishes the paper’s forecast-risk question from prior memory bounds, then derives an exponential first-chaos envelope for uniformly contractive nonlinear recurrences. It uses that envelope to show that accurate forecasting of an algebraic target requires moving quantitatively toward criticality, though this condition is not sufficient.

5.1 RELATION TO EXISTING CURSE-OF-MEMORY THEORY

  • Prior curse-of-memory results use different notions and model classes; this section instead relates forecast accuracy to the target tail and contraction margin.The resulting forecast-risk lower bound can be inverted into an explicit accuracy-to-criticality requirement.Source: §5.1 · p. 6

5.2 CONTEXTUAL NONLINEAR RECURRENCE

  • Uniformly contractive contextual nonlinear recurrences have an exponentially decaying envelope for their first-chaos coefficients under the stated regularity conditions.The envelope follows from a Gaussian Stein identity and contraction of the state Jacobian across lags.Source: §5.2 · p. 7

5.3 ACCURACY FORCES NEAR-CRITICAL DYNAMICS

  • For an algebraic target with α > 1, vanishing forecasting error requires the contraction margin to approach criticality at a quantitative rate.The result holds with derivative budgets K and M and target amplitude c fixed.Source: §5.3 · p. 7
  • Near-criticality is necessary, not sufficient: it removes the fixed-contraction obstruction but does not guarantee accurate representation of algebraic memory.The lower envelope inverts the exponential first-chaos tail against the algebraic target.Source: §5.3 · p. 7

6 FINITE-SAMPLE INFORMATION CONSEQUENCES

The section identifies forecast risk with conditional Gaussian information and examines finite-section corrections for fractional processes. It also reports a uniform O(log T) correction under a fixed-pole, fixed-strength condition, while limiting that uniformity to pole locations.

6.1 CONDITIONAL TRAJECTORY INFORMATION EQUALS FORECAST RISK

  • For Gaussian causal predictors with common innovation variance σ2, expected conditional trajectory KL divergence equals T/(2σ2) times excess forecast risk.This identity links the predictive geometry to statistical distinguishability.Source: §6.1 · p. 7

6.2 FRACTIONAL FINITE-SECTION CORRECTIONS

  • The fractional finite-past prediction gap, normalized by d2, collapses onto one, supporting the sharp 1/L law of Corollary 3.The figure describes this as an exact finite-past prediction gap.Source: §6.2 · p. 8
  • The finite stationary block has the same local information constant as the entropy-rate experiment on the d = h/√T scale.Source: §6.2 · pp. 7–8
  • For a fixed number of fractional poles with fixed strengths and a uniform cluster-integrability margin, finite-section corrections are O(log T), uniformly over reflection-symmetric pole locations that may move and merge.The location-uniform result does not establish the corresponding strength-uniform refinement for triangular arrays of vanishing strengths.Source: §6.2 · p. 8

7 EXPERIMENTS

The experiments evaluate deterministic resource laws and test width and stability predictions in controlled sequence models. Across the reported controls, outcomes distinguish compatibility with the contraction envelope from functional form alone, while stability sweeps show differing behavior for diagonal and gated recurrences.

Resource-law calculations

  • Figure 1’s resource-law calculations use closed-form identities, spectral-risk evaluation, and controlled deterministic optimization rather than stored table values.The fractional context panel uses exact finite-past prediction variance.Source: §7 · p. 8

Architecture-facing experiment

  • As ρ approaches one, the diagonal state-space model keeps improving while the gated model retains a residual floor.At L = 64, the largest-ρ points are finite-context illustrations rather than asymptotic tests, and the gated floor may include higher-chaos variance.Source: Figure 2 · p. 9, §7 · p. 9
  • The slower exponential leaves a width-insensitive floor despite being exponential, while the contraction-matched exponential is fit almost exactly, so functional form alone does not explain the benchmark floor.The targets have different variances, and the reported outcome is unexplained fraction of target variance: held-out mean squared error divided by target variance.Source: Figure 2 · p. 9, §7 · p. 9
  • At the largest width, the diagonal model’s unexplained target-variance fractions are 7 × 10−6 for the boundary-matched exponential, 1.1 × 10−2 for the algebraic target, and 4.5 × 10−2 for the slower exponential.The slower-exponential floor is insensitive to width; the gated recurrence shows the same ordering, with a much smaller gap on the boundary-matched control.Source: §7 · p. 9, Figure 2 · p. 9

8 DISCUSSION AND LIMITATIONS

The discussion bounds the scope of the resource laws and distinguishes forecast-risk approximation from other memory metrics. The appendices give the risk identity, approximation construction, and spectral lower-bound machinery supporting the algebraic-memory result.

8 DISCUSSION AND LIMITATIONS

  • The sharp state-complexity theorem applies to the specified finite-state and exponential architecture class, not arbitrary neural networks; the nonlinear stability theorem assumes uniform contraction and derivative budgets.Selective or noncontractive architectures may fall outside the treated class, and near-criticality is necessary only.Source: §8 · p. 9
  • The results pose all three resource questions for a scalar target; the small-d architecture boundary below the d4 scale and growing numbers of moving Fisher–Hartwig singularities remain unresolved.Source: §8 · pp. 9–10
  • A state-dependent contraction budget replaces the single stability margin with a spectrum of margins, while a readout outside derivative budgets removes the sensitivity control used for the first-chaos envelope.Both settings are identified as outside the analyzed class.Source: §8 · p. 10

A.2 REGULAR AND SINGULAR SPECTRA

  • Under a regular spectrum bounded above and away from zero, forecast risk is equivalent up to constants to coefficient ℓ2 error; the upper equivalence fails for a fractional spectral pole.For the singular spectrum, the paper identifies a different predictive norm.Source: §A.2 · p. 14

A.3 FINITE-STATE KERNELS AND HANKEL RANK

  • The finite-state architecture includes finite-context truncations because truncating a state-space sequence need not preserve globally bounded Hankel rank.For blocks contained within context, every candidate has Hankel rank at most r, supporting the lower bound across exponential, diagonal, and general linear states.Source: §A.3 · p. 14

A.5 GAUSSIAN FIRST-CHAOS PROJECTION

  • First-chaos projection turns a bound on the linear part of a nonlinear predictor into a lower bound on full risk, because higher chaoses only add error.The projection bound does not imply that higher-chaos components are small.Source: §A.5 · p. 15

B.1 COMPARISON WITH EXISTING APPROXIMATION METRICS

  • The paper’s forecast-risk metric optimizes over context as well as state for the infinite predictive law, unlike fixed-horizon kernel approximation.A finite-window construction need not control risk because the unseen tail still contributes.Source: §B.1.2 · pp. 15–16, §B.1.3 · p. 16

B.3 POSITIVE EXPONENTIAL QUADRATURE

  • Positive exponential quadrature constructs finite-context approximations to algebraic memory, and balancing approximation and truncation errors yields a root-exponential risk curve.For sufficiently large r, risk is at most Cup e^(−cup √r), so reaching τ needs at most Cmode log2(Cup/τ) modes.Source: §B.3 · pp. 16–17, §B.4 · p. 17

B.8 PROOF OF THM. 1

The proof establishes matching root-exponential forecast-risk bounds for finite-state approximations, then extends the same resource exponent to an analytic class of algebraic Laplace mixtures.

B.8 PROOF OF THM. 1

  • For the canonical target, the lower bound holds for every finite-context r-state kernel, yielding best risk e^(-Theta(sqrt(r))) and r*(tau) = Theta(log^2(1/tau)).The proof handles short contexts by the coefficient tail and longer contexts by a rank-constrained Hankel block.Source: §B.8 · pp. 18–19

B.9 ANALYTIC ALGEBRAIC LAPLACE-MIXTURE EXTENSION

  • The extension assumes positive h with h(0)>0, sectorial and near-origin analyticity, and exponential decay; these conditions support positive modes and the quadrature and lower-bound arguments.Its constructive compression uses dyadic intervals and analytic Gauss–Jacobi quadrature, with node count O((1 + log L + log eta^-1)^2).Source: §B.9 · pp. 19–20
  • Across the analytic algebraic Laplace-mixture class, the same e^(-Theta(sqrt(r))) risk and Theta(log^2(1/tau)) state law hold, so the exponent is not specific to an exact power.The result includes ordinary algebraic memory and the fractional kernel.Source: §B.9 · pp. 19–20

B.10 CURRENT-TOKEN AFFINE-GATING BENCHMARK

The affine-gating benchmark reduces first-chaos memory to an averaged linear transition, while the fractional sections identify the predictive-risk geometry and its finite-context consequences.

B.10 CURRENT-TOKEN AFFINE-GATING BENCHMARK

  • For the current-token affine-gated benchmark, the best m-state risk is bracketed by root-exponential bounds, giving m*(tau) = Theta(log^2(1/tau)); gating does not improve the exponent.A gated linear-attention head with key width dk contributes at most dk linear predictive modes.Source: §B.10 · pp. 20–21

C.1 PREDICTIVE HILBERT-SPACE COMPLETION

  • The predictive Hilbert-space completion defines perturbed predictors as affine objects, since abstract elements need not have convergent coefficient expansions; fractional-class coordinates are recovered by Theorem 2.The predictive seminorm maps Cauchy sequences to Cauchy filters in L2, whose limits are orthogonal to the next innovation.Source: §C.1 · p. 21

C.2 FRACTIONAL INTEGRATION OPERATOR

  • Fractional integration with transfer function (1-z)^-d changes the singular predictive norm into an ordinary Euclidean norm, though the operator is unbounded on l2 for 0<d<1/2.It is a coordinate change rather than a bounded map on l2.Source: §C.2 · p. 21

C.3 FARIMA COEFFICIENTS AND EXACT SINGULAR GEOMETRY

  • For FARIMA(0,d,0), coefficients decay as j^(-1-d), but truncating at context L has predictive error of order L^-1, larger than the unweighted coefficient tail of order L^(-1-2d).The discrepancy follows from singular predictive geometry, so coefficient error alone gives the wrong context law.Source: §C.3 · p. 21

C.4 PROOF OF THM. 2

  • Theorem 2 identifies the fractional predictive inner product through fractional integration, with the pure fractional-noise risk exactly equal to sigma^2 times the squared l2 norm of the integrated error.In this class, predictive-norm convergence also gives canonical l2 coefficient coordinates.Source: §C.4 · p. 22

C.5 FINITE-CONTEXT PREDICTION AND PARTIAL AUTOCORRELATIONS

  • Finite-context prediction is related both to partial autocorrelations through Durbin–Levinson and to Toeplitz determinant ratios, linking the context law to determinant asymptotics.For FARIMA, the partial autocorrelations are available in closed form.Source: §C.5 · p. 22

C.6 PROOF OF COROLLARY 3

The fractional results quantify finite-context costs, parameter misspecification, and state compression. They show how pole structure affects prediction and how fractional strength and location differ statistically.

C.6 PROOF OF COROLLARY 3

  • For one fractional pole, VL - V∞ = V∞ d^2/L + o(L^-1), with V∞ = sigma^2.The proof obtains the leading term from the exact variance product and its asymptotics.Source: §C.6 · p. 23

C.7 MULTIPLE SEPARATED FRACTIONAL POLES

  • For separated fractional poles, the context-risk leading coefficient is the sum of squared strengths; equally spaced seasonal poles multiply the context cost by the period.The general multiple-pole result requires a regular factor suitable for Fisher–Hartwig asymptotics.Source: §C.7 · pp. 23–24

C.8 EXACT EXPONENT MISSPECIFICATION AND RELATIVE ENTROPY

  • For fractional-exponent misspecification, forecast risk depends only on the exponent error, and its relative-entropy rate is exactly half the risk normalized by sigma^2.A shared spectral singularity cancels in the likelihood ratio, leaving the exponent as the quantity distinguishing the models.Source: §C.8 · p. 24

C.9 FISHER INFORMATION AND ROOT-T ESTIMATION

  • Under the stated likelihood regularity conditions, estimating fractional strength has asymptotic variance 6/pi^2 and expected excess risk sigma^2/T + o(T^-1); finite-context cost dominates unless LT grows at least linearly in T.The strength’s forecast curvature is pi^2 sigma^2/6.Source: §C.9 · pp. 24–25

C.10 UNKNOWN POLE LOCATION AND CUSP GEOMETRY

  • Pole-location error has locally linear risk, unlike the quadratic risk for strength, because displacement leaves an uncancelled singularity and introduces a spurious one.Thus pole location is not a regular parameter and the usual T^-1 forecast cost does not apply.Source: §C.10 · pp. 25–26

C.11 FRACTIONAL EXPONENTIAL-STATE LOWER BOUND

  • For fractional noise, the best r-state forecast risk has a root-exponential lower bound, transferring the Hankel obstruction into fractional predictive risk.The conversion uses the fractional-integrated norm to relate coefficient-level obstruction to forecast risk.Source: §C.11 · p. 27

C.12 FRACTIONAL EXPONENTIAL-STATE UPPER BOUND

  • For fixed 0<d<1/2, the fractional kernel admits positive exponential compression with relative coefficient error at most eta over context L.The constructive guarantee uses the analytic Laplace-mixture method; its constant is not claimed uniform as d approaches zero.Source: §C.12 · p. 27

C.13 PROOF OF COROLLARY 4

For fixed fractional strength d, the square-log state-complexity law persists under fractional predictive geometry, while near the short-memory boundary the relevant accuracy scales are d^2 and d^4. The section also develops a uniform intermediate construction and sets out its unresolved regimes, alongside the contraction-based first-chaos argument.

C.13 PROOF OF COROLLARY 4

  • For fixed d, the fractional-risk state complexity remains Θ_d(log^2(1/τ)), despite the change from ordinary coefficient error to fractional predictive geometry.The constants depend on d and are not uniform as d approaches zero.Source: §C.13 · p. 28

C.14 ZERO-MEMORY BOUNDARY AND GAMMA EXPANSION

  • Once target accuracy satisfies τ ≳ σ^2d^2, the zero-output architecture already attains it, requiring no memory; this is the upper edge of the window in Thm. 5.Source: §C.14 · p. 28

C.15 HARMONIC LINEARIZATION OF THE FRACTIONAL COEFFICIENTS

  • The harmonic surrogate’s fractional-risk error is O_d0(σ^2d^4), and below σ^2d^4 it is no longer adequate for the construction.The stated bound is uniform for 0 < d ≤ d0 < 1/2.Source: §C.15 · p. 29

C.16 HARMONIC TAIL IN THE FRACTIONAL PREDICTIVE NORM

  • The harmonic-tail bound is designed to remain uniform as d approaches zero, avoiding constants that blow up in the boundary layer.The proof controls the tail in the fractionally integrated predictive norm.Source: §C.16 · p. 29

C.17 PROOF OF THM. 5

  • In the intermediate regime, the relevant scale is σ^2d^2/τ: the bound degenerates to a constant as τ approaches σ^2d^2 from below.This reflects that the memory being represented has predictive mass of order σ^2d^2, rather than order one.Source: §C.17 · p. 30

C.18 REMAINING DOUBLE-SCALING PROBLEM

  • Two boundary-layer gaps remain: no matching lower law is uniform in d, and the regime τ ≪ σ^2d^4 is not covered by the harmonic construction.The deeper regime requires an exact fractional-kernel construction whose head count is unknown.Source: §C.18 · p. 30

D.1 MODEL CLASS AND STATIONARY SOLUTION; D.2 DIFFERENTIABILITY AND INTEGRABILITY ASSUMPTIONS

  • The recurrence analysis assumes uniform contraction on a forward-invariant state domain, with differentiability and integrability conditions sufficient for Gaussian integration by parts.The authors identify uniform contraction as the substantive hypothesis, not a claim about recurrent models generally.Source: §D.1 · p. 30, §D.2 · p. 31

D.3 GAUSSIAN STEIN IDENTITY; D.4 DERIVATIVE PROPAGATION THROUGH THE RECURRENT STATE

  • Gaussian integration by parts identifies first-chaos memory with averaged output sensitivity to past inputs; each added recurrent-state derivative factor then yields geometric decay at rate ρ.The resulting envelope is |β_j| ≤ MKρ^(j−1) for j ≥ 2.Source: §D.3 · p. 31, §D.4 · p. 31

D.5 PROOF OF THM. 6

Uniformly contractive recurrences have an exponentially decaying first-chaos memory envelope, which yields a necessary accuracy-to-margin condition for algebraic targets. The appendices also connect forecast risk to conditional and finite-sample information, under stated assumptions and scope boundaries.

D.5 PROOF OF THM. 6

  • For lags j ≥ 2, the first-chaos coefficients obey |β_j| ≤ MKρ^(j−1), yielding an exponential envelope under the contraction assumption.The lag-one input also enters the readout directly and is not bounded by the stated derivative budgets.Source: §D.5 · p. 31

D.6 ALGEBRAIC FIRST-CHAOS TAIL LOWER BOUND; D.7 SHARP INVERSION TO THE NEAR-CRITICALITY LAW

  • Using the full algebraic tail beyond its crossing lag yields the risk exponent 2α−1; inverting the crossing condition introduces a logarithmic factor in the contraction margin.The whole tail gives J^(1−2α), stronger than a single-coefficient bound of J^(−2α).Source: §D.6 · p. 32, §D.7 · p. 32

D.8 PROOF OF COROLLARY 7; D.9 WHY THE CONDITION IS NECESSARY BUT NOT SUFFICIENT

  • A recurrence achieving risk τ must have contraction margin at most order τ^(1/(2α−1)) log(e/τ), but this condition is necessary, not sufficient.A small margin does not ensure the right first-chaos profile, and higher-chaos mass can add risk.Source: §D.9 · p. 33

D.10 SCOPE BEYOND UNIFORM CONTRACTION

  • The recurrence result is limited to white-noise input, pointwise uniform contraction, constant derivative budgets, and first-chaos targets.Autocorrelated inputs can carry long memory themselves, while nonlinear targets require a different projection.Source: §D.10 · p. 33

E.1 CONDITIONAL TRAJECTORY KL IDENTITY; E.2 STATIONARY ENTROPY-RATE IDENTITY

  • For finite T, the conditional trajectory KL identity holds when excess forecast risk is finite, including for singular spectra; the stationary-rate identity needs an additional rate-coincidence assumption.A known common singular factor cancels from the likelihood ratio when models differ only in a regular factor.Source: §E.1 · p. 33, §E.2 · p. 34

E.3 CONDITIONAL SCORE AND PREDICTIVE CURVATURE; E.4 LOCAL ASYMPTOTIC MINIMAX TRANSFER

  • Under local asymptotic normality and the stated regularity conditions, the p-dimensional minimax forecast constant is pσ^2/T; for the fractional exponent, the reported risk is σ^2/T + o(T^−1).The fractional result uses the cited efficient asymptotic variance and forecast curvature, which cancel.Source: §E.4 · p. 34

E.5 EXACT ONE-POLE FISHER–HARTWIG DETERMINANT; E.6 TRIGAMMA PROOF OF THE O(d2 log T) REMAINDER

  • For a single fractional pole, the finite-section boundary correction is logarithmic rather than order T, with a uniform O_d0(d^2 log(eT)) remainder.The remainder bound is stated uniformly for |d| ≤ d0 < 1/2.Source: §E.5 · p. 35, §E.6 · p. 35

E.7 DIRECT LOCAL INFORMATION AT d = 0; T. Then DKL(P (T )

  • On the T^−1/2 local scale, the finite stationary block has the same local relative-entropy constant as the stationary conditional experiment; this alone does not establish local asymptotic normality.The calculation shows the Fisher–Hartwig boundary does not alter that local information constant.Source: T. Then DKL(P (T ) · p. 35

E.8 FIXED-Q MOVING AND MERGING FRACTIONAL POLES

The paper extends finite-section information results to a fixed number of moving or merging fractional poles, subject to an integrability constraint. It then records boundaries on pole-number and strength uniformity and gives a secondary multivariate extension of predictive geometry and factorized memory.

E.8 FIXED-Q MOVING AND MERGING FRACTIONAL POLES

  • The single-pole finite-section result extends to a fixed number Q of fractional poles whose locations may move with T and merge.The expansion is uniform over distinct reflection-symmetric locations with the strength vector fixed.Source: §E.8 · p. 36
  • Pole interactions change across separation T^−1: more widely separated poles interact logarithmically, while more closely spaced poles behave as one pole of combined strength.The stated crossover scale is separation of order T^−1.Source: §E.8 · p. 36

E.9 UNIFORM FISHER–HARTWIG INPUT AND THE ROLE OF THE INTEGRABILITY CONSTRAINT

  • The integrability margin both keeps poles and collapsing clusters in the stationary fractional class and enables the negative-exponent Fisher–Hartwig expansion; uniformity is in locations, not strengths.The stated result holds the strength vector fixed.Source: §E.9 · p. 37

E.10 WHAT IS NOT COVERED FOR GROWING Q OR VANISHING POLE STRENGTHS

  • The finite-section theory covers fixed Q, not a number of poles growing with T, and does not provide strength-uniform remainders for vanishing strengths.The authors note that growing-period seasonal fractional models fall outside the stated result.Source: §E.10 · p. 37

F MULTIVARIATE PREDICTIVE MEMORY

  • The multivariate appendix is a secondary extension for matrix-valued predictive memory; the three main resource laws do not depend on it.Its main text is stated for a scalar target.Source: F MULTIVARIATE PREDICTIVE MEMORY · p. 37

F.1 MATRIX-VALUED PREDICTIVE GEOMETRY

  • With a regular bounded spectrum, matrix predictive geometry yields the same downstream mechanism as the scalar theory, with differences in bookkeeping.The stated two-sided spectral bound supports the matrix predictive-risk form.Source: §F.1 · p. 38

F.2 SHARED ROW/COLUMN FACTOR MEMORY

  • Factorized memory shares r temporal laws across variable pairs, while matrices B_q encode cross-variable structure; rank-s factorizations correspond to head width s.The shared-subspace variant uses common input and output subspaces with head-specific cores.Source: §F.2 · p. 38

F.3 FIXED TEMPORAL HEADS

  • When temporal heads are fixed and only their matrices are estimated, the reported excess forecast risk is of order rd^2σ^2/T under the stated conditions.The local parametric law is expressed in terms of the number r of heads and ambient dimension d.Source: §F.3 · pp. 38–39

F.4 GROWING CORE AND SHARED-SUBSPACE LOWER BOUNDS

The section derives matching lower scales for core and shared-subspace resources using packings calibrated by predictive separation and trajectory KL. Together, these bounds establish lower scales for three parameter components, without a separate temporal-parameter packing term.

F.4 GROWING CORE AND SHARED-SUBSPACE LOWER BOUNDS

  • The flat-core construction gives coefficient separation of order rs^2/T and predictive separation of order σ^2rs^2/T, enabling a Fano lower bound.The trajectory KL is bounded by Cρcϵrs^2, so choosing cϵ small keeps it below a small multiple of the packing log-cardinality.Source: §F.4 · pp. 39–40
  • A local shared-subspace perturbation yields coefficient and predictive separation of order s(d −s)/T, with trajectory KL calibrated for Fano’s inequality.The construction uses a one-active-lag subclass and remains within the stability class under the stated sample-size condition.Source: §F.4 · pp. 40–41
  • The core, output-subspace, and input-subspace packings yield lower-bound orders rs^2/T, s(d −s)/T, and s(d −s)/T, respectively.Their maximum dominates a third of their sum under condition (86); no independent temporal-parameter packing supplies an additional rpθ term.Source: §F.4 · p. 41
  • The lower-bound construction’s flatness condition improves the threshold to condition (86), compared with a naive threshold of order r^2s^2.The results are nonasymptotic same-realization statements that allow d, r, and s to grow.Source: §F.4 · p. 41

F.5 LEARNED SMOOTH TEMPORAL HEADS ON A REGULAR CHART

This section studies learned smooth temporal heads on regular predictive charts, giving finite-sample forecasting guarantees and clarifying how whitening, head crowding, singular parameter strata, and architectural resources shape their scope and approximation error. Its experiments control parameter budgets and gated-state contraction to isolate memory capacity and evaluate first-chaos memory.

F.5 LEARNED SMOOTH TEMPORAL HEADS ON A REGULAR CHART

  • The sample condition retains Λ^2 dependence: Λ = O(1) for structured spans with bounded ℓ1/ℓ2 ratio, but may grow with L for long-memory or power-law heads.Uniformly exponentially localized spans are among the structured cases with bounded ratio.Source: §F.5 · pp. 41–42
  • On uniformly regular predictive charts, ERM satisfies a nonasymptotic forecast-risk guarantee when n ≥ CΛ^2(p + x), with image-geometry assumptions invariant to reparameterization.The entropy condition applies to the normalized base used in the concentration argument, while the transfer envelope controls an additional difference closure.Source: §F.5 · pp. 41, 43
  • For bounded-head exponential charts, pexp = 2s(d − s) + rs^2 + r and Λ = O(1), giving no extra context-length factor and matching parametric risk order on a nondegenerate patch.The matching lower scale is established for bounded r and n ≍ T; uniformity as r grows is not claimed.Source: §F.5 · pp. 44–45

F.6 DEGREES OF FREEDOM 2s(d −s) + rs2 + rpθ

  • At locally identifiable regular points, shared-subspace dimension is 2s(d − s) + rs^2 + rpθ and governs the regular-chart upper law, but no temporal-parameter minimax packing is claimed.The count includes both Grassmannians, the cores, and learned temporal shapes; the intrinsic dimension can change on singular strata.Source: §F.6 · pp. 45–46

F.7 WHITENING TEMPORAL SPANS AND RAW GRAM CROWDING

  • Whitening makes the temporal-span map isometric, so frequency leverage—not the raw Gram condition number—is statistically meaningful; nevertheless, compact smooth head families crowd as r grows.The smallest eigenvalue of the normalized Gram matrix tends to zero as r grows, making raw parameterizations singular.Source: §F.7 · p. 46

F.8 HEAD COLLISIONS, VANISHING CORES, AND SINGULAR STRATA

  • Head collisions, vanishing cores, or gauge-related Jacobian rank loss define singular strata where minimax behavior remains open and the regular-chart dimension count does not apply.The theorem assumes uniform separation from these degeneracies.Source: §F.8 · p. 46

F.9 RELATION TO LOW-RANK VAR AND TUCKER TIME-SERIES MODELS

  • Using too few temporal heads leaves residual error governed by the mode-three unfolding, while too small a shared spatial subspace leaves mode-one and mode-two residuals under the matrix regular-spectrum condition.These mismatch results identify which tensor unfolding controls each architectural resource in forecast risk.Source: §F.9 · pp. 46–47

G.1 ARCHITECTURE DEFINITIONS

  • The benchmark defines a noiseless L = 64 Gaussian target and compares a diagonal m-mode state-space model, a bounded gated recurrence, and a causal Transformer.The recurrence is affine in its previous state, not a genuinely nonlinear-in-state model; the Transformer has two layers, width 32, four heads, and feedforward width 64.Source: §G.1 · p. 47

G.2 WIDTH-INDEPENDENT DERIVATIVE AND AMPLITUDE BUDGETS

  • Width-independent caps constrain recurrent parameter blocks at every step, preventing wider models from reducing error through larger weights rather than more modes.The effective budgets are ∥B∥≤2, ∥C∥≤2, gate and value norms at most 2, and input scale at most 4; contraction ρ is fixed, not learned.Source: §G.2 · p. 47

G.4 TRAINING, VALIDATION, AND SEED PROTOCOL

  • Model selection uses the validation split, while all reported numbers are on the test split; width and stability sweeps use seeds 0 through 3, and attention references and exponential controls use seeds 0 through 2.Source: §G.4 · p. 48

G.5 EXACT SSM IMPULSE RESPONSE AND FIRST-CHAOS ESTIMATION

  • State-space first-chaos memory is read exactly from its impulse response, while gated and attention coefficients use Monte Carlo estimates; fitted log-log slopes are descriptive finite-window summaries.Nonlinear coefficients use an independent diagnostic sample of 200,000 Gaussian windows, and slopes regress log |βj| on log j over lags 3–30; exponential decay need not have a constant log-log slope.Source: §G.5 · p. 48

G.6 ALGEBRAIC TARGET RESULTS

In the algebraic-target benchmark, attention recovers the target exponent while contractive recurrent models decay faster. Width increases alone leave error floors, whereas moving toward the stability boundary substantially improves the diagonal state-space model.

G.6 ALGEBRAIC TARGET RESULTS

  • Attention's fitted first-chaos slope is −1.110 against true exponent −1.1, while diagonal state-space and gated recurrence slopes are −2.26 and −2.56.Attention therefore recovers the algebraic law almost exactly, while both contractive recurrences decay faster than the target.Source: §G.6 · p. 48

G.7 MATCHED EXPONENTIAL-TARGET SPECIFICITY CONTROL

  • The matched exponential controls use β = 0.8 at the contraction boundary and β = 0.9 beyond the contraction envelope, with budgets, optimizer, and splits held fixed.Reported comparisons use held-out error divided by target variance; the controls use three matched seeds and the algebraic sweep uses four.Source: §G.7 · p. 48

G.8 WIDTH SWEEP

  • At m=128, diagonal state-space unexplained variance fractions are 6.8 × 10−6 on the boundary-matched control, 1.1 × 10−2 on the algebraic target, and 4.5 × 10−2 on the slower exponential.The gated recurrence reaches 2.0 × 10−3 on the boundary-matched control and 1.4 × 10−2 on the algebraic target; curves improve at small widths, then plateau above zero, so width alone does not remove the floor at fixed contraction and derivative budgets.Source: §G.8 · pp. 48–49

G.9 STABILITY SWEEP

  • Across ρ=0.5 to 0.995, diagonal state-space unexplained variance falls from 6.2 × 10−2 to 7.5 × 10−6; the gated recurrence falls from 6.8 × 10−2 to about 2.0 × 10−3, then plateaus.At the largest ρ, the contraction timescale exceeds L=64, so the sweep does not directly test the untruncated asymptotic tail; the gated floor may also include higher-chaos variance.Source: §G.9 · p. 49

G.10 COMPLETE NUMERICAL TABLES

  • The supplement provides numerical summaries for both figures; Hankel singular values are truncated at a machine-precision threshold, and fractional prediction gaps use a cancellation-free suffix sum with a leading-tail correction.The archived Fig. 2 summary records seed means, standard errors, first-chaos slopes, target variances, and seed lists for each configuration.Source: §G.10 · p. 49

Related papers

Loading 2610.08816v1…