PaperlayerView exact revision on arXiv

Source-linked AI summary

Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models

Hung Phan, Thuy T. Nguyen, Minh Ngoc Dinh, Nhat-Quang Tran

arXiv:2609.39445v1cs.LGstat.ML

TL;DR

A fixed adapter head must handle heterogeneous time-series regimes, while hidden-state MoE routing can collapse when instance normalization removes regime statistics. The paper routes on raw inputs while experts use frozen TSFM representations; RR-MoA wins 54/54 primary-benchmark comparisons, and frozen RR-MoA outperforms full fine-tuning in the reported setting.

  • Problem

    A single fixed adapter does not specialize across heterogeneous regimes, and instance-normalized TSFMs can lose routing information when standard MoE routers read hidden states.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    RR-MoA routes each window from its raw, pre-normalization input while adapter experts consume hidden states from a strictly frozen TSFM.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    RR-MoA wins 54/54 primary-benchmark cells, and the conclusion reports frozen RR-MoA as strongest on every (µ, σ)-stripping backbone while full fine-tuning loses by 12–79%.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The training-free R(D) signal ratio predicts collapse risk and is proposed as a pre-training go/no-go test, with Traffic (R=0.14) predicting no improvement.

    Source: §1 · p. 2, §5 · p. 9

    The evaluated setup uses a frozen pretrained feature extractor and trains only an adapter with at most 500K parameters for forecasting or imputation.

    Source: §3 · p. 3

Abstract

from arXiv · show

Time series foundation models (TSFMs) commonly adapt to new data by attaching a single trainable head to a frozen backbone, a one-size-fits-all setup that underfits heterogeneous regimes. Replacing the head with a mixture of experts is the standard upgrade, but on instance-normalized backbones (the dominant TSFM design class) it fails: routing entropy collapses to zero and one expert absorbs every input, a failure we call normalization-induced routing collapse. Standard MoE rescue mechanisms do not repair it, because the cause is in the router's input, not its optimization. Pre-encoder normalization strips the statistics a router would need to tell regimes apart. A mutual-information decomposition makes this precise and yields a signal-ratio that, computed before training, predicts dataset vulnerability (Spearman $\rho = -0.88$). Eight causal controls, including a vision-modality replication, isolate instance normalization as the cause. The prescription is a minimal causal intervention: Raw-Routed Mixture of Adapters (RR-MoA), which routes on the raw, pre-normalization input. Under a strictly frozen backbone, RR-MoA wins 54/54 comparisons against the strongest fixed adapter and significantly outperforms LoRA, TRACE, AdaMix, and full fine-tuning. The effect generalizes across six backbones and an imputation task. Frozen RR-MoA also beats full fine-tuning by 12-79% (the Frozen Paradox); two architecturally distinct variants confirm the principle generalizes beyond this specific router.

1 Introduction

A single adapter head cannot specialize to heterogeneous time-series regimes, while standard MoE routing collapses on instance-normalized TSFMs. The paper traces this failure to information removed before routing and proposes routing on raw inputs instead.

1 Introduction

  • Instance normalization removes per-window statistics before routing, causing gating mass to become one-hot and leaving other experts without gradients.Eleven optimization-side rescues close only 10.9% of the gap that an input-side fix closes.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • RR-MoA changes only the router input: it routes on raw, pre-normalization windows while keeping the backbone strictly frozen.The authors report that naïvely re-injecting mean and variance worsens MSE by 42–239%, with 11/18 cells re-collapsing.Source: §1 · p. 2
  • The Frozen Paradox prediction is that unfreezing serves only the live expert; empirically, freezing wins on 4/6 datasets and unfreezing never recovers more than 13%.The paper motivates adaptation because a uniform head must handle every window despite differing regime-specific signals.Source: §1 · pp. 1–2
  • R(D) predicts collapse risk across six backbones and a vision modality, with Spearman ρ=−0.88 (p<0.002), and abstains on Traffic (R=0.14).The training-free signal ratio is derived from a mutual-information decomposition of the information removed before routing.Source: §1 · p. 2
  • RR-MoA wins 54/54 primary-benchmark cells, beating LoRA, TRACE, AdaMix, and full fine-tuning at p<0.001.The reported MSE improvements are 26–79%.Source: §1 · p. 2

2 Related Work

The paper positions RR-MoA across TSFM adaptation, MoE design, routing-collapse research, and time-series normalization. It distinguishes its contribution as per-window adapter specialization whose router receives information preserved before normalization.

TSFM adaptation and MoE design

  • Prior TSFM adaptation work varies finetuning strategies, backbones, or per-task adapter topology, but the authors know of no prior per-window mixture over topologically distinct adapters.Existing MoE scaling and in-backbone TS-MoE work instead acts on token routing within the encoder.Source: §2 · p. 2

MoE routing collapse and rescue

  • Existing collapse diagnoses target optimization, representation geometry, or routing paradigms, whereas this setting lacks routing information in the input before those mechanisms act.The cited alternatives include load balancing, z-loss, hyperspherical scores, sub-token routing, expert-choice, soft mixtures, and dense-gradient methods.Source: §2 · p. 3

Normalization in Time Series

  • Prior time-series normalization work recovers stripped statistics in various ways, while this paper identifies a distinct MoE consequence of instance normalization under unfreezing.The reported consequence differs from the optimization-induced collapse addressed by standard MoE rescues.Source: §2 · p. 3

3 Raw-Routed Mixture of Adapters (RR-MoA)

RR-MoA separates the information needed to route from the representation used by adapters: a raw-input gate selects experts that consume frozen TSFM hidden states. Its design addresses the loss of routing information caused by instance normalization.

Task setup

  • The task setup freezes the pretrained feature extractor and trains only an adapter with at most 500K parameters for forecasting or imputation.The architecture also uses distinct expert topologies; five identical experts degrade MSE by 8–15%.Source: §3 · pp. 3, 5

Problem statement

  • The standard hidden-state MoE recipe loses the per-window first- and second-order statistics that distinguish regimes, so its router collapses to one expert.The remaining experts receive no gradient; the authors name this failure normalization-induced routing collapse.Source: §3 · p. 3

Architecture

  • RR-MoA wins 54/54 comparisons against the strongest fixed adapter, using a raw-input gate while experts consume frozen TSFM representations.Figure 2 depicts the two paths: a frozen TSFM produces hidden states for experts, and a Conv1d gate reads raw input to select Top-2 experts.Source: §3 · p. 3, Figure 2 · p. 4

Mechanism

  • The routing-information decomposition identifies the loss as I(X; E) − I(S; E) = I(M, Σ; E | S) ≥ 0, where normalization retains shape S and discards mean M and scale Σ.The lower bound also accounts for residual leakage of stripped statistics through shape, without assuming independence.Source: §3 · p. 4
  • R(D) is a training-free monotone proxy for routing information in stripped statistics, and its correlation with raw-routing benefit and boundary cases are evaluated in §4.2.The supplied method text describes the proxy as high when stripped statistics carry routing information and low otherwise.Source: §3 · p. 4

Architecture

  • A lightweight Conv1d gate reads raw time series while the frozen TSFM supplies hidden states to topologically distinct adapter experts.With K=5 and k=2, only two experts execute per sample, using 40% of dense expert FLOPs while retaining 63% of dense MSE improvement over the best fixed adapter.Source: §3 · p. 4

4 Experiments

Experiments show that raw-input routing outperforms fixed adapters and other tested methods, while hidden-state routing collapses on instance-normalized backbones. Causal controls, a training-free signal ratio, and cross-backbone and task tests support the proposed diagnosis and its limits.

4.1 Setup

  • The evaluation uses six LTSF benchmarks, forecasting at H=96 as the primary task, 20%-masked imputation, and six backbones spanning three normalization regimes.Core results use five seeds and ablations use three; significance tests are Wilcoxon signed-rank with Bonferroni correction.Source: §4.1 · p. 5

Standard MoE collapses; RR-MoA wins 54/54 vs fixed adapters

Optimization-side rescues fail; raw-input fixes transfer

  • Five optimization-side rescue families recover at most 10.9% MSE, while tested raw-input alternatives support the normalization-signal diagnosis.SR-MoA outperforms RR-MoA by 13–42% on six datasets; eight causal controls identify stripped per-window (µ, σ) as the common mechanism.Source: Optimization-side rescues fail;… · p. 8, Table 3 · p. 9

RR-MoA leads the baseline suite and transfers across backbones, horizons, and tasks

5 Conclusion and Limitations

The conclusion identifies normalization-induced routing collapse and presents raw-input routing as a minimal fix, with reported gains across benchmarks and variants. It limits the claim to statistics-stripping normalizers and names several tasks and settings left open.

5 Conclusion and Limitations

  • RR-MoA, SR-MoA, and Residual-IA+ support the diagnosis; frozen RR-MoA wins 54/54 comparisons, and full fine-tuning loses by 12–79%.The conclusion reports that eleven optimization-side rescues recover at most 10.9% MSE, versus 54–80% for the input-side fix.Source: §5 · p. 9
  • The reported scope is (µ, σ)-stripping normalizers; classification, anomaly detection, horizons above 720, and learned expert pools remain open.The conclusion also states that the prediction extends to FAN and vision.Source: §5 · p. 9

A Experimental Setup Details

  • Experiments use six specified backbone families, Adam at 10−3, MSE loss, batch size 128, and 15 epochs; core grids use five seeds.Full fine-tuning unfreezes all encoder blocks; the listed baselines include fixed adapters, LoRA, TRACE, AdaMix, and DLinear.Source: A Experimental Setup Details · pp. 16–17

B Deployment-Regime Motivation

  • Three motivating deployment settings are multi-tenant cloud serving, on-device or edge deployment, and cross-task reuse of a frozen backbone.The paper demonstrates forecasting and imputation in the main text.Source: B Deployment-Regime Motivation · p. 17

B.1 Inference Benchmark

The inference benchmark weighs adapter overhead against DLinear’s per-call speed and memory advantages, while describing serving capabilities adapters provide in exchange.

B.1 Inference Benchmark

  • RR-MoA adds +3.1 ms, while dense SR-MoA and Residual-IA+ run faster than RR-MoA Top-2 on GPU.The sparse expert selection loop dominates RR-MoA’s routing overhead; its router adds < 0.1 ms.Source: §B.1 · p. 17
  • DLinear is ∼1400× faster per call (0.04 vs 56 ms) and uses ∼2× less peak GPU memory.For N independent tasks served in isolation, N DLinears are also smaller in aggregate than one TSFM plus N adapters.Source: §B.1 · p. 17
  • All three architectural fixes add only single-digit milliseconds of adapter overhead, preserving the frozen-backbone fast-inference property.Across the measured deployment regimes, the adapters stay within +5.9% of raw-backbone latency and add at most 6 MB of peak GPU memory.Source: Table B.1 · p. 18, §B.1 · p. 18
  • Adapters enable hot-swapping 354–466 K parameters from host RAM in sub-millisecond time and sharing one GPU-resident backbone across tasks.The described capabilities also include zero-shot onboarding with adapter weights trained on related tenants.Source: §B.1 · p. 18

C Multi-Horizon Evaluation

Across the evaluated LTSF horizons, RR-MoA’s improvement over the best fixed adapter persists across datasets and prediction lengths. DLinear serves as a supervised calibration anchor, and its gap narrows at longer horizons on ETTh1 and Weather.

C Multi-Horizon Evaluation

D Proposition Proofs

The appendix formalizes routing information lost under normalization, examines its leakage and predicted effects, and reports a broad LoRA comparison. It also analyzes when backbone co-adaptation drives routing collapse and how raw routing breaks that feedback loop.

D Proposition Proofs

  • The exact decomposition is I(X; E) − I(S; E) = I(M, Σ; E | S) ≥ 0, and a lower bound subtracts leakage ε = I(M, Σ; S) from I(M, Σ; E).Equality in the bound requires I(M, Σ; S | E) = 0; unconditional independence of (M, Σ) and S alone does not guarantee tightness.Source: D Proposition Proofs · p. 19
  • The Gaussian leakage estimate ranges from 0.09 nats on Weather to 1.26 nats on Exchange; KSG estimates agree within ±0.26 nats across nine datasets.The canonical-variate projection can discard nonlinear dependence, so the KSG cross-check does not measure the full leakage; direct H(E | S) measurement corroborates the conclusions.Source: D Proposition Proofs · p. 19

D.1 Theoretical Depth of Observation 1

  • R(D) is a training-free monotone proxy for routing-relevant location-scale variance, while its −0.88 correlation with collapse risk is empirical evidence rather than a direct consequence of the Gaussian-MI formula.The rate-distortion interpretation treats normalization as lossy compression and routing distortion as I(M, Σ; E | S), under stated Gaussian and soft-routing assumptions.Source: §D.1 · p. 20, D Proposition Proofs · p. 20
  • At α = 1, routing loses its location-scale signal and the softmax saturates; the predicted soft onset α* instead marks continuous degradation before that discontinuity.For median R(D) = 0.67, the predicted onset is α* ≈ 0.40; the five-point sweep is too coarse to distinguish alternate variance conventions.Source: §D.1 · pp. 20–21
  • The dose-response shows +5.4% MSE degradation for α ≤ 0.75 and +283% at α = 1.0, while routing entropy stays in [1.43, 1.57].The account connects the routing loss to an information bottleneck and notes a parallel InstanceNorm1d effect in ViT patches, but not ResNet spatial features.Source: §D.1 · p. 21, Figure D.1 · p. 21
  • With an unfrozen backbone, co-adaptation can reinforce the favored expert; freezing removes that feedback, though routing can still collapse if one expert is uniformly better.Routing on raw input breaks the feedback loop by making the router independent of the backbone parameters; the distinct expert heads support specialization across windows.Source: §D.1 · p. 22

E LoRA Sweep (Full Results)

  • RR-MoA Top-2 beats the best LoRA configuration by 67.2% on ETTh1, 67.4% on ETTm1, and 107.6% on Weather.The strongest LoRA configuration was selected by enumerating 12 configurations per dataset.Source: E LoRA Sweep (Full Results) · p. 23
  • Across the LoRA sweep, the best head type depends on dataset, rank does not monotonically help, and several Weather MLP runs converge to identical values.The sweep covers 12 configurations per dataset, varying rank, target projections, and forecast head; RR-MoA allocates routing capacity per window rather than using a static head.Source: E LoRA Sweep (Full Results) · p. 23

F Routing Ablations

These routing ablations test what information the router needs and whether alternative routing designs improve on the default. Across the tests, raw-input access matters, while the useful signal and the value of design changes vary by setting.

F Routing Ablations

  • On ETTh1, routing partitions align with amplitude and volatility, while RR-MoA wins all four amplitude quartiles across nine dataset-seed pairs.The largest gain is 4.6× on quiet Q1 windows (0.24 vs 1.09); the smallest is 1.3× on Q4 (1.54 vs 1.97).Source: F Routing Ablations · p. 23, Figure F.2 · p. 25
  • 60–88% MSE degradation follows replacing the raw input with the RevIN-normalized signal in the rawness-versus-bypass control.The control keeps the rest of the setup fixed.Source: Table F.1 · p. 24
  • SR-MoA outperforms RR-MoA by 13–42% across six datasets, while uniform-gradient unfreezing degrades MSE by 35–100% on three datasets.SR-MoA uses per-expert sigmoid gates on raw inputs; RDGF's result is consistent with the Frozen Paradox being structural.Source: Figure F.1 · p. 25, F Routing Ablations · p. 27
  • +48% average MSE degradation results from routing on only [µ, σ]; temporal shuffling stays within ±3% on four of six datasets, consistent with distributional statistics carrying most tested routing signal.The shuffle also improves Weather by 8.5% and ETTm2 by 16.0%.Source: F Routing Ablations · p. 27
  • FFT features degrade MSE by 6% on Weather and 22% on ETTm1; SAIB raises routing entropy without improving accuracy, and IA-Gating wins on four of six datasets.IA-Gating loses on Weather and ETTh1; learned gates cluster near 0.48, consistent with a roughly constant backbone contribution.Source: F Routing Ablations · p. 27

F.1 Statistics Re-Injection Ablation

  • Re-injecting (µ, σ) into mean-pooled hidden states leaves MSE 42–57% worse than raw routing even when routing entropy is not fully collapsed.Across the table's 18 runs, re-injection degrades MSE by 42–239%, and 11 runs re-collapse.Source: §F.1 · p. 27, Table F.3 · p. 28

G Extended RR-MoA Freeze Grid

  • The extended freeze sweep completes the 54-cell comparison; on Solar, last-4 unfreezing reaches zero routing entropy and degrades MSE by 104.5% versus the best fixed baseline.On Exchange, collapsed AdaMix improves over the best fixed adapter but RR-MoA still performs 2.5× better (1.531 vs 3.851).Source: G Extended RR-MoA Freeze Grid · p. 28, Table G.1 · p. 28, Table G.2 · p. 28

G.1 Rescue-Baseline Sweep Details

  • Across 720 rescue-sweep runs, no tested MoE rescue mechanism recovers routing to within 2.7× of RR-MoA.The sweep covers five rescue families across 12 configurations.Source: Table G.3 · p. 29, §G.1 · p. 29

G.2 AdaMix Implementation Details

The AdaMix setup tests whether standard MoE safeguards address collapse, while further controls examine raw routing and other normalization choices. The reported results associate collapse with instance-level normalization that removes routing-relevant statistics.

G.2 AdaMix Implementation Details

  • With AdaMix's hidden-state router, load balancing does not prevent collapse under unfreezing; disabling RevIN restores routing entropy without routing regularization.The collapsed entropy is 0.000 with load-balancing coefficient 0.01; without RevIN it is 0.66–1.32.Source: §G.2 · p. 29, Figure G.1 · p. 29
  • Replacing AdaMix's hidden-state router with a raw-input router recovers routing entropy and MSE on all 12 tested cells.The expert heads and training loop remain unchanged.Source: Table G.4 · p. 30, §G.2 · p. 30

H Normalization Generalization

  • RevIN, BatchNorm1d, and GroupNorm each collapse routing entropy to ≤0.004 under unfreezing, while removing normalization keeps entropy stable at 0.817.The reported mechanism is that these instance-level normalizers remove per-window statistics before encoding; a vision control shows the effect depends on whether stripped statistics carry routing signal.Source: H Normalization Generalization · p. 30, Table H.1 · p. 30

H.1 Cross-Modality Control: Vision MoE

The vision control tests whether normalization-induced routing collapse extends beyond time series. Its contrasting ResNet and ViT results support a modality-general account conditioned on whether normalization strips routing-relevant statistics.

H.1 Cross-Modality Control: Vision MoE

  • InstanceNorm2d prevents collapse on ResNet-18, whereas InstanceNorm1d collapses ViT-B/16 routing to 0.000 ± 0.000.Without normalization, entropy is 0.184 ± 0.260 on ResNet-18 and 0.494 ± 0.130 on ViT-B/16.Source: Table H.2 · p. 31, §H.1 · p. 31
  • The reported contrast tracks which statistics are stripped: ResNet spatial patterns appear to carry little routing information, while ViT patch-level statistics carry routing information.InstanceNorm2d preserves spatial routing patterns in the tested ResNet control; InstanceNorm1d removes per-patch mean and scale in ViT.Source: §H.1 · p. 31
  • The authors frame the collapse mechanism as modality-general, with normalization's effect depending on overlap between stripped statistics and the routing signal.The control uses CIFAR-10 and five architecturally diverse expert classifiers.Source: §H.1 · pp. 30–31

I Top-k Sparse Routing Ablation

The ablations characterize Top-2 routing, expert-pool composition, and expert count. They find that routing contributes substantially over a single adapter, while compute overhead is small and expert-pool diversity is useful across the benchmark.

I Top-k Sparse Routing Ablation

I.1 Expert Pool Diversity Ablation

  • Identical-expert pools degrade performance by 10–29% on ETTh1 and ETTm1, while improving Weather by 9–16%; the diverse pool is the safer benchmark-wide default.Conv1d on ETTh1 is an exception and matches the diverse pool.Source: §I.1 · p. 32, Table I.2 · p. 32

I.2 Expert Count Scaling

  • K=1 is 57–107% worse than K=5, while K=2 is best among K≥2 and K=3–10 remain within ±10% of K=5.With K=2, both experts are always active, avoiding cold-start from rarely activated experts.Source: §I.2 · p. 32, Table I.3 · p. 32

J Imputation Results

Across imputation and forecasting, raw-input routing retains advantages where normalized hidden-state routing collapses, while measurements and controls identify when that advantage should transfer. The results also show that backbone normalization and encoding can limit the information available to frozen adapters.

J Imputation Results

  • RR-MoA wins 7/8 imputation datasets, improving MSE by 21–79%; hidden-state AdaMix collapses in 16/18 runs and has 40–79% worse MSE.The imputation task reconstructs 20% randomly masked values; the dominant AdaMix expert receives 83–99.9% of routing weight.Source: J Imputation Results · pp. 32–33, Table J.1 · p. 33

K Learnable Normalization Coefficient

  • All 30 learned-coefficient runs converge to α<0.5, supporting the prediction that raw input carries more routing information than normalized input.The converged per-dataset means span 0.459–0.482, so the result is a consistent preference for raw input rather than a strong dataset-specific coefficient.Source: K Learnable Normalization Coefficient · p. 33, Table K.1 · p. 33

L Exact MI Loss and Bound Tightness

  • R(D) predicts raw-routing benefit (Spearman ρ=−0.88), whereas H(E | S) does not (ρ=−0.12); the measured bound gap averages 0.075 nats.Synthetic Gaussian-mixture validation finds the bound tight at ρMS=0, but vacuous as ρMS increases, while H(E | S) remains informative.Source: L Exact MI Loss and Bound Tightness · p. 34, Table L.1 · p. 34

M Information Content Diagnostic

N Cross-Backbone RR-MoA

  • Across five additional backbones, RR-MoA wins every tested configuration with per-window normalization, while LayerNorm-only Chronos and Timer-XL show no collapse.On Moirai-MoE, RR-MoA wins all six datasets by 37–90%; on Timer-XL, AdaMix wins 5/6 when frozen, and an independent ensemble beats RR-MoA on three extended datasets.Source: N Cross-Backbone RR-MoA · pp. 35–36, Table N.1 · p. 35, Table N.2 · p. 35

O Extended Full Fine-Tuning Ablation

The ablations examine how normalization, information loss, and raw-input or residual-branch designs relate to performance gaps, then assess Residual-IA+ and SR-RIA+ across datasets, backbones, and forecast horizons.

O Extended Full Fine-Tuning Ablation

  • The strongest extended full-fine-tuning schedule improves only 4–5% over the 15-epoch baseline, while frozen RR-MoA retains a 33–41% advantage.A 90-run, 100-epoch sweep also leaves a 51–71% gap to frozen RR-MoA; its best per-dataset results are worse than the 50-epoch reference.Source: O Extended Full Fine-Tuning Ablation · p. 36, Table O.1 · p. 37
  • Disabling RevIN improves full-fine-tuning MSE by 54–61%, identifying RevIN rather than the unfreezing protocol as the source of the Frozen Paradox.MSE changes are ETTh1 (1.063 → 0.490), ETTm1 (0.910 → 0.364), and Weather (0.481 → 0.187).Source: O Extended Full Fine-Tuning Ablation · p. 37, Table O.2 · p. 37
  • Extending frozen RR-MoA adapter training from 15 to 50 epochs improves MSE by 8–14% but does not close the supervised gap.The passage attributes the remaining gap to information discarded by backbone normalization, which additional adapter training cannot recover.Source: O Extended Full Fine-Tuning Ablation · p. 37, Table O.3 · p. 37

P Closing the DLinear Gap

  • Dual-stream experts reduce the average DLinear gap from +57% to +5% and beat DLinear on Weather (0.198 vs 0.208, p=0.03).Learned αk ≈0.49 indicates complementary backbone and raw-signal information on nonlinear-dynamics datasets, while raw-only MoE is comparable or better on linear-dominated datasets.Source: P Closing the DLinear Gap · p. 38, Table P.1 · p. 38
  • Gap-closing performance is dataset-dependent: multi-resolution reaches +7.7% on ETTh2, while FiLM remains +48–147% and does not recover the full raw signal.Dual-stream reaches +8% on Electricity and +26% on ETTm2; multi-resolution reaches +12% on ETTm2 and +13% on Electricity.Source: P Closing the DLinear Gap · p. 38, Table P.2 · p. 38

P.1 Residual-IA: Closing the Remaining Gap

  • With symmetric initialization, residual gates plateau at ≈0.5; combining a residual formulation, b=−2 gate initialization, and five-epoch raw-branch warmup yields 4/6 match-or-beat.Each fix alone produces only 0–2 wins; their combination yields two wins, two parity results, and one marginal result.Source: §P.1 · pp. 40–41, Table P.4 · p. 40

Residual-IA+: three orthogonal levers close the gap

  • At H=96, Residual-IA+ matches or beats DLinear on 5/6 datasets, with three significant wins and a −2.8% mean gap.The significant wins are ETTm2 (−10.8%), Weather (−5.1%), and Electricity (−1.7%); ETTh1 remains +2.2%.Source: Table P.5 · p. 42, Residual-IA+: three orthogonal levers… · p. 42
  • 107/123 comparisons (87%) match or beat DLinear across six backbones, six datasets, and four horizons; Residual-IA+ also reaches 6/6 at H=192.It records 65 significant wins versus 11 losses, and MOMENT-large reaches 23/24 match-or-beat.Source: Residual-IA+: three orthogonal levers… · p. 43
  • 87/99 pooled comparisons are wins for Residual-IA+ across the five-backbone, multi-horizon grid.MOMENT-large wins 23/24 comparisons, Timer-XL 12/12, Moirai-MoE 15/15, Chronos 22/24, and Moirai 15/24.Source: Table P.8 · p. 44, The recipe is not RevIN-specific:… · p. 44
  • At H=96, all 27 tested cells are wins; Moirai is weakest at 15/24, with ETTh2 long-horizon instability and Electricity losses.The recipe transfers across five backbones spanning learnable-affine RevIN, RMSNorm-internal with non-learnable I/O scaling, and LayerNorm/RMSNorm-only without instance normalization.Source: The recipe is not RevIN-specific:… · p. 44

P.2 SR-RIA+: Self-Routed Residual-IA+

  • SR-RIA+ combines per-expert sigmoid self-gates on raw input with Residual-IA+’s shared NLinear raw branch and gated backbone residual.Each expert outputs NLinear(Xraw) + σ(blendk(Xraw)) · Adapterk(H).Source: §P.2 · p. 44
  • 6/6 datasets achieve match-or-beat for frozen SR-RIA+ versus DLinear, including wins of −0.4% on ETTm1, −6.0% on ETTm2, and −7.5% on Weather.The other three datasets are within +1.6%.Source: Table P.9 · p. 45

P.3 Pure Raw-MLP MoE Ablation: Quantifying the TSFM’s Contribution

The backbone-free control shows that the TSFM’s added value varies by dataset, while removing it shifts routing toward near-uniform ensembling. The checklist records support for the paper’s main claims and explicit scope limits.

P.3 Pure Raw-MLP MoE Ablation: Quantifying the TSFM’s Contribution

  • −1.5% mean delta to Dual-Stream; the Raw-MLP MoE beats it on 3 of 6 datasets.It matches Dual-Stream within ±7% on every dataset; it loses by 3–7% on ETTh1, ETTm1, and Weather, but wins on ETTh2, ETTm2, and Electricity.Source: Table P.10 · p. 45, §P.3 · p. 45
  • 1.49 mean routing entropy (93% of the uniform maximum) makes the backbone-free Raw-MLP MoE behave like an ensemble, versus 1.34 with the backbone.Removing the backbone shifts entropy toward uniform mixing across all six datasets; with the backbone, the router makes substantively non-uniform decisions.Source: §P.3 · pp. 45–46
  • On ETTh1, ETTm1, and Weather, the TSFM contributes 3–6% MSE; on DLinear-near-optimal datasets, it contributes no measurable value.The backbone-free Raw-MLP MoE wins by 0.7–17.9% over Dual-Stream on ETTh2, ETTm2, and Electricity.Source: §P.3 · p. 46

NeurIPS Paper Checklist

  • The checklist identifies uncharacterized extensions to attention-based or learned normalizers and defers anomaly detection, classification, and irregularly sampled tasks.The stated diagnosis covers instance-level normalization, including RevIN, BatchNorm, and GroupNorm.Source: NeurIPS Paper Checklist · p. 47

Related papers

Loading 2609.39445v1…