PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Context Routing Help? A Systematic Study of Multi-Modal Fusion in Time Series Forecasting

Ruizhe Zhou, Gaoyuan Du, Xiaoyang Liu, Haoqi Yao, Deepayan Chakrabarti, Jiating Lin, Yixuan Shen

arXiv:2608.25128v1cs.LGcs.AIstat.AP

TL;DR

The paper addresses whether multimodal forecasting improvements come from context itself or incidental architecture effects. It tests shortcut dominance and residual context informativeness through controlled interventions and diagnostic procedures. Context helps in the tested settings when both conditions hold, while negative evidence is broader than the large positive magnitudes, which come mainly from MoME.

  • Problem

    Reported multimodal forecasting gains vary across datasets and settings, and it remains unclear whether they exploit context or arise from incidental architectural effects.

  • Method

    The paper uses controlled fusion interventions, residual mutual-information permutation testing, shortcut analysis, context corruption, and multi-scale validation to assess context-attributable value.

  • Results

    When both conditions hold, MoME text modulation contributes sizeable MSE reductions; when either fails, contribution collapses to the modulation pathway’s capacity floor.

  • Takeaways & Limitations

    Compute horizon autocorrelation and run a one-minute MI permutation test before training to decide whether to try, skip, or defer multimodal fusion.

  • Takeaways & Limitations

    The residual test conditions only on the latest value, so seasonal or longer-memory series may overstate context value; large positive magnitudes come from one model family.

Abstract

from arXiv · show

Multi-modal time series forecasting methods integrate auxiliary context into temporal predictions through increasingly sophisticated fusion mechanisms. A growing body of work reports substantial gains, yet it is often unclear whether they reflect genuine use of the context or incidental architectural effects. We ask a narrower, checkable question: when can auxiliary context help a forecaster at all? We identify two dataset-level conditions that must both hold: (1) the target is not dominated by a last-value shortcut (low autocorrelation rho_h), and (2) the context carries information about the target beyond history (non-zero conditional mutual information delta; when delta=0 no predictor can benefit---a distribution-free result). Through controlled experiments on MoME (a 14.3B-parameter mixture-of-experts model, 6 datasets, 10 seeds) and four additional fusion mechanisms implemented within a single-backbone testbed (5 datasets), we find that when both conditions hold, text-conditioned expert modulation contributes a sizeable MSE reduction; when either fails, the contribution collapses to the capacity floor of the modulation pathway and carries no context-attributable signal. We establish causality through two interventions: adding a shortcut to MoME suppresses routing contribution by 77-93% across 3 datasets; progressively corrupting context quality drives the context-specific benefit from +44% to negative. We validate the autocorrelation component of our diagnostic on 27 Monash Archive datasets. We provide a calibrated pre-training diagnostic that, on the datasets we test, yields no false positives in well-powered settings. We are explicit about the asymmetry of our evidence: the negative arm is broadly reliable, while the large positive magnitudes come from a single model family (MoME) and are corroborated only in direction by the testbed.

1 Introduction

The paper asks when auxiliary context provides genuine forecasting value rather than architectural side effects. It identifies two conditions—no competing shortcut and statistically informative context—and evaluates them through controlled fusion interventions and a pre-training diagnostic.

  • 1.1 Background and Motivation: Multimodal forecasting gains may reflect architectural effects rather than genuine use of auxiliary context.Reported improvements vary across datasets and settings, motivating a checkable test of context-attributable value.
  • 1.2 The Problem: Two conditions must both hold for auxiliary context to help: no competing shortcut and statistically informative context.The paper treats these as dataset-level conditions for context-attributable improvement.
  • 1.3 Key Findings: A last-value shortcut captures predictable variance on autocorrelated data, suppressing routing contribution by 77–93% across three datasets.The intervention compares the same MoME model with and without the shortcut.
  • 1.3 Key Findings: Context must contain information beyond recent history; a non-significant residual MI test implies no fusion mechanism helps in a well-powered setting.The residual is defined as R = X_t+h − ρ̂_h X_t, and the claim follows from Fact 1 when conditional MI is zero.
  • 1.3 Key Findings: When both conditions hold, text modulation yields sizeable MSE reductions, while failure of either condition reduces contribution to the modulation pathway’s capacity floor.Reported MoME gains include HealthUS +51%, Environment +41.5%, SocialGood +32%, and HealthAFR +29% over 10 seeds.
  • 1.4 Contributions: The study combines causal interventions, a four-mechanism testbed, 27-dataset validation, and a calibrated TRY_FUSION/SKIP_FUSION/INCONCLUSIVE diagnostic.The diagnostic produced no false positives in well-powered tested settings and marked insufficient power as INCONCLUSIVE.
  • 1.4 Contributions: The diagnostic is modality-agnostic in formulation but is evaluated only on embedding-level textual and financial-news context, not image or audio modalities.The paper uses “fusion” in this embedding-level sense.

2 Theoretical Foundation

The paper's diagnostic combines autocorrelation-based shortcut opportunity with conditional mutual information to determine whether context can improve forecasting. A zero conditional mutual information implies, distribution-free, that no predictor can benefit from context, while the Gaussian routing-benefit formula quantifies available improvement but has limited scope.

  • Conditional mutual information δ = I(C; X_t+h | X_t) measures context information about the target beyond the latest history observation.
  • When δ = 0, context is conditionally independent of the target given history, so no predictor can benefit from it.
  • The last-value shortcut's linear MSE is σ^2(1 − ρ_h^2), leaving less residual variance for context as horizon autocorrelation ρ_h approaches one.
  • Under joint Gaussianity, the routing benefit upper bound is σ^2(1 − ρ_h^2)(1 − 2^-2δ), combining residual room with context-explainable information.The first factor is the residual variance left by the shortcut, and the second is the fraction context can explain.
  • The Gaussian routing-benefit expression is not generally a bound off Gaussianity, and its finite-sample plug-in estimate tends to underestimate population RBU.
  • The MI diagnostic estimates residual context information with top-20 PCA components and a nonparametric Kraskov k-nearest-neighbor estimator, then tests significance using 200 context permutations.The estimator is intended to capture nonlinear dependencies that linear diagnostics miss.
  • Small samples or low context diversity can cause false negatives, while single-lag conditioning may overstate context value when longer-memory structure remains in the residual.The test assesses informativeness beyond a last-value baseline, not beyond an optimal history-based predictor.

3 Experimental Setup

The experiments combine a large MoE forecasting model with a controlled single-backbone testbed to isolate context-dependent fusion effects. Evaluations cover diverse datasets, fusion mechanisms, forecasting metrics, and shortcut conditions.

  • The evaluation spans 8 datasets across diverse domains, autocorrelation profiles, and sample sizes, including text context represented by 384-dimensional sentence embeddings.
  • MoME is a 14.3B-parameter mixture-of-experts model whose ablated EiLM pathway applies text-conditioned FiLM-style affine modulation to expert outputs.
  • The routing testbed uses a PatchTransformer backbone with toggleable sparse routing and an optional last-value shortcut to probe mechanisms under controlled conditions.
  • Four fusion mechanisms—Cross-Attention Alignment, Gating, Text-as-Variable, and Output Fusion—are compared using real versus zeroed context within the same backbone.
  • MoME reports point-forecasting MSE and MAE, while the testbed reports probabilistic weighted quantile loss at τ ∈ {0.1, 0.5, 0.9}.
  • Routing contribution is defined so that positive values indicate text modulation improves performance.

4 Results

Across controlled MoME and testbed experiments, context helps only when history does not already provide a dominant shortcut and context contains target-relevant information. Shortcuts suppress routing gains, corrupted context can become harmful, and the diagnostic identifies these regimes with calibrated power handling.

  • Finding 1: 51% routing contribution on HealthUS, with additional gains of +41.5% on Environment, +32% on SocialGood, and +29% on HealthAFR.These gains occur on the identical 14.3B-parameter MoME configuration across datasets.
  • Finding 1: +44% MSE reduction with clean context fell to negative −3 to −5% under 50–75% masking, making partially corrupted text worse than constant text.The model attempts to use partially real-looking entries and is misled by corrupted context.
  • Finding 2: 77–93% suppression reduced MoME routing contributions from +62.9% to +14.1% on HealthUS, +34.3% to +3.3% on SocialGood, and +24.2% to +1.7% on HealthAFR.Suppression was near-total on SocialGood and HealthAFR, but residual benefit remained on moderately autocorrelated HealthUS.
  • Findings 2–3: With shortcuts present, testbed routing benefit was statistically indistinguishable from zero across all datasets; removing shortcuts revealed positive benefit only on MI-significant datasets.On MI-non-significant datasets, mechanisms showed no positive context contribution, and some incurred small negative effects.
  • Finding 4: The pre-training diagnostic produced no false positives in well-powered tested settings and correctly returned INCONCLUSIVE when power was insufficient.Its recommendations use autocorrelation, MI significance, and statistical power before model training.
  • Finding 5: Cross-validated linear context gains were zero or negative even where MoME achieved +51%, indicating that tested context-target relationships were nonlinear.The testbed’s smaller gains were attributed to limited capacity relative to MoME.
  • Finding 5: Spearman r=0.888 (p<0.0001, n=27) linked horizon autocorrelation to shortcut dominance, validating the ρ-based diagnostic component at scale.Monash datasets validate only the autocorrelation component because they lack auxiliary context.

5 Discussion

The discussion frames fusion gains as conditional: simple shortcuts can dominate on highly autocorrelated data, while diagnostic and causal analyses distinguish context value from architectural effects.

  • On FinMultiTime (ρ=0.999), the repeat-last-value baseline outperforms a 14.3B-parameter model, so fusion should be skipped.
  • On HealthUS (ρ=0.77), MoME with text modulation reaches MSE 0.399 versus 0.409 for the naive baseline and 0.817 without text.
  • Two datasets are INCONCLUSIVE because of low MI-test power, reflecting the higher cost of false positives than of one additional verification run.
  • Absolute MSE and MAE values should not be compared across datasets, although within-dataset relative routing contributions remain comparable.
  • The RBU interpretation separates shortcut room from context information, but its closed form is exact only under joint Gaussianity and is not a general bound.
  • Routing benefits are context- and architecture-sensitive: large gains concentrate in MoME, while the testbed confirms direction but not magnitude.

6 Related Work

Related work spans increasingly varied forecasting architectures and fusion mechanisms, but this paper focuses on whether auxiliary context adds value beyond history rather than proposing another fusion method.

  • Time series forecasting architectures: Modern forecasting includes classical, Transformer-based, linear, and foundation-model approaches, with DLinear showing that simple models can match Transformers on many benchmarks.
  • Time series foundation models: Chronos, TimesFM, Moirai, and MOMENT provide strong zero-shot transfer, but these foundation models do not incorporate auxiliary context.
  • Multi-modal time series forecasting: Multimodal forecasting uses output fusion, cross-attention, text-as-variable, gating, and mixture-of-experts routing; this paper characterizes their genuine value versus architectural confounds.
  • Strong simple baselines: The paper extends shortcut-learning comparisons by showing that adding a shortcut suppresses routing contribution and corrupting context degrades it monotonically.
  • Relation to recent strong uni-modal forecasters: Uni-modal forecasting advances remain orthogonal because they improve extraction from history, whereas this work tests whether context adds information beyond history.
  • When does multimodality help?: Compared with Zhang et al., this work adds a mechanistic explanation, causal interventions, and a pre-training diagnostic for condition-dependent fusion benefits.
  • Mutual information estimation: Mutual-information testing uses the Kraskov k-NN estimator with permutation testing, while sparse routing builds on entmax.

7 Conclusion

The conclusion identifies two jointly necessary conditions for useful auxiliary context and supports them with controlled interventions, multi-scale validation, and a calibrated diagnostic.

  • Auxiliary context can help only when no competing shortcut dominates and the context is statistically informative.
  • 77–93%: adding a shortcut suppresses routing contribution across HealthUS, SocialGood, and HealthAFR, while context corruption drives benefit from +44% to negative.
  • The autocorrelation diagnostic component correlates with shortcut dominance across 27 Monash Archive datasets (Spearman r=0.888, p<0.0001).
  • The diagnostic yields no false positives in well-powered tested settings and enables TRY_FUSION, SKIP_FUSION, or INCONCLUSIVE decisions before model training.
  • Practitioners are advised to compute horizon autocorrelation and run a one-minute MI permutation test before investing in complex fusion.

A Proof of Proposition 1 (RBU)

The proof derives the RBU expression under joint Gaussian assumptions by relating conditional mutual information to conditional variances and subtracting the two MMSEs.

  • Under joint Gaussianity, MMSE equals the corresponding conditional variance achieved by the linear conditional expectation.
  • The conditional mutual information is converted into a ratio of prediction conditional variances using the Gaussian entropy–variance identity.
  • RBU is defined as the reduction in minimum achievable MSE when context C is added to the conditioning set.
  • Non-Gaussian behavior: Outside the Gaussian case, the population RBU formula is generally neither an upper nor a lower bound on the true MMSE reduction.
  • Non-Gaussian behavior: The finite-sample Kraskov estimator under-detects nonlinear dependence, so plug-in d_RBU tends to underestimate population RBU; this estimator issue is distinct from formula invalidity.

B.1 MoME Modulation Ablation Detail

MoME’s modulation ablation removes only post-expert text conditioning while leaving text-independent expert routing, weights, and structure unchanged.

  • The –modulation flag removes Expert-level Language Modulation (EiLM), a FiLM-style text-conditioned affine transformation applied after expert routing.EiLM applies γ·x + β to each expert’s output.
  • Expert routing remains text-independent and unchanged when modulation is disabled.The gate, softmax, and top-k routing path is preserved.
  • The ablation changes approximately 74K of MoME’s 14.3B parameters, or less than 0.001%.

B.2 MoME Shortcut Experiment (Finding 2)

Adding a last-value shortcut consistently suppresses MoME’s routing contribution across three datasets, while the near-unit-root FinMultiTime series already favors repeating the last value.

  • 77–93%: adding a last-value shortcut suppresses routing contribution across three datasets.The experiment used three seeds per dataset.
  • HealthUS: HealthUS modulation advantage narrows from MSE 0.334 versus 0.901 without the shortcut to 1.092 versus 1.271 with it.These are modulation-on versus modulation-off MSE values.
  • SocialGood: SocialGood modulation advantage narrows from MSE 0.435 versus 0.662 without the shortcut to 0.874 versus 0.904 with it.These are modulation-on versus modulation-off MSE values.
  • HealthAFR: HealthAFR modulation advantage narrows from MSE 0.610 versus 0.804 without the shortcut to 0.926 versus 0.942 with it.These are modulation-on versus modulation-off MSE values.
  • Negative control: On FinMultiTime, repeating the last value achieves MSE 8.8×10−5 versus 9.1×10−5 for MoME with modulation, leaving routing contribution effectively zero.FinMultiTime has ρ = 0.999.

B.3 Context Degradation Detail (Finding 1)

Context corruption reduces MoME’s routing contribution, and the remaining benefit under constant text is attributable to EiLM capacity rather than context content.

  • 44.2%: real text improves MSE over constant text under identical architecture and parameter count.The comparison isolates context-specific benefit from modulation capacity.
  • At 100% masking, EiLM receives identical text for every sample, reducing its contribution to a learned bias and scale of approximately 74K parameters.The reported +16% residual reflects this capacity benefit, not context exploitation.
  • HealthUS: On HealthUS, MSE rises from 0.399 with real text to 0.715 with constant text at identical architecture and parameter count.
  • Testbed: Effects across the additional fusion mechanisms are generally small, within ±5% in the single-backbone testbed.The testbed result is consistent with the paper’s capacity-related finding.

B.5 Power Analysis Detail (Finding 4)

The power analysis calibrates the mutual-information diagnostic, while broader checks show negative routing contributions when context is uninformative or last-value structure dominates.

  • At n ≥ 500 with unique context per sample, MI-test power is at least 0.6 for δ ≥ 0.5; Type I error remains ≤ 0.07.Power was estimated from 30 synthetic datasets per sample-size/effect-size combination using 200 permutations at α = 0.05.
  • MI-non-significant datasets show no positive contribution, with negative values attributed to instability when attending to uninformative context.
  • Testbed evaluation: Cross-Attention has t = 1.50 with p > 0.05 in the 10-seed evaluation, while Text-as-Variable has t = 2.27 and is borderline.The original three-seed Cross-Attention estimate of −18.5% was inflated by one outlier seed.
  • FinMultiTime: Across 50 S&P500 tickers, all have ρ3 > 0.99, nonsignificant MI on daily returns, and routing benefit within ±3%.This confirms the diagnostic’s negative prediction at scale.
  • Time-MMD validation: The low-RBU threshold predicts negative routing benefit at 11 of 12 horizon-dataset configurations.The sole exception is Health at h = 1 with +5.3%, attributed to regularization rather than context exploitation.
Loading 2608.25128v1…