PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting

Karthik Sridhar, Atharva Gupta, Nishant Pradhan, Murari Mandal, Dhruv Kumar, Saurabh Deshpande

arXiv:2608.22321v1cs.CL

TL;DR

The paper asks whether reported multimodal forecasting gains on Time-MMD come from text semantics or another signal sharing the text pathway. It audits three architectures with controlled text perturbations and mechanism probes, finding that text substitutions barely affect error while a co-shipped numeric column explains gains in two models. The authors conclude that text content is not the operative signal within this benchmark and frozen-encoder setting.

  • Problem

    The standard text-disabled comparison does not establish whether multimodal lifts reflect sensitivity to text content.

  • Method

    The study holds numeric inputs fixed while substituting text across five conditions and probes the text pathway and co-shipped numeric column.

  • Results

    Less than 0.5%: replacing text changes mean MSE by no more than this amount on all three architectures, while lifts persist through substitutions.

  • Takeaways & Limitations

    Reported Time-MMD gains do not verify genuine use of text semantics; benchmarking should intervene on text content and isolate co-shipped numeric features.

  • Takeaways & Limitations

    The audit covers Time-MMD and three frozen-encoder architectures, leaving end-to-end trained encoders and higher-diversity benchmarks for future work.

Abstract

from arXiv · show

Multimodal time-series forecasting has emerged as a promising paradigm in which natural-language context is expected to improve predictive performance. Recent multimodal foundation models, including Aurora, as well as early- and late-fusion approaches such as MM-TSFlib and TaTS, report substantial gains over unimodal baselines on the Time-MMD benchmark, attributing these improvements to textual information. However, whether these models are actually sensitive to the semantic content of the text remains unverified. We address this question through controlled text perturbations, attribution analyses, and probes of Aurora's text pathway. On Time-MMD, swapping each row's text for any other real text (empty, constant, within-domain shuffled, or cross-domain) moves mean MSE by less than $0.5\%$ on all three architectures. The improvement reported in the literature is recovered when a co-shipped numeric column is removed without touching text. We conclude that, on this benchmark and within this family of frozen-encoder architectures, text content is not the operative signal behind the reported gains. To support future work on text integration in multimodal foundation models for structured data, we release our perturbation protocol and evaluation harness as a reusable diagnostic toolkit.

1. Introduction

The paper questions whether multimodal forecasting gains on Time-MMD arise from text semantics, testing this with controlled perturbations across three architectures. It reports text-content insensitivity and localises the gains to a co-shipped numeric signal.

  • Research gap: The standard multimodal-versus-unimodal comparison cannot establish whether models respond to text content.A fixed, content-blind text-branch output could produce the same multimodal lift.
  • Approach: The study swaps each row’s text while holding numeric inputs unchanged across five substitute conditions.The conditions include empty, constant, within-domain shuffled, cross-domain, and oracle text.
  • Finding: All three architectures are insensitive to natural-language text content within the benchmark’s precision.The protocol spans one foundation model and two paradigm implementations on Time-MMD.
  • Mechanistic localisation: A co-shipped numeric column accounts for the multimodal lift in two architectures, while Aurora’s text pathway is trained but content-blind in the forward pass.The numeric column is blended alongside text in the relevant fusion path.
  • Reproducibility: Code, patched runners, and the evaluation harness are released for reproducibility.The release supports reuse of the perturbation protocol and evaluation setup.

2. Background

Time-MMD pairs numeric target series with temporally aligned text across nine domains and varying forecast horizons. The audited architectures use distinct dedicated text pathways, motivating a test of whether those pathways respond to text content.

  • Evaluation setting: Time-MMD spans nine domains with numeric targets and temporally aligned text from public reports and domain bulletins.Forecast horizons range from 6 to 336 steps depending on sampling frequency.
  • Text pathways: TATS performs early fusion by projecting frozen GPT-2 text representations and concatenating them with numeric history.The combined representation is passed to a time-series backbone.
  • Text pathways: MM-TSFLIB performs late fusion by adding a normalized frozen BERT text embedding as a residual to backbone output.Its text representation enters after the backbone’s main processing.
  • Text pathways: Aurora compresses frozen BERT token features into 10 learnable query tokens and injects them through cross-attention.A text guider connects the compressed text representation to the temporal backbone.
  • Motivation: All three architectures have dedicated text pathways, motivating the question of whether those pathways are sensitive to what the text says.The paper measures multimodal contribution using each model’s text-disabled unimodal configuration.

3. Perturbation Protocol

The perturbation protocol replaces text independently of numeric inputs and evaluates whether forecasts change when language presence, alignment, topical relevance, or content is altered. Results are aggregated across domains, horizons, seeds, and trained backbones.

  • Protocol design: Each model is rerun with five text substitutes while every numeric input remains identical across conditions.The schematic Table 1 row keeps numeric history at 11.9 in every condition.
  • Perturbations: Empty and Constant text test whether the encoder reacts to text presence, while Shuffled and Cross-domain text break alignment or topical relevance.These conditions preserve or remove different properties of the original language.
  • Perturbations: The Oracle condition uses a templated sentence listing future ground-truth values as a necessary but insufficient sensitivity test.Frozen encoders may not represent numeric literals as quantities because numerals are tokenized into sub-word pieces.
  • Evaluation: Relative MSE changes are computed against the original-text baseline and matched on domain, horizon, seed, and backbone.The aggregation uses the ratio of mean MSEs across cells with 95% bootstrap confidence intervals from 10,000 resamples.

4. Results

Across all three architectures, text substitutions barely change forecasting error, despite real multimodal lifts over unimodal baselines. The null is robust across conditions and backbones, while lift magnitude varies with backbone choice.

  • Text sensitivity: Less than 0.5%: no text substitution changes mean MSE by more than this amount on any architecture.The largest shift is +0.16% for MM-TSFLIB with Constant text; TATS stays within ±0.001% and Aurora within ±0.05%.
  • Text sensitivity: Empty, Constant, Shuffled, Cross-domain, and Oracle text all leave forecasts essentially unchanged.The null holds whether natural language is preserved, removed, or replaced with a fixed placeholder.
  • Multimodal lift: The multimodal lifts over unimodal baselines are real, but the experiments do not show that they reflect models reading text content.Cross-domain and future-value substitutions would be expected to change outputs if semantic content drove the gain.
  • Backbone robustness: Backbone choice changes lift magnitude substantially, while the text-content null remains uniform.TATS lifts range from +1.2% to +15.0%, and MM-TSFLIB lifts range from +0.3% to +3.8%.
  • Backbone robustness: Text-only perturbations stay within ±0.01% on TATS and ±2.4% on MM-TSFLIB across all backbones.These values contrast with substantial backbone-dependent variation in structural-column effects.

5. Localising the Lift

The reported multimodal lift is localized to a co-routed numeric column in TATS and MM-TSFLIB, while Aurora’s text pathway is active but content-blind. Text perturbations and structural diagnostics show that changing text content does not meaningfully alter forecasting error.

  • Lift localization: The published multimodal lift is real, yet text content does not move error.The architecture inspection identifies a gap between the lift and semantic text sensitivity.
  • Lift localization: In TATS and MM-TSFLIB, a co-routed numeric column p is blended into the output alongside the text pathway.Both models use the same residual pathway for p and the text representation, while Aurora never reads p.
  • Factorial analysis: The 2×3 factorial crosses text conditions with p intact or zeroed, isolating text effects from the numeric-column effect.Within-column contrasts measure text changes, while column contrasts measure p changes.
  • Factorial analysis: Less than 0.2% on MM-TSFLIB, 0.001% on TATS, and 0.05% on AURORA: changing text moves MSE negligibly when p status is fixed.For MM-TSFLIB, the p-zeroed confidence interval overlaps the unimodal confidence interval, supporting clean isolation of p’s contribution.
  • Robustness: Backbone choice changes the apparent contribution substantially, with TATS p-zeroed effects ranging from +4.5% to +49.9%.On MM-TSFLIB, the unimodal-baseline gap ranges from +0.3% to +3.8%, so a single backbone can misstate p’s contribution by an order of magnitude.
  • Aurora probes: Aurora’s text pathway has non-zero gradient norms, uniform attention entropy of 0.975, and prediction change at most 0.058.The prediction change is two orders of magnitude below the approximately 8.6 test-MSE scale, indicating a trained but content-blind forward pathway.
  • Text diagnostics: Moderate TTW and ETA coexist with uniformly low SDI, while perturbations change all three diagnostics without moving downstream MSE beyond 0.5%.None of TTW, ETA, or SDI reliably predicts whether a perturbation changes the forecast.

6. Discussion and Conclusion

Across the three architectures, plausible text substitutions leave Time-MMD error nearly unchanged, while the reported gains are attributable to a numeric feature or a content-blind text pathway. The audit therefore calls for direct text interventions and explicit isolation of co-shipped numeric inputs, within the study’s limited scope.

  • Conclusion: Less than 0.5%: replacing original text with empty, constant, shuffled, cross-domain, or oracle text changes mean MSE on all three architectures.The reported lifts survive every tested text substitution.
  • Conclusion: On TATS and MM-TSFLIB, the lift localizes to numeric column p, while Aurora’s text pathway is trained but content-blind in the forward pass.The three architectures therefore do not exhibit meaningful sensitivity to the tested text content.
  • Implications: A disabled-text baseline cannot distinguish a model that reads text from one that gates on a co-routed numeric prior.The standard comparison verifies that the text pathway contributes some signal, but not what kind.
  • Implications: Rigorous benchmarking requires direct text-content intervention and isolation of numeric features co-shipped through the text-fusion path.The paper also identifies higher per-row semantic diversity as a way to provide attentive encoders with more distinctive text.
  • Limitations: The audit covers Time-MMD and three frozen-encoder architectures; end-to-end trained encoders and higher-diversity benchmarks remain next steps.A directional oracle may probe semantic sensitivity more directly than the numeric oracle because sub-word tokenization bottlenecks it.

Impact Statement

This paper is a methodological audit of multimodal time-series forecasting on an existing public benchmark and released research code. By identifying confounded baselines, it aims to improve evaluation rigor in the subfield.

  • Scope: The work audits multimodal time-series forecasting methods on a public benchmark rather than introducing new models, datasets, or applications.It uses the existing Time-MMD dataset and publicly released research code.
  • Impact: Identifying confounded baselines is intended to improve the rigor of evaluation standards in multimodal time-series forecasting.The impact statement reports no specific ethical concerns, direct deployment risks, or applications involving harm.

A. Architecture Equations

The three architectures combine numeric histories with text through distinct fusion designs, but their published unimodal baselines do not isolate text from the co-shipped numeric column p. Data-loader reconstruction clarifies which columns each architecture actually consumes and how evaluation perturbations preserve numeric inputs.

  • TATS: TATS projects frozen GPT-2 text features, concatenates them with numeric history before the backbone, and blends the forecast with p.Its default blend weight is w=0.5.
  • TATS: TATS’s unimodal baseline zeros projected text channels and sets w=0, removing both text and p simultaneously.Therefore, its multimodal lift cannot be attributed to text alone.
  • MM-TSFLIB: MM-TSFLIB projects a pooled frozen-BERT embedding into a horizon-shaped vector, adds it to p, and blends the result with the backbone.Its default blend weight is w=0.1.
  • MM-TSFLIB: MM-TSFLIB’s unimodal baseline sets w=0, removing the shared gated residual containing both text and p.The ablation therefore removes both signals together.
  • AURORA: AURORA uses frozen BERT token features, a trainable distiller, and cross-attention to inject ten distilled text tokens into the temporal backbone.Unlike TATS and MM-TSFLIB, AURORA does not consume p; its unimodal baseline disables only the text path.
  • Data and evaluation: Across the released data loaders, OT is the target series, p is an LLM-derived numeric forecast from preceding history, and AURORA reads fact while ignoring other text columns.The evaluation uses univariate features='S', with text on a separate path and p entering as a residual where supported.
  • Data and evaluation: Perturbed files preserve row count, dates, targets, and all unchanged numeric columns, while runs are evaluated per model, backbone, condition, seed, domain, and horizon.Cross-domain substitutions use deterministic date-matched paired-domain text without future target information.

E. Probe Definitions and Full Results

Aurora’s probes separate training signal, attention concentration, and forward-pass influence at the interface between its frozen encoder and trainable text distiller. The pathway receives gradients but has negligible effect on predictions.

  • Probe setup: Aurora compresses frozen BERT token features into ten distilled tokens that the temporal backbone reads through cross-attention.The probes target this encoder–distiller interface.
  • Probe definitions: Probe A measures root-mean-square loss-gradient flow at the distilled tokens to test whether the trainable text path receives optimization signal.A zero value would indicate that the path was never optimized.
  • Probe definitions: Probe B measures normalized cross-attention entropy, where 1 denotes uniform attention and 0 denotes one-token concentration.It tests whether the guider discriminates among distilled tokens.
  • Probe definitions: Probe C compares forecasts from identical numeric inputs with text present versus the text branch ablated, using averaged stochastic samples.Its divergence is small when the forward-pass prediction is unchanged.
  • Full results: The text-only conditions show non-zero gradient norms, attention entropy of 0.975 on every row, and prediction divergence at most 0.058.The divergence is two orders of magnitude below the approximately 8.6 test-MSE scale, indicating a trained but forward-pass content-blind pathway.

F. Text-Side Diagnostics: Definitions and Full Results

Text-side diagnostics measure alignment, temporal persistence, and row-to-row semantic diversity, but their substantial changes under perturbation do not predict downstream forecast changes. The benchmark’s text embeddings are often structurally similar across consecutive rows.

  • Definitions: TTW measures frequency-spectrum alignment between text-embedding trajectories and target changes; lower values indicate closer dominant-frequency matching.The metric was introduced as a text–target alignment measure and proposed as a covariate-suitability indicator.
  • Definitions: ETA is mean lag-1 autocorrelation across embedding dimensions: high values indicate smooth temporal evolution, while low values indicate near-independent consecutive embeddings.It captures temporal persistence rather than semantic distinctiveness.
  • Definitions: SDI measures semantic diversity between consecutive rows; low SDI means neighboring embeddings are nearly identical and cannot provide row-specific content through attention alone.This makes SDI a structural property relevant to encoder-based text use.
  • Original-text results: Across the original text, TTW and SDI are uniformly low, while ETA varies widely across domains; the highest GPT-2 SDI is 0.013 on Traffic.SocialGood and Energy evolve smoothly, whereas Security and Traffic resemble more independent draws.
  • Perturbation results: Despite order-of-magnitude diagnostic changes, downstream MSE changes by less than 0.5% in every text-only perturbation condition.Thus TTW, ETA, and SDI do not reliably predict whether a perturbation moves the forecast.
  • Perturbation results: Empty and Constant text collapse SDI to zero and push ETA toward one, while Shuffled raises BERT SDI from 0.031 to 0.084 and Cross-domain text raises BERT TTW from 0.037 to 0.108.These conditions produce qualitatively different diagnostic signatures across the three measures.

G. Full Results: Per-Backbone, Per-Domain, Per-Condition

Across backbones and domains, text-content perturbations remain near-null, while structural perturbations produce substantial changes. Paired bootstrap intervals confirm that the text-only effects are precisely estimated rather than merely underpowered.

  • Per-domain pattern: The five text-only rows appear grey or near-grey across domains, whereas p=0 and Unimod. produce bold deltas.Isolated bold text-only cells on tiny baselines reflect floating-point and sampling noise rather than a real text effect.
  • Backbone-level summary: Text-only perturbations stay within ±0.6% of the original on every TATS and MM-TSFLIB per-backbone average.The text-only conditions are uniformly null across all eight backbones.
  • Backbone-level summary: The p=0 effect varies from +4.5% to +49.9% on TATS and from 0.0% to +3.8% on MM-TSFLIB across backbones.These ranges show that structural sensitivity depends strongly on the backbone.
  • Backbone-level summary: The unimodal-baseline effect ranges from +1.2% to +15.0% on TATS and from +0.3% to +3.8% on MM-TSFLIB.A single-backbone multimodal lift can therefore misstate the contribution of the shipped numeric column.
  • Paired bootstrap intervals: With n = 864 matched cell pairs, text-only confidence intervals include zero or remain below 0.001% on TATS, 0.5% on MM-TSFLIB, and 0.1% on AURORA.The paired bootstrap uses B = 10,000 resamples and ratio-of-means estimates.
  • Paired bootstrap intervals: On TATS, the p-zeroed CI is [+16.8, +24.7], while the Unimodal CI is [+6.4, +8.8].Because zeroing p leaves TATS at half scale, the Unimodal row is the cleaner amplitude-matched comparison.

I. The TaTS Text-Projection Gradient Patch

Auditing TaTS revealed detached tensors that prevented the text-projection MLP from receiving gradients during training. After patching this path, text perturbations remained null, while Aurora’s zero-shot scope leaves possible pretraining contamination unresolved.

  • Gradient-path audit: Detached concatenated inputs block gradients from reaching TaTS’s text-projection MLP ψ during fine-tuning.Although a separate optimizer steps ψ, the MLP remains at random initialization without a gradient signal.
  • Patch: The --fix text grad patch gates two detach calls and replaces three in-place divisions that otherwise break autograd.The flag is off by default, and pre-existing JSON results remain bit-comparable when it is off.
  • Patch validation: All reported TaTS numbers use the patch, with --fix text grad on for the headline results.The patch makes ψ trainable in practice rather than only in name.
  • Patch validation: Text-content perturbations still leave TaTS MSE unchanged within ±0.001% across all eight backbones after ψ becomes trainable.The null therefore does not hinge on the detach bug.
  • Aurora scope: Aurora’s predictions are essentially identical for original text, column-zeroed input, and the unimodal baseline.On this benchmark, its forward pass does not branch on the text input regardless of the supplied text.
  • Aurora scope: Aurora’s pretraining corpus is large and only partially documented, so Time-MMD-derived content seen during pretraining cannot be ruled out.This contamination uncertainty does not explain the observed forward-pass silence on the benchmark.
Loading 2608.22321v1…