PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting

Emma Andrews, Gianmarco Mengaldo

arXiv:2609.11282v1cs.AIcs.ITmath.IT

TL;DR

Text annotations can be predictive, misleading, or irrelevant, but ground-truth information content is unavailable in real datasets, complicating evaluation of information metrics. The paper creates a controlled synthetic benchmark and evaluates six estimators; correct annotations are identified as most informative, while performance on weak signals varies by estimator.

  • Problem

    Real-world datasets lack ground-truth information content, so it is difficult to evaluate whether information metrics reliably measure text's predictive contribution.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs a synthetic time-series benchmark with correct, incorrect, and irrelevant text annotations and evaluates six information estimators.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    CCA, V-Information, and PID identify correct annotations as more informative on MMTT-Bench and real-world datasets, while neural estimators are unreliable at low conditional MI.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Information metrics can support annotation auditing and fusion selection before training, with pre-training metric values correlating with downstream forecasting performance across architectures.

    Source: §1 · p. 2, §4 · p. 6, §7 · p. 9

    On real-world datasets with small conditional MI, MINE can collapse to near-zero or negative estimates and InfoNCE can inflate, with both showing high variance.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multimodal forecasting models that combine time series with text annotations promise richer prediction through textual context, but how do we know whether a text annotation meaningfully contributes to the forecasters prediction? This is an information-theoretic question, but to evaluate whether information-theoretic metrics can reliably measure the predictive value an annotation provides, a ground truth benchmark is needed, and none currently exist. We create a synthetic time series signal with annotations in three categories: semantically correct, incorrect, and irrelevant. Because the data generation process is fully controlled, ground-truth information content is known exactly, enabling principled evaluation of six complementary mutual information estimators (KSG, MINE, InfoNCE, CCA, PID and V-information). We show that all six estimators identify correct annotations as most informative, and are able to audit the quality of mixed text corpora, choosing the annotations that result in the best downstream forecasting results without the need for model training. Our benchmark identifies limitations of each estimator, and these are validated on seven real-world datasets, which show how estimator performance differs on weak signals. Finally, we establish practical rules for implementing these metrics for annotation auditing and fusion selection.

1 INTRODUCTION

Text may be predictive, wrong, or irrelevant, yet practitioners lack a principled way to assess its contribution before training. The paper introduces a controlled benchmark and evaluates information metrics as tools for auditing text annotations and selecting fusion.

1 INTRODUCTION

  • Practitioners need to distinguish predictive annotations from misleading or irrelevant text before investing in model training.Without a reliable informativeness measure, they cannot audit annotation quality or identify where text fusion may help.Source: §1 · p. 1
  • Existing datasets lack ground-truth information content, making principled evaluation of text–time-series information estimators impossible.Prior evaluations rely on human judgment even though the true information content is commonly unknown.Source: §1 · p. 2
  • MMTT-Bench pairs a synthetic signal with correct, incorrect, and irrelevant annotations, enabling controlled evaluation with known information content.The paper also validates its findings on Time-MMD and FinTexTS.Source: §1 · p. 2
  • The paper evaluates six complementary information metrics as pre-training diagnostics for forecasting performance across model architectures.It presents practical rules for using these metrics to assess annotation quality across corpora, signals, and use cases.Source: §1 · p. 2

2 MMTT-BENCH DESIGN

MMTT-Bench uses a synthetic signal with controlled text annotations to compare how information metrics distinguish correct, incorrect, and irrelevant text. Its construction makes annotation information content known by design.

2 MMTT-BENCH DESIGN

  • MMTT-Bench pairs a time-series signal with controlled natural-language annotations, so the information each annotation carries about future values is known.This enables principled evaluation across annotation categories unavailable in existing multimodal datasets.Source: §2 · p. 2
  • The signal is a sine wave with randomly placed flat segments, creating transitions where current values do not determine the next value.At these transition states, correct annotations become the dominant source of predictive information.Source: §2 · p. 2
  • Chronological, non-overlapping train, validation, and test segments are used to prevent leakage and reflect forecasting on future observations.Each time point receives correct, incorrect, and irrelevant annotation variants based on the signal's known future state.Source: §2 · pp. 2–3
  • Incorrect annotations include transition templates at 30% of points, matching the true transition frequency so lexical diversity alone cannot distinguish them.Because 70% give the opposite future direction, incorrect text remains statistically related to future values under an inverted mapping.Source: §2 · p. 3

3 ESTIMATING MUTUAL INFORMATION

The paper estimates how much unique predictive information text adds beyond the time series, using six complementary estimators and a representation pipeline designed to preserve category ordering. It situates this evaluation against prior MI work and discusses estimator constraints.

3.1 NOTATION AND BACKGROUND

  • Conditional MI, I(Xtext; Y | Xts), measures text's unique contribution beyond the time series and is the primary quantity for deciding whether fusion is worthwhile.The task estimates information functionals from finite samples of time-series input, embedded text, and a future signal target.Source: §3.1 · p. 3

3.2 ESTIMATORS

  • The six estimators differ in target and assumptions: KSG, MINE, InfoNCE, and CCA estimate Shannon MI, PID uses a unique-information atom, and V-Information measures Ridge-exploitable gain.Weak signals are harder to estimate; MINE and InfoNCE require sufficient signal strength and can systematically underestimate MI as lower bounds.Source: §3.2 · pp. 3–4

3.3 RELATED WORK

  • Prior work uses MI to predict model behavior or guide training, while existing evaluations often lack oracle information content or controlled annotation-quality labels.MMTT-Bench extends synthetic MI benchmarking to natural language, whose embedding creates a representational bottleneck absent from numeric-only distributions.Source: §3.3 · pp. 4–5

3.4 REPRESENTATION PIPELINE

  • The representation sweeps examine PCA dimension and lookback effects on category ordering; KSG's negative estimates near its dimensionality limit motivate dtext = 16 and nlookback ≤ 2.Conditional MI ordering remains stable across eight embedding models, and DistilRoBERTa is selected for its information-to-computational-cost balance.Source: Figure 2 · p. 5, §3.4 · p. 5

4 RESULTS

On MMTT-Bench, conditional-MI estimates tracked corpus quality and downstream forecasting performance. Across seven real-world datasets, estimator pathologies recurred, while annotation information was often diffuse rather than timestamp-aligned.

4 RESULTS

  • The results compare corpus categories as the independent variable, MI estimates as the first dependent measure, and forecasting MSE as the second across 14 architectures and, where applicable, 10 fusion mechanisms.Conditional MI represents text's unique information about future predictions, with estimator-specific definitions for KSG, MINE, InfoNCE, CCA, PID, and V-information.Source: §4 · p. 6

4.1 MMTT-BENCH

  • Correct text was most informative on MMTT-Bench, while KSG and PID nearly tied incorrect and irrelevant text; other estimators significantly ranked incorrect above irrelevant.Incorrect annotations encode the opposite future direction for 70% of points, creating real dependence through an inverted mapping; irrelevant text encodes no systematically related dimension.Source: §4.1 · pp. 6–7
  • All six estimators showed a positive relationship between metric value and downstream performance across model baselines, with aligned trend-line slopes.The time-series-only baseline anchors each panel at zero MI for added text, attributing the measured gains to text rather than the signal.Source: §4.1 · p. 7
  • 0.849 average |ρ| linked conditional MI and MSE across mixture corpora and five models; auditing selected the best corpus, including one with 25% irrelevant text when pure-category corpora were excluded.CCA achieved |ρ| of 0.90–1.00, while KSG reached 0.40–0.73; the selected corpus improved MSE by −0.07 over random selection in the restricted test.Source: §4.1 · p. 7, Figure 4 · p. 7

4.2 REAL-WORLD VALIDATION

  • For Social Good, Public Health, and Energy, time-series-only R2 was 0.65–0.93 versus 0.37 on MMTT-Bench; despite this higher baseline predictability, estimator pathologies recurred while CCA, V-information, and PID remained stable and correctly ordered.KSG’s conditional MI turned negative when the marginal-to-conditional ratio was large; MINE turned negative and InfoNCE inflated under weak signal.Source: §4.2 · p. 8
  • Conditional MI was positive and significant across seven datasets, but alignment contrasts were usually indistinguishable from zero; adding text worsened median MSE except on Climate.On Agriculture and Public Health, irrelevant text degraded performance less than correct text, distinguishing aggregate information from timestamp-specific alignment.Source: §4.2 · p. 8

5 RECOMMENDATIONS

The recommendations match estimator choice to data conditions and favor paired contrasts when judging whether text is aligned with the time series.

5 RECOMMENDATIONS

  • Use CCA and V-information as the default audit pair; their conditional MI rank-correlates with downstream MSE at |ρ| =0.90-1.00 across five model families.They are deterministic given a resample, require no training, are scale-invariant, and never produce negative conditional MI.Source: §5 · p. 8
  • Use MINE and InfoNCE only with large N and substantial conditional MI; weak real-world signals or small samples make them unreliable.They separate incorrect from irrelevant text through sensitivity to sign-inverted dependence, but MINE approaches zero and InfoNCE has high variance on weak signals.Source: §5 · p. 8
  • Avoid KSG for conditional MI when marginal I(Xts; Y) is large, because dimensionality-dependent bias can dominate the difference and produce negative values.Source: §5 · pp. 8–9
  • Use PID only when Y has interpretable discrete structure; arbitrary binning for continuous targets materially changes the estimate.PID was the most precise estimator on clean data, including essentially zero unique information for irrelevant text.Source: §5 · p. 9
  • Prefer paired contrasts with shuffled or misaligned corpora over absolute conditional MI values, because the contrast cancels estimator bias.The authors demonstrate this alignment check on Time-MMD and FinTexTS, where text carried measurable information not tied to its timestamp.Source: §5 · p. 9

6 LIMITATIONS

The benchmark's controlled synthetic design enables oracle evaluation, while its representation choices and estimator behavior impose practical boundaries. Results are also validated with noise and real-world datasets.

6 LIMITATIONS

  • MMTT-Bench's sine wave and imposed constant segments create an unusually clean separation, enabling oracle ground truth unavailable in real-world datasets.The estimator evaluation was also validated after adding noise to the sine signal and on seven real-world datasets.Source: §6 · p. 9
  • Pair-fine-tuned text embeddings may perform better, but would prevent isolating I(Xtext; Y | Xts), the text's unique contribution beyond the time series.Simple embeddings are presented as a realistic auditing choice when computational cost constrains embedding selection.Source: §6 · p. 9
  • On real-world datasets with small conditional MI, MINE can collapse to near-zero or negative estimates, while InfoNCE inflates; both have high variance despite tuning.The authors identify critic-based estimation's sensitivity to signal strength as a structural limitation.Source: §6 · p. 9
  • PID is limited to targets with interpretable discrete structure; continuous real-world targets lack natural bins, making arbitrary discretization choices sensitive hyperparameters.The four-value discretization is semantically motivated for MMTT-Bench's sine signal, but not for continuous targets such as Time-MMD.Source: §6 · p. 9

7 CONCLUSION

The paper benchmarks six information-theoretic estimators and reports that some identify informative annotations reliably while neural estimators struggle with low conditional MI. It offers practical recommendations and releases benchmark resources.

7 CONCLUSION

  • CCA, V-Information, and PID identify correct annotations as more informative on MMTT-Bench and real-world datasets, while MINE and InfoNCE are unreliable at low conditional MI.Pre-training metric values correlate with downstream MSE across multiple model architectures.Source: §7 · p. 9
  • The authors present seven recommendations for using MI estimators in multimodal settings and release MMTT-Bench, estimator implementations, and model-training pipelines.Source: §7 · p. 9

ETHICS STATEMENT

The authors describe potential benefits and misuse risks of tools for assessing text annotations in consequential forecasting domains. They recommend treating MI metrics as one part of human-in-the-loop quality assurance.

ETHICS STATEMENT

  • MI-based annotation auditing could support reliability and transparency in consequential forecasting, but misleading high-MI annotations could be crafted in adversarial settings.The authors consider near-term risk low because computing MI requires access to time series and future values, limiting manipulation to retrospective settings.Source: ETHICS STATEMENT · p. 10
  • The synthetic data contain no personal, sensitive, or proprietary information; Time-MMD and FinTexTS are public datasets used under their terms.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper makes its benchmark, estimator and model-training code, and experiment-regeneration scripts available for reproducibility.

REPRODUCIBILITY STATEMENT

  • The full dataset-generation code is released with MMTT-Bench, allowing the construction process to be audited, regenerated, modified, or extended.The dataset is available in the repository and on GitHub and HuggingFace.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The repository includes the mutual-information estimators and training code for all fourteen model architectures, with implementation and training details documented in appendices.Estimator parameters, training parameters, and compute requirements are recorded there.Source: REPRODUCIBILITY STATEMENT · p. 10
  • A single script generates the paper’s results, and additional scripts regenerate tables, plots, and experiments.The authors state that MMTT-Bench experiments can be reproduced in one command; preprocessing scripts and source references are also provided where appropriate.Source: REPRODUCIBILITY STATEMENT · p. 10

A DATASET GENERATION DETAILS

MMTT-Bench pairs a controlled sine-wave signal with sparse natural-language annotations, using known signal phases to create correct, incorrect, and irrelevant text. Its construction varies both signal regimes and annotation wording to test whether estimators distinguish their predictive information.

Dataset availability

  • The authors provide the dataset-generation code so that the benchmark can be inspected and regenerated.The code is available from the project’s GitHub repository.Source: A DATASET GENERATION DETAILS · p. 12

Signal construction

  • A 512-period sine wave is randomly masked across regions totaling 30% of the full signal, with masked values set to -0.5, then split into train (62.5%), validation (18/75%), and test (18/75%) files.Source: A DATASET GENERATION DETAILS · p. 12

Annotation points and phases

  • Annotations are placed at quarter-cycle intervals and assigned phases from their positions, including ordinary sine-wave phases and transition phases before signal changes.Points before transitions are labeled drop_to_constant, while the point immediately before oscillation resumes is recover_from_constant.Source: A DATASET GENERATION DETAILS · p. 12

Non-transition annotations

  • Outside transitions, correct and incorrect text use the same templates, with future-oriented wording and direction or magnitude choices based on the local gradient; steepness is defined as a gradient exceeding ±0.7.The non-transition templates draw on varied signal names and paraphrasing word banks.Source: A DATASET GENERATION DETAILS · pp. 12–13

Transition annotations

  • Correct transition annotations use templates at actual transition points, while 30% of incorrect annotations use transition templates distributed randomly across the signal.This preserves comparable template frequency while making most incorrect text directly conflict with the signal.Source: A DATASET GENERATION DETAILS · pp. 14–15

Annotation information

  • The incorrect category retains more predictive information than irrelevant text but less than correct text, because its transition wording is mixed with mostly conflicting annotations.This graded ordering enables more fine-grained evaluation of mutual-information estimators.Source: A DATASET GENERATION DETAILS · p. 15

Annotation templates

  • The text is generated from varied templates and word banks, including separate transition wording for changes to and recovery from the constant signal level.The examples include alternative verbs, nouns, and sentence structures.Source: A DATASET GENERATION DETAILS · pp. 15–16

B INFORMATION THEORETIC METRIC DETAILS

This section describes six complementary approaches for estimating information between text, time-series features, and future values. The estimators differ in their assumptions, computational approach, and limits, while PID and V-information address distinct questions about modality contributions and model-usable information.

B INFORMATION THEORETIC METRIC DETAILS

  • The benchmark compares six information-theoretic estimators; InfoNCE has a log N ceiling, and KSG reliability is bounded by sample size relative to dimensionality.At N = 64, the InfoNCE ceiling is 4.16 nats; KSG’s conservative and liberal bounds are d ≤ N/2 and d ≤ N/5.Source: Table 1 · p. 16, B INFORMATION THEORETIC METRIC DETAILS · p. 16

B.1 K-NEAREST-NEIGHBOR ESTIMATOR

  • KSG estimates mutual information from joint-space nearest-neighbor distances and corresponding marginal neighbor counts, with a corrected variance estimator used instead of bootstrap variance.Its estimates become unreliable beyond the stated sample-size bounds on dimensionality.Source: §B.1 · pp. 16–17

B.2 PARTIAL INFORMATION DECOMPOSITION

  • Partial Information Decomposition separates joint information into redundancy, each modality’s unique contribution, and synergy; its inputs are discretized after PCA and k-means clustering.The decomposition is expressed as I(Xts, Xtext; Y) = R + Uts + Utext + S.Source: §B.2 · p. 17

B.3 V-USABLE INFORMATION

  • V-information measures information usable by a predictive family, and the conditional estimate here measures the gain in explained variance from adding text to time-series features.The Ridge-based estimate is reliable at small sample sizes and high embedding dimensionality, but depends on the representation and linear model’s capabilities.Source: §B.3 · p. 17

B.4 NEURAL MUTUAL INFORMATION ESTIMATORS

  • MINE and InfoNCE estimate text, time-series, joint, and conditional information using neural critics, with conditional estimates obtained by subtracting the time-series estimate from the joint estimate.Estimating the joint and marginal quantities in the same representational space avoids the bias difference that can cause KSG chain-rule violations.Source: §B.4 · p. 17

B.4.1 MINE

  • MINE uses a statistics network to maximize a Donsker–Varadhan lower bound, with shuffled negative samples and an exponential moving average to reduce gradient variance.The critic is trained for 500 iterations per estimate and is not subject to KSG’s density-estimation curse of dimensionality.Source: §B.4.1 · p. 18

B.4.2 INFONCE

  • InfoNCE frames estimation as N-way classification and uses off-diagonal scores as implicit negatives; it provides a tighter bound than MINE in practice but saturates at log N nats.Its critic is trained for 500 iterations per estimate.Source: §B.4.2 · p. 18

B.5 CANONICAL CORRELATION ANALYSIS ESTIMATOR

  • CCA gives a closed-form mutual-information estimate from canonical correlations under a multivariate Gaussian assumption, requiring only a matrix decomposition rather than tuning or neural training.It provides a reference upper bound on linear-Gaussian MI for the benchmark; its Gaussian assumption is unsuitable for heavy-tailed and strongly nonlinear distributions.Source: §B.5 · p. 18

C EMBEDDING VISUALIZATIONS

The embedding visualization shows that correct and incorrect annotations overlap in text-only space, while irrelevant annotations form a distinct cluster. In joint text-and-time-series space, time-series states dominate the structure and text categories are nearly superimposed within each state.

C EMBEDDING VISUALIZATIONS

  • In text-only embeddings, correct and incorrect annotations overlap, whereas irrelevant annotations form a distinct cluster.The visualization is a two-dimensional PCA of the training-set text embeddings.Source: C EMBEDDING VISUALIZATIONS · pp. 18–19, Figure 6 · p. 19
  • In joint embeddings, the first principal component explains 41.2% of variance and separates the four time-series states into vertical stripes.The states are −1, 0, +1, and −0.5.Source: C EMBEDDING VISUALIZATIONS · p. 19
  • Text categories are almost entirely superimposed within each joint-embedding stripe, indicating a fine-grained conditional effect relative to the dominant time-series structure.Correct and incorrect text use identical template structures and are difficult to distinguish without the time-series information.Source: C EMBEDDING VISUALIZATIONS · p. 19

D EXPERIMENT DETAILS

The experiments test estimator and forecasting behavior across controlled perturbations and real-world datasets, using multiple model architectures, fusion strategies, and estimator settings. Correct text remains identifiable under noisy signals, while temporal misalignment reveals a gap between estimated information and downstream performance.

D EXPERIMENT DETAILS

  • The supplied experiment details report MINE taking approximately 53 ± 0.64 seconds per MI estimate on CPU for the MMTT-Bench training split of N = 1279 points.The reported experiment runs 20 estimates for each result.Source: D EXPERIMENT DETAILS · p. 19

D.1 MODEL TRAINING

  • The experiments train five scikit-learn regressors and nine transformer architectures, with ten text-fusion strategies per transformer and a shared unimodal baseline per backbone.The simple regressors concatenate the time-series lookback and PCA-reduced text features; transformer models use the Controlled Fusion Adapter implementation.Source: §D.1 · p. 19, §D.1.1 · p. 19, §D.1.2 · p. 19

E.1 TEXT TOKENIZATION AND DIMENSIONALITY REDUCTION

  • DistilRoBERTa achieves the highest Xtext information relative to Xts among sentence-transformer models and is the baseline; Xtext information rises with text dimension, while KSG is unreliable above d ≤ N/2 ≈25, so dtext = 16 is the highest viable value.Source: §E.1 · p. 20

E.2 TIME SERIES TOKENIZATION

  • A two-step raw-value lookback supplies some predictive information without making sine-wave prediction trivial; raw values are retained instead of PatchTST tokenization, and PID uses four bins for Yfuture.The four classes correspond to annotation-point values 1, 0, −1, and −0.5.Source: §E.2 · p. 21, Table 5 · p. 22, §E.3 · p. 22

F.3.1 SHUFFLE RESULTS

  • Shuffling annotations across timestamps lowers MINE and InfoNCE estimates for incorrect text, while estimates for irrelevant text do not change.This supports the interpretation that the inverted signal comes from the text–future-value pairing, not the embedding distribution alone.Source: §F.3.1 · p. 26, Table 10 · p. 27

F.3.2 SIGNAL PERTURBATIONS

  • Across four noisy-signal levels, the estimators still rank correct text highest.The benchmark permits controlled signal perturbations while preserving known annotation information.Source: §F.3.2 · p. 27, Table 11 · p. 27

F.3.3 TEMPORAL JITTER

  • Under temporal jitter, estimated mutual information decays monotonically to the irrelevant-text level, but forecasting performance is worst at an intermediate displacement and then partially recovers.Slightly misaligned text is locally plausible and actively misleading; heavily displaced text is de-correlated.Source: §F.3.3 · p. 27, Table 12 · p. 28

G REAL WORLD DATASETS

  • For Time-MMD, text from another domain supplies the irrelevant category; FinTexTS experiments use ten selected stocks, and real-world alignment and conditional-MI contrasts are reported across seven datasets.Time-MMD’s processed text is labeled correct despite not being verified ground truth; Table 18 reports MSE reduction versus no-text across transformer architectures and fusion strategies.Source: G REAL WORLD DATASETS · p. 28, Table 18 · p. 31

Related papers

Loading 2609.11282v1…