PaperlayerView exact revision on arXiv

Source-linked AI summary

Modalities Should Talk to Each Other: Dual-Stream Multimodal Learning for Long-Horizon Influenza Forecasting

Seyed Mohammad Hossein Hashemi, Mohsen Hooshmand, Parvin Razzaghi

arXiv:2608.23373v1cs.AIstat.AP

TL;DR

Long-horizon ILI forecasting must reconcile numerical surveillance with temporally mismatched text, which fixed-feature fusion does not model as mutually informative. DSA lets the streams condition each other through bidirectional cross-modal attention; across ten seeds, it outperformed TaTS and had the lowest median MSE and tail-risk among the top four models.

  • Problem

    Existing fusion that combines independently encoded modalities does not explicitly model how numerical and textual signals should influence one another’s interpretation.

    Source: §1 · p. 1

  • Method

    DSA uses bidirectional Cross-Modal Attention to let numerical epidemiological history and concurrent text condition each other before causal temporal aggregation and forecasting.

    Source: §6 · p. 13

  • Results

    1.217 ± 0.194 vs. 1.371 ± 0.177 ten-seed mean MSE, DSA’s bidirectional CMA outperformed TaTS; across ten seeds, it had the lowest median MSE and tail-risk among the top four models.

    Source: §5.7 · p. 12, §6 · p. 14

  • Takeaways & Limitations

    Ablations support cross-modal interaction as a contributor to DSA’s advantage rather than a consequence of a particular language model.

    Source: §6 · p. 14

    DSA does not lead in every epidemiological phase, underperforming TaTS in the rising phase and GPT4MTS in the off-season and declining phases.

    Source: §5.7 · p. 13

Abstract

from arXiv · show

Forecasting long-range influenza-like illness (ILI) matters for public health readiness. Publicly available surveillance datasets typically pair numeric epidemiological signals with textual information that is noisy, loosely structured, only indirectly related to near-term trends, and often lagged relative to the numeric signal. Fusing the two therefore requires careful design. We propose Dual-Stream Attention (DSA), a multimodal deep learning framework that forecasts 12-week-ahead ILI activity from a 36-week multimodal history by letting the numerical and textual streams condition each other. Using the Time-MMD health-domain dataset, DSA separately encodes the two modalities with a Transformer-based numerical encoder and a domain-adapted headline encoder, then couples them through a bidirectional Cross-Modal Attention (CMA) mechanism: the text (news headlines) conditions the interpretation of the numeric signal and vice versa. The CMA output then passes to a causal temporal model for forecasting. Evaluated across ten random seeds, DSA achieves a median test MSE of 0.416, versus 0.668, 0.607, and 0.851 for iTransformer, TaTS, and GPT4MTS, corresponding to mean-error reductions of 54.95%, 37.29%, and 67.23%, with paired Cohen's d of 0.555, 0.337, and 0.345, respectively, and ranks first in 100% of bootstrap draws. It also has substantially lower worst-window error than all baselines. On an external-geography dataset, DSA again ranks first among nine evaluated baselines. Ablations show the advantage does not depend on text-encoder choice or language-model fine-tuning, and that bidirectional attention outperforms either direction alone. Finally, perturbation-based faithfulness analysis shows the learned CMA is functionally informative under targeted masking, with a stronger effect in the text-to-numerical direction.

1. Introduction

ILI forecasting must account for seasonal, non-stationary numerical trends and accompanying text that may be noisy or temporally offset. DSA addresses this by letting separate numerical and text streams condition one another through bidirectional cross-modal attention before forecasting.

1. Introduction

  • DSA keeps numerical and text streams separate, then uses bidirectional Cross-Modal Attention to condition each representation on the other before causal temporal processing and 12-week forecasting.The numerical history uses a Transformer encoder, weekly text uses BioLinkBERT, and the resulting representation is processed by a causal temporal Transformer.Source: §1 · pp. 1–2
  • The evaluation uses a 36-week lookback and 12-week horizon on Time-MMD Health_US, screening a broad candidate set before comparing strong representatives across ten seeds and complementary robustness analyses.Analyses include tail risk, forecast horizon, epidemic phase, bootstrap rank probabilities, Model Confidence Sets, and external validation on Health_Africa.Source: §1 · pp. 1–2
  • DSA’s median test MSE was 0.4163 on Health_US; paired comparisons favored it in all ten seeds versus iTransformer (Cohen’s d=0.555), TaTS (d=0.337), and GPT4MTS (d=0.345).The introduction also reports markedly lower tail-risk than competing models and a best-model ranking in the preliminary Health_Africa evaluation.Source: §1 · p. 2
  • DSA led every baseline in the peak phase, while in off-season, rising, and declining phases it was not always best but remained on par with the leader and among top performers.The introduction identifies peak as the hardest and most operationally consequential phase; DSA also had lower tail-risk and ranked best in preliminary Health_Africa evaluation.Source: §1 · p. 2
  • Controlled ablations examine fusion, bidirectionality, fine-tuning, and text-encoder choice; targeted perturbations find greater forecast degradation from highly attended information, especially Text→Numerical, with a recency confound limiting causal interpretation.The perturbation analysis compares highly attended historical information with random or weakly attended information.Source: §1 · p. 2

2. Related work

Related work spans numerical forecasting, text-augmented and LLM-based forecasting, and epidemic prediction; explicit interaction between structured epidemic signals and semantic health text remains comparatively underexplored.

2.1. Unimodal time-series forecasting

  • Numerical forecasting includes Transformer variants such as PatchTST [4] and iTransformer, DLinear [3], and multi-attention models, but these use structured series rather than free text.PatchTST tokenizes sub-series patches, iTransformer embeds variates, and DLinear decomposes trend and seasonality.Source: §2.1 · p. 2

2.2. Multimodal and Text-Augmented forecasting

  • Influenza work combined observations with Google queries, while TaTS [14] integrates text embeddings by addition or concatenation; these approaches typically lack explicit bidirectional text–numeric interaction.Cross-modal attention models such interactions in vision-language work but has seen comparatively little use in time-series forecasting.Source: §2.2 · pp. 2–3

2.3. LLM-based time-series forecasting

  • GPT4MTS [16] adapts a frozen GPT-2 backbone [17] for multimodal forecasting using patch-based numerical embeddings and a textual prompt, with larger language-model components than purpose-built fusion architectures.This line draws on general-purpose representations learned through large-scale language pretraining.Source: §2.3 · p. 3

2.4. ILI and epidemic forecasting

  • Traditional SIR/SEIR models [1] and statistical ARIMA or autoregressive neural models [18] frame epidemic prediction through disease dynamics or surveillance trajectories; DSA adds text and bidirectional cross-modal conditioning.DSA combines contemporaneous textual surveillance with epidemiological measurements, letting both streams shape each other before temporal forecasting.Source: §2.4 · p. 3

3. Methods

DSA forecasts 12 weeks of ILI from a 36-week paired history, keeping numerical and text streams separate until bidirectional attention lets each condition the other before temporal aggregation. Its architecture then uses causal temporal modeling to produce the forecast.

3.1. Problem Formulation

  • Given 36 weeks of 14-dimensional numerical features and paired text, DSA directly predicts all 12 future ILI weeks in one forward pass.Source: §3.1 · p. 3

3.2. The Dual-Stream Attention Architecture

3.2.1. Numerical Encoder Module

  • The numerical stream is projected to hidden size and encoded by a 4-layer Transformer without positional encoding, which hurt performance in experiments.Source: §3.2.1 · p. 3

3.2.2. Headline Encoder Module

  • Weekly headlines are tokenized to 256 tokens, encoded with biomedical-pretrained BioLinkBERT-base using masked mean pooling and last-eight-layer unfreezing, then mixed across weeks by inter-week self-attention.Source: §3.2.2 · pp. 3–4

3.2.3. Cross-Modal Attention Module

  • Bidirectional CMA lets each modality condition how the other is interpreted, addressing the weak interaction of pooled or one-way text conditioning before temporal aggregation.The numerical state helps determine which text matters, while text also conditions interpretation of the numerical trend.Source: §3 · p. 3, §3.2.3 · p. 4
  • CMA runs two attention lanes in opposite directions, then concatenates and projects their outputs, adds residuals to both normalized streams, and applies LayerNorm.Source: §3.2.3 · p. 4

3.2.4. Temporal Modeling

  • A second positional embedding precedes a 2-layer causal Transformer, whose mask allows attention only to current and past positions; its final hidden state is pooled for forecasting.Source: §3.2.4 · pp. 4–5

3.2.5. Regressor and Training Objective

  • A two-layer MLP maps the final temporal representation to the 12-week forecast, and end-to-end training minimizes mean squared error against ground-truth ILI values.Source: §3.2.5 · p. 5

4. Experiments

Section 4 establishes a reproducible evaluation using aligned epidemiological and text data, broad baseline families, multiple seeds, and consistent statistical tests. It describes the data, protocol, implementation, and statistical methods underlying the results in Section 5.

4.1. Data and preprocessing

  • Time-MMD Health_US pairs weekly ILI surveillance with factual search summaries, aligning the modalities and excluding GPT-3.5-generated predictions to avoid potential leakage.The data are split chronologically 80/10/10; numerical features use a training-fitted MinMax scaler, and ISO week is seasonally encoded with sine and cosine. DSA is also evaluated on Time-MMD Health_Africa.Source: §4.1 · pp. 5–6

4.2. Baselines

  • Baselines span unimodal forecasters, naive text-feature concatenation, TaTS on three numerical backbones, and GPT4MTS with a frozen six-layer GPT-2 backbone.The unimodal models are DLinear [3], PatchTST [4], and iTransformer [5]; TaTS is cited as, and GPT4MTS as [16], with GPT-2 cited as [17].Source: §4.2 · p. 6

4.3. Evaluation protocol

  • Four models—DSA, iTransformer, TaTS with an iTransformer backbone, and GPT4MTS—were selected by single-seed screening for full evaluation across ten independent seeds.The protocol reports median test MSE alongside horizon-averaged error, tail risk, phase-stratified results, and rank-probability and Model Confidence Set analyses.Source: §4.3 · p. 6

4.4. Implementation details

  • DSA uses AdamW at a 10^-5 learning rate, dropout 0.15, cosine scheduling, and gradient accumulation to an effective batch size of 16.Training uses a 10-epoch warmup, automatic mixed precision, and Kaggle dual-T4 GPUs; the configuration addresses a scheduler–accumulation interaction linked to seed-to-seed variance.Source: §4.4 · p. 6

4.5. Statistical methodology

  • Pairwise accuracy is assessed with Diebold–Mariano tests using Newey–West variance at lag 11, alongside paired Cohen’s d, exact sign tests, and Model Confidence Set analysis.Phase-stratified comparisons control false discovery rate with Benjamini–Hochberg at α = 0.05.Source: §4.5 · pp. 6–7
  • Primary claims rely on median MSE, tail risk, effect size, percentage improvement, directional consistency, and Model Confidence Set inclusion rather than underpowered per-seed tests alone.Per-seed Diebold–Mariano testing is treated as a supplementary robustness check because ten seeds may leave individual tests underpowered.Source: §4.5 · p. 7

5. Results & Discussion

Across ten seeds, DSA leads the overall benchmark and maintains advantages in tail-risk, most epidemic phases, and external-geography evaluation. Ablations and targeted masking support bidirectional cross-modal fusion as useful, while phase-specific exceptions and other limitations qualify the findings.

5.1. Main Results

  • DSA had the best values on all four reported overall metrics; its horizon-averaged MSE was 1.120, below TaTS (1.371), GPT4MTS (1.712), and iTransformer (1.753).Across all ten seeds, DSA’s advantage was favored against every baseline, with effect sizes d=0.34–0.56 and mean-error improvements of 37–67%.Source: §5.1 · p. 7

5.2. Tail-Risk and Horizon-Wise Robustness

  • DSA’s mean maximum-window error was 6.4036, below iTransformer’s 10.7543, TaTS’s 10.1020, and GPT4MTS’s 14.5867.The largest single-window MSE across seeds was 7.703 for DSA versus 15.643, 13.748, and 20.407, respectively. DSA’s advantage over iTransformer grew with horizon, peaked around week 7 against TaTS and week 11 against GPT4MTS, and no baseline outperformed DSA on average at any horizon.Source: §5.2 · pp. 8–9, Figure 5 · p. 9, Figure 6 · p. 10

5.3. Phase-Stratified Robustness

  • DSA had the lowest mean MSE in the off-season, rising, and peak phases, but GPT4MTS led in the declining phase, where its MSE was 0.97 versus DSA’s 2.26.DSA remained in the Model Confidence Set at all tested thresholds in off-season, rising, and peak phases, but only at the two more permissive thresholds during decline; averaged across phases, DSA still had the lowest mean MSE and was strongest during peak.Source: §5.3 · p. 9, Figure 8 · p. 11

5.4. Ablation Studies

  • Bidirectional CMA beat either single-direction variant in a single-seed test, while a frozen text encoder still beat TaTS; causal masking was load-bearing, and BioLinkBERT was not essential.Removing the causal mask raised test MSE from 1.09 to 1.71; mean pooling beat CLS representation (1.09 vs. 1.28), while RevIN degraded performance (1.80). Frozen-encoder mean MSE was 1.217 ± 0.194 versus TaTS at 1.371 ± 0.177; general-domain encoders cost some accuracy but preserved the qualitative result, and the directional ablation was not tested across multiple seeds.Source: §5.4 · pp. 9–11

5.5. Generalization to a Second Geography

  • On Health_Africa, DSA remained best on all three error metrics, though this was a single external dataset and broader multi-region evaluation is needed.GPT4MTS was the strongest baseline; TaTS underperformed its unimodal backbones, and supplementary dimension-mapping tuning left results essentially unchanged or worse.Source: §5.5 · p. 11, Table 4 · p. 12

5.6. Interpretability: CMA Faithfulness

  • Targeted masking showed Top-k > Random-k > Bottom-k effects in both attention directions; masking the top three Text→Numerical weeks raised MSE by 1.44, about 40 times the Numerical→Text effect of 0.035.The ordering held for k=1 to k=12, but the authors treat this as functional informativeness, not proof that attention is causal or that every attended week is necessary.Source: §5.6 · p. 11, Table 5 · p. 12

5.7. Discussion and Limitations

  • Newey–West-corrected per-seed significance reached conventional levels only for DSA versus iTransformer; recency confounds the Text→Numerical masking result, and fixed-size pooling limited extension beyond H=12.The reported p-values were 0.025 against iTransformer, 0.106 against TaTS, and 0.142 against GPT4MTS; the authors also caution that the correction is conservative and that the dataset cannot separate genuine recency informativeness from a confound.Source: §5.7 · p. 13

6. Conclusion

DSA forecasts 12-week ILI from 36-week numerical and textual histories by using bidirectional cross-modal attention before causal temporal aggregation. Across ten seeds, it showed the strongest overall error profile among the top four models, while phase-stratified results, ablations, and perturbations further characterize its performance and limitations.

6. Conclusion

  • DSA uses bidirectional Cross-Modal Attention so numerical history informs text interpretation and text informs numerical interpretation before causal aggregation and direct multi-step decoding.This design contrasts with simply appending text to a numerical forecast representation.Source: §6 · pp. 13–14
  • Across ten seeds, DSA had the lowest median MSE, lower tail-risk, and highest bootstrap rank-1 probability among the top four models.Its advantage had a consistent direction across all ten seeds, with favorable effect sizes against iTransformer, TaTS, and GPT4MTS.Source: §6 · p. 14
  • Peak-phase MSE was 6.738 for DSA, versus 11.953 for TaTS, 14.405 for iTransformer, and 28.591 for GPT4MTS; DSA ranked among the top two in all four phases.The preliminary Health_Africa evaluation without retuning was also consistent with transferability of the learned fusion strategy.Source: §6 · p. 14
  • Ablations support cross-modal interaction as a contributor, while targeted perturbations indicate functional importance, especially for the Text→Numerical pathway.DSA retains an advantage with frozen BioLinkBERT and remains competitive with general-domain text encoders; single-direction attention variants underperform full DSA, but this comparison is single-seed. The perturbation result is non-causal, and recency may partly explain the Text→Numerical effect.Source: §6 · p. 14
  • Future work should test generalization across diseases, geographies, and text-paired time-series domains, and extend forecasting beyond 12 weeks with a richer decoder.Attempts to extend beyond H = 12 with the current single-pooled-representation decoder did not hold up; broader tests would establish whether cross-modal conditioning generalizes beyond this dataset.Source: §6 · p. 14

Supplementary Material … B.5. Text-encoder sensitivity

Across ten matched seed comparisons, DSA consistently beats the three core baselines; supplementary analyses examine reproducibility, architecture choices, fusion direction, and text-encoder sensitivity. The ablations support bidirectional cross-modal fusion as a major contributor to DSA’s advantage, while showing that encoder choice does not produce a uniform ordering across metrics.

A. Training Stability and Reproducibility; A.1. Sources of run-to-run variation; A.2. Final reproducibility configuration

  • A learning-rate-schedule and gradient-accumulation interaction motivated fixing one training protocol before the ten-seed evaluation, rather than tuning each seed independently.The final setup uses AdamW, a 10^-5 learning rate, dropout of 0.15, cosine scheduling, 10 warmup epochs, and four accumulation steps with micro-batches of four.Source: A. Training Stability and… · p. 14, §A.1 · p. 14

A.3. Random-seed protocol

  • Ten independent seeds used the same chronological split, preprocessing, architecture, optimizer, scheduler, and evaluation procedure to estimate stability rather than search for a favorable initialization.The principal configuration is summarized in Table 6, with unvaried or unreported values omitted.Source: §A.3 · pp. 14–15, §A.2 · p. 14

A.4. Why the seed distribution matters

  • DSA wins all ten matched comparisons against iTransformer, TaTS, and GPT4MTS, showing the principal result is not based on a single favorable seed.The ten-seed analysis also reports median test MSE, horizon-averaged error, tail-risk statistics, and directional consistency.Source: §A.4 · p. 15

B.1. Single-seed screening protocol

  • A fixed-seed screening stage selected DSA, iTransformer, TaTS with an iTransformer backbone, and GPT4MTS for the more costly ten-seed comparison.Screening covered unimodal numerical forecasters, naive multimodal fusion, TaTS variants, GPT4MTS, and DSA; it was not treated as a ten-seed statistical experiment.Source: §B.1 · p. 15

B.2. Architecture sensitivity

  • Removing the causal mask raises test MSE from approximately 1.09 to 1.7092; CLS-token pooling yields 1.2825, and a six-layer numerical encoder yields 1.1940.The final model uses masked mean pooling over BioLinkBERT representations; the deeper encoder did not improve performance at the available dataset size.Source: §B.2 · p. 15, Table 8 · p. 16

B.3. Directional Cross-Modal Attention

  • Both single-direction CMA variants underperform bidirectional DSA on all three metrics in the single-seed ablation, but this pattern has not been tested across seeds.The unidirectional variants use fusion projections sized to their single attention branch, isolating directionality rather than parameter count.Source: §B.3 · p. 16, Table 9 · p. 16

B.4. Frozen text encoder

  • 1.217±0.194 mean MSE for frozen-text DSA compares with 1.120±0.096 for full DSA and 1.371 ± 0.177 for TaTS with an iTransformer backbone across ten seeds.Freezing BioLinkBERT measurably reduces performance, but the comparison attributes most of DSA’s advantage over TaTS to cross-modal attention, with fine-tuning adding a smaller improvement.Source: §B.4 · p. 16

B.5. Text-encoder sensitivity

  • BioLinkBERT remains strongest on median MSE and P90 MSE, while MiniLM has the lowest mean worst-case MSE; either general-domain encoder retains DSA’s fusion advantage.The encoder experiments use five seeds, rather than the ten seeds in the principal comparison.Source: §B.5 · p. 16

B.6. Health_Africa supplementary analysis · C. Additional Statistical Analyses · C.1. Per-seed directional consistency

In the preliminary Health_Africa evaluation, multimodal DSA had the lowest reported test errors among the compared configurations. Across ten matched seeds, DSA favored each of the three core baselines in every comparison.

B.6. Health_Africa supplementary analysis

  • With frozen encoders, DSA-Frozen’s per-sample MSE was 0.5793 versus TaTS’s 0.6070.The mean per-sample error difference was -0.1536 (DSA-Frozen minus TaTS); mean worst-case single-window MSE was 7.36 for DSA-Frozen versus 10.10 for TaTS.Source: Figure 13 · p. 17, Figure 15 · p. 18
  • DSA-Frozen had lower, more stable MSE than TaTS across all forecast horizons, with the advantage particularly evident at longer steps.Source: Figure 14 · p. 17
  • Increasing TaTS’s mapping dimension from 12 to 32 across three numerical backbones produced no consistent improvement and was not adopted.This result does not establish that dimension 12 is globally optimal or that the tested increase resolved the cross-geography weakness.Source: §B.6 · p. 17
  • DSA had the lowest reported Health_Africa test MSE, MAE, and RMSE among the compared configurations.The evaluation used one external dataset and was not a full ten-seed run, so the authors treat it as preliminary transfer evidence.Source: §B.6 · pp. 17–18, Table 11 · p. 18

C.1. Per-seed directional consistency

  • Across ten matched comparisons, DSA had a favorable direction against each of the three core baselines in all ten seeds.The sign test addresses paired seed-level outcome direction, complementing the DM analysis of error-difference magnitude and temporal dependence; the observed direction is unlikely under an equal-win-probability null.Source: §C.1 · p. 18

C.2. Diebold–Mariano analysis · D. Tail-Risk and Horizon-Wise Supplementary Results · D.1. Worst-window behavior

DSA shows lower worst-window error than all three baselines, while Newey–West-corrected Diebold–Mariano tests are negative but individually significant at α = 0.05 only against iTransformer. Bootstrap ranking and Model Confidence Set results add supporting, but qualified, evidence of robustness.

C.2. Diebold–Mariano analysis

  • Newey–West long-run variance correction addresses serial dependence from overlapping test windows and forecast horizons, making inference more conservative than a lag-0 variance estimate.The analysis combines effect sizes, directional consistency, median performance, tail-risk, bootstrap ranks, MCS inclusion, and DM testing rather than relying on one statistic.Source: §C.2 · p. 18
  • Only DSA versus iTransformer crossed the two-sided α = 0.05 threshold in the Newey–West-corrected DM tests; all three DM statistics were negative.The tests average results over ten seeds; the other two pairwise tests did not cross the threshold.Source: §C.2 · pp. 18–19, Table 12 · p. 19, Figure 17 · p. 19
  • DSA ranked first in every reported bootstrap draw and had paired Cohen’s d values of 0.34–0.56, but rank-one frequency is not a formal probability of universal optimality.Bootstrap ranking resampled seed-level performance across all four core models and answers a different question from pairwise DM testing.Source: §C.2 · p. 19
  • DSA was the only model retained in the main MCS across reported confidence levels, but its phase-specific membership was less stable in the declining phase.MCS retention means the model could not be eliminated under the specified loss differential and confidence threshold, not that it is superior under every loss function.Source: §C.2 · p. 19

D.1. Worst-window behavior

  • 7.703 was DSA’s largest single-window MSE across ten seeds, versus 15.643 for iTransformer, 13.748 for TaTS, and 20.407 for GPT4MTS.These are maximum single-window errors across seeds, not average worst-window errors.Source: §D.1 · p. 19, Table 13 · p. 19
  • 6.4036 was DSA’s mean maximum-window MSE, versus 10.7543 for iTransformer, 10.1020 for TaTS, and 14.5867 for GPT4MTS.These means use unrounded evaluation results; Table 13 instead reports the per-seed distribution.Source: §D.1 · p. 19

D.2. P90 error · TaTS GPT4MTS · D.3. Horizon-wise behavior

DSA has the lowest reported P90 error among the four core models. Across the 12 forecast positions, it has lower average MSE than every baseline, with its advantage generally increasing at longer horizons.

D.2. P90 error

  • P90 MSE is the 90th percentile of per-window squared errors pooled across ten seeds, measuring upper-tail error without relying on the single worst window.It lies between median and maximum error in sensitivity to the upper tail.Source: §D.2 · p. 19, TaTS GPT4MTS · p. 20

TaTS GPT4MTS

  • DSA has the lowest reported P90 error among the four core models, consistent with its lowest horizon-averaged error in that comparison.P90 captures performance on harder test windows while avoiding dependence on a single extreme window.Source: TaTS GPT4MTS · p. 20

D.3. Horizon-wise behavior

  • The horizon-wise analysis evaluates each of the 12 forecast positions independently.Source: §D.3 · p. 20
  • DSA and iTransformer are close at horizon 1, after which DSA’s advantage grows and is largest around horizon 10.Source: §D.3 · p. 20
  • DSA’s largest reported advantage over TaTS occurs around horizon 7, while its largest advantage over GPT4MTS occurs around horizon 11.Source: §D.3 · p. 20
  • No analyzed forecast horizon has a baseline with lower average MSE than DSA, and its error margin grows over the horizon rather than reflecting a uniformly large gain.Source: §D.3 · p. 20

E. Phase-Stratified Analysis · F. CMA Faithfulness Analysis · F.1. Perturbation protocol

Phase-stratified results show that DSA’s performance varies across epidemic regimes, while the CMA faithfulness analysis tests whether attention-selected information affects forecasts when masked.

E. Phase-Stratified Analysis

  • Test windows are grouped into off-season, rising, peak, and declining regimes to test whether aggregate performance is driven by one part of the seasonal cycle.Source: E. Phase-Stratified Analysis · p. 20
  • DSA is reported best by mean MSE in the off-season, rising, and peak phases, yet GPT4MTS is marginally better off-season and substantially better during decline, while TaTS is better during rise.The phase analysis cautions that aggregate performance does not imply uniform dominance.Source: E. Phase-Stratified Analysis · p. 20
  • Peak-phase mean MSE is 6.738 for DSA, versus 11.953 for TaTS, 14.405 for iTransformer, and 28.591 for GPT4MTS.Peak has the largest reported errors for all four models.Source: E. Phase-Stratified Analysis · p. 20
  • For rising-phase DSA relative to TaTS, Cohen’s d is −0.19, with TaTS’s mean MSE approximately 16.8% lower.Source: E. Phase-Stratified Analysis · p. 20
  • During decline, GPT4MTS has substantially lower error than DSA, with reported Cohen’s d = −1.13.Source: E. Phase-Stratified Analysis · p. 20

F. CMA Faithfulness Analysis

  • The CMA faithfulness analysis tests whether attention weights are functionally informative, since attention weights alone do not establish that attended information is used by the forecasting function.It compares forecast changes after masking weeks selected by their attention values.Source: §F.1 · p. 20

F.1. Perturbation protocol

  • Masking compares Top-k, Random-k, and Bottom-k weeks separately for both CMA pathways; the primary budget is k = 3 of 36 weeks (about 8.3%), with additional budgets from k = 1 to k = 12.The forecast is recomputed after masking, and the change in MSE from the unperturbed prediction is recorded.Source: §F.1 · p. 20

F.2. Three-week masking results … CRediT authorship contribution statement

Targeted masking suggests attention-ranked weeks are functionally relevant to forecasting, but interpretation is limited by recency confounding and the masking intervention itself. Supplementary analyses support DSA on the tested forecasting problem while bounding broader claims.

F.2. Three-week masking results

  • For k=3, masking the top-attended weeks increased MSE by 1.4436 in Text→Numerical and 0.0352 in Numerical→Text; the former effect was roughly 41× larger.Source: §F.2 · p. 20

F.3. Masking-budget sensitivity

  • Across masking budgets k=1 to k=12, removing more historical weeks produced larger effects, supporting useful attention rankings but not individual necessity.The budgets span approximately 2.8% to 33.3% of the 36-week lookback; simultaneous masking may involve nonlinear interactions and redundant information.Source: §F.3 · p. 21

F.4. Interpretation and limitations

  • The masking experiment is a faithfulness test, not a causal intervention, and its Text→Numerical effect is confounded by attention’s correlation with recency.Masking also removes information abruptly and creates out-of-distribution inputs; separating attention-driven relevance from recency likely requires a larger, more temporally diverse dataset and a design that varies both independently.Source: §F.4 · p. 21

F.5. Integrated Gradients

  • Integrated Gradients complements CMA perturbation by attributing forecasts to individual numerical and text-derived features rather than whole weeks ranked by cross-modal attention.Its global attribution visualization appears in the main manuscript.Source: §F.5 · p. 21

G. Summary of Supplementary Evidence

  • Supplementary evidence supports DSA’s reproducibility, deliberate architecture, fusion-specific advantage, and functionally informative CMA, without establishing universal superiority.DSA won directionally in all ten seeds against the three core baselines; ablations favored selected architecture choices, fusion remained competitive with frozen or alternative text encoders, and single-direction variants underperformed bidirectional DSA in a single-seed test.Source: G. Summary of Supplementary Evidence · pp. 21–22
  • The supported claim is limited to the tested 12-week ILI forecasting problem, not every epidemic phase, geography, horizon, or dataset.Phase-stratified and cross-geography results identify conditions where superiority is not yet established.Source: G. Summary of Supplementary Evidence · p. 22

CRediT authorship contribution statement

  • Hashemi led conceptualization, methodology, investigation, software, visualization, and drafting; Hooshmand and Razzaghi contributed conceptualization, methodology, supervision, validation, and review, with Hooshmand also providing resources.Source: CRediT authorship contribution statement · p. 22

Related papers

Loading 2608.23373v1…