Source-linked AI summary
Modalities Should Talk to Each Other: Dual-Stream Multimodal Learning for Long-Horizon Influenza Forecasting
Seyed Mohammad Hossein Hashemi, Mohsen Hooshmand, Parvin Razzaghi
TL;DR
Long-horizon ILI forecasting must combine seasonal numerical surveillance with noisy, temporally offset text without assuming either modality is uniformly reliable. DSA uses bidirectional cross-modal attention to let the streams condition one another, achieving the strongest overall error profile among the top four models across ten seeds.
Problem
ILI forecasting requires integrating numerical surveillance with textual signals that differ in reliability, relevance, and temporal scale, but late fusion does not model cross-modal conditioning.
Method
DSA separately encodes 36 weeks of numerical history and concurrent text, couples them through bidirectional Cross-Modal Attention, and directly forecasts the next 12 ILI weeks.
Results
DSA delivers the lowest median MSE, markedly lower tail-risk, and the highest bootstrap rank-1 probability among the top four models across ten independent seeds.
Takeaways & Limitations
The results support bidirectional cross-modal interaction as a major contributor to DSA’s forecasting advantage, rather than text-encoder choice or fine-tuning alone.
Takeaways & Limitations
DSA does not outperform every baseline in every epidemiological phase, underperforming TaTS during the rising phase and GPT4MTS during off-season and declining phases.
Abstract
from arXiv · showhide
Forecasting long-range influenza-like illness (ILI) matters for public health readiness. Publicly available surveillance datasets typically pair numeric epidemiological signals with textual information that is noisy, loosely structured, only indirectly related to near-term trends, and often lagged relative to the numeric signal. Fusing the two therefore requires careful design. We propose Dual-Stream Attention (DSA), a multimodal deep learning framework that forecasts 12-week-ahead ILI activity from a 36-week multimodal history by letting the numerical and textual streams condition each other. Using the Time-MMD health-domain dataset, DSA separately encodes the two modalities with a Transformer-based numerical encoder and a domain-adapted headline encoder, then couples them through a bidirectional Cross-Modal Attention (CMA) mechanism: the text (news headlines) conditions the interpretation of the numeric signal and vice versa. The CMA output then passes to a causal temporal model for forecasting. Evaluated across ten random seeds, DSA achieves a median test MSE of 0.416, versus 0.668, 0.607, and 0.851 for iTransformer, TaTS, and GPT4MTS, corresponding to mean-error reductions of 54.95%, 37.29%, and 67.23%, with paired Cohen's d of 0.555, 0.337, and 0.345, respectively, and ranks first in 100% of bootstrap draws. It also has substantially lower worst-window error than all baselines. On an external-geography dataset, DSA again ranks first among nine evaluated baselines. Ablations show the advantage does not depend on text-encoder choice or language-model fine-tuning, and that bidirectional attention outperforms either direction alone. Finally, perturbation-based faithfulness analysis shows the learned CMA is functionally informative under targeted masking, with a stronger effect in the text-to-numerical direction.
1. Introduction
ILI forecasting supports public-health preparedness but is difficult because seasonal, non-stationary dynamics and noisy, temporally offset text require more than unconditional multimodal fusion. DSA addresses this gap through separate numerical and text streams coupled by bidirectional Cross-Modal Attention, with evaluation spanning repeated seeds, robustness analyses, and external validation.
- Motivation: ILI forecasting is challenging because future activity depends on recent epidemiological state within a strongly seasonal and non-stationary process.Reliable near- and medium-term estimates support preparedness, resource allocation, and interpretation of changing seasonal dynamics.
- Methodological gap: Text and numerical surveillance signals may differ in reliability, relevance, and temporal scale, motivating interaction-aware rather than late unconditional fusion.Text may precede numerical changes, while numerical trends can contextualize whether textual signals are epidemiologically meaningful.
- Proposed approach: DSA uses separate Transformer-based numerical and BioLinkBERT text streams connected by bidirectional Cross-Modal Attention.The architecture explicitly lets each modality condition and reweight the other instead of concatenating independently encoded features.
- Evaluation and results: 0.4163 median test MSE was achieved by DSA on Health_US, with paired Cohen’s d values of 0.555, 0.337, and 0.345 versus iTransformer, TaTS, and GPT4MTS.DSA was favored in paired comparisons across all ten random seeds and also showed lower tail-risk than competing models.
- Faithfulness analysis: Targeted perturbations produced greater forecasting degradation for highly attended historical information, especially in the Text→Numerical direction.The analysis supports functional informativeness of learned attention while acknowledging a recency confound limiting causal interpretation.
2. Related work
Related work progresses from numerical and traditional epidemic forecasting to text-augmented and LLM-based approaches. DSA distinguishes itself by combining contemporaneous epidemiological and textual signals through explicit bidirectional cross-modal attention.
- Research landscape: Related work spans numerical time-series forecasting, external-information and text-augmented forecasting, LLM-based forecasting, and influenza or epidemic forecasting.These categories reflect a progression from unimodal numerical prediction toward multimodal epidemiological prediction.
- Numerical forecasting: Transformer architectures dominate recent time-series forecasting, including PatchTST’s patch tokens and iTransformer’s variate-token formulation.PatchTST reduces sequence length through sub-series patches, while iTransformer embeds each variate rather than each timestep.
- Text-augmented forecasting: Text-augmented forecasting establishes the value of auxiliary information but typically combines text embeddings with numerical backbones through addition or concatenation.Influenza work has also used structured Google search-query features, whereas TaTS represents text through projected embeddings rather than contextual bidirectional interaction.
- LLM-based forecasting: GPT4MTS repurposes a frozen GPT-2 backbone for multimodal forecasting by combining patch-based numerical embeddings with a textual prompt.This line benefits from general-purpose language representations but uses substantially larger language-model components than purpose-built fusion architectures.
- DSA’s distinction: DSA combines contemporaneous textual surveillance with structured epidemiological measurements and uses explicit bidirectional cross-modal attention instead of one-directional conditioning or unconditional concatenation.Traditional epidemic forecasting commonly used mechanistic transmission or statistical time-series models, including SIR- and SEIR-family formulations.
3. Methods
DSA forecasts 12 weeks of ILI from a 36-week multimodal history using separate numerical and text encoders, bidirectional cross-modal attention, causal temporal aggregation, and an end-to-end regressor. Its central design lets each modality condition how the other is interpreted rather than simply concatenating representations.
- Problem formulation: DSA takes 36 weeks of 14-dimensional numerical features and paired text snippets to produce all 12 forecast weeks in one forward pass.This is a direct multi-step forecasting setup rather than recursive one-step prediction.
- Model architecture: The architecture comprises numerical and headline encoders, a Cross-Modal Attention module, a causal temporal model, and a regressor.The numerical stream uses a 4-layer Transformer, while text is encoded with BioLinkBERT-base and inter-week self-attention.
- Cross-Modal Attention: CMA’s bidirectional attention lets numerical trends condition text interpretation and text condition numerical-trend interpretation.The two attention lanes run in opposite directions, then their outputs are combined through projection, residual addition, and LayerNorm.
- Temporal modeling: A 2-layer causal Transformer aggregates fused representations while restricting attention to current and past positions, preventing access to future positions.The final temporal representation is taken from the last position for forecasting.
- Forecasting objective: A two-layer MLP maps the final temporal representation to the 12-week forecast, and DSA is trained end-to-end with mean squared error.The forecast horizon is fixed at 12 weeks in this study.
4. Experiments
The experiments evaluate DSA on a temporally aligned multimodal influenza benchmark using broad baselines, a ten-seed protocol, and robustness-oriented metrics. Statistical analyses emphasize paired comparisons, directional consistency, ranking, and limitations of low-powered per-seed tests.
- Dataset: DSA is evaluated on Time-MMD Health_US, which pairs weekly ILI surveillance statistics with text aligned to the same timeline.This alignment avoids separate event matching or cross-modal synchronization.
- Evaluation protocol: The two-stage protocol screens all candidate configurations under one seed, then evaluates DSA, iTransformer, TaTS, and GPT4MTS across ten independent seeds.The selected seeds are 0, 1, 2, 3, 5, 7, 10, 13, 21, and 42.
- Evaluation metrics: The primary metric is median test MSE, supplemented by horizon-averaged error, tail risk, epidemic-regime breakdowns, rank probabilities, and Model Confidence Set analysis.Tail risk includes maximum and 90th-percentile per-window MSE.
- Statistical analysis: The statistical framework uses paired Diebold–Mariano tests with Newey–West variance, paired Cohen’s d, matched-unit percentage improvement, exact sign tests, and Model Confidence Sets.False discovery rate is controlled with Benjamini–Hochberg at α=0.05 for phase-stratified comparisons.
- Limitations: Per-seed Diebold–Mariano tests are treated as supplementary because ten seeds make individual tests comparatively underpowered.Primary claims instead rely on aggregate error, tail-risk, effect-size, improvement, directional-consistency, and Model Confidence Set statistics.
5. Results & Discussion
DSA outperforms the evaluated baselines across overall accuracy, tail-risk, bootstrap ranking, and external-geography performance, while ablations and perturbations identify cross-modal fusion as its main advantage. Its performance is strong but phase-dependent, with a substantial weakness during influenza decline.
- Overall performance: DSA achieves the best value on all four reported overall metrics and ranks first in every bootstrap draw.Its horizon-averaged MSE is 1.120, versus 1.371 for TaTS, 1.712 for GPT4MTS, and 1.753 for iTransformer.
- Tail-risk: 7.703 is DSA’s largest observed single-window MSE, below iTransformer’s 15.643, TaTS’s 13.748, and GPT4MTS’s 20.407.Mean maximum-window errors are 6.4036 for DSA, 10.7543 for iTransformer, 10.1020 for TaTS, and 14.5867 for GPT4MTS.
- Phase-stratified performance: DSA has the lowest mean MSE in three of four phases, but GPT4MTS decisively outperforms it in the declining phase with Cohen’s d = −1.13.DSA also underperforms TaTS in the rising phase and GPT4MTS marginally in the off-season, while remaining inside the Model Confidence Set at every tested threshold in those exceptions.
- Ablations: Removing the causal mask raises test MSE from 1.09 to 1.71, while the frozen-text-encoder variant still achieves 1.217 ± 0.194 versus TaTS’s 1.371 ± 0.177.These ablations identify the causal mask as load-bearing and indicate that fusion, rather than text-encoder fine-tuning, accounts for the advantage.
- External generalization: DSA remains best on all three error metrics on the external-geography dataset, with GPT4MTS the strongest baseline.Increasing TaTS’s text-to-numerical dimension mapping from 12 to 32 left results essentially unchanged or worse across all three backbones.
- Attention faithfulness: 1.44 is the average MSE increase from masking the top-3 attended weeks in the Text→Numerical direction, roughly 40 times the Numerical→Text effect of 0.035.The Top-k > Random-k > Bottom-k ordering holds from k = 1 to k = 12, supporting functional informativeness without establishing causality.
6. Conclusion
DSA forecasts 12-week ILI activity by bidirectionally conditioning numerical epidemiological history and weekly text before causal temporal aggregation. Across seeds and epidemiological phases, it shows a strong, consistently competitive error profile, while ablations support cross-modal interaction as a key contributor.
- Method: DSA uses bidirectional Cross-Modal Attention so numerical history informs text interpretation and text informs numerical-history interpretation before causal temporal aggregation.This design replaces simple concatenation with mutual conditioning between modalities.
- Overall results: Across ten independent seeds, DSA has the lowest median MSE, lower tail-risk, highest bootstrap rank-1 probability, and favorable effects against iTransformer, TaTS, and GPT4MTS.Its advantage is directionally consistent across all ten seeds.
- Phase-stratified results: DSA ranks among the top two models in all four epidemiological phases, despite not being best in every phase.The conclusion emphasizes full-season robustness, especially where errors are largest.
- Ablations and faithfulness: DSA retains a substantial advantage with BioLinkBERT frozen and remains competitive with general-domain text encoders, indicating the interaction mechanism is not tied to one language model.A single-seed attention ablation further examines the contribution of the bidirectional design.
- Future work: The findings motivate testing DSA across diseases, geographies, and paired numerical-text domains, while extending horizons beyond 12 weeks without increasing error.Longer horizons may require a richer decoder than the study’s single pooled representation, alongside a domain-appropriate text encoder.
Supplementary Material … B.5. Text-encoder sensitivity
The supplementary analyses establish a reproducible, bidirectional DSA configuration and test its sensitivity across candidate models, architectural choices, text-encoder training, and encoder identity. DSA’s advantage persists across these evaluations, although results vary by metric and fine-tuning contributes additional performance.
- A. Training Stability and Reproducibility: Substantial seed-to-seed variation motivated selecting DSA’s final configuration through a reproducibility investigation rather than treating the lowest individual run as representative.The investigation examined training-control choices before fixing the configuration used for the ten-seed evaluation.
- A.1. Sources of run-to-run variation: The final protocol used AdamW, a 10−5 learning rate, 0.15 dropout, cosine scheduling, 10-epoch warmup, mixed precision, four-step accumulation, and effective batch size 16.The learning-rate schedule and accumulation steps were identified as important sources of early seed-to-seed variation.
- A.3. Random-seed protocol: The same training protocol was applied to all seeds, preserving the ten-seed distribution as an estimate of architectural stability rather than independently tuned runs.The principal comparison used ten independent random seeds with the same splits, preprocessing, architecture, optimization, scheduling, and evaluation procedures.
- A.4. Why the seed distribution matters: The ten-seed principal result reports median test MSE, horizon-averaged error, tail-risk statistics, and directional consistency rather than performance from the best seed.Across ten seeds, DSA wins against iTransformer, TaTS, and GPT4MTS in all ten matched comparisons.
- B.1. Single-seed screening protocol: Single-seed screening selected DSA, iTransformer, TaTS with an iTransformer backbone, and GPT4MTS for the more expensive ten-seed evaluation.The broader candidate space included unimodal numerical forecasters, BERT-family fusion variants, TaTS variants, GPT4MTS, and DSA; the screening was not treated as a ten-seed statistical experiment.
- B.2. Architecture sensitivity: Removing the causal mask increased test MSE from approximately 1.09 to 1.7092, while replacing masked mean pooling with CLS-token pooling increased it to 1.2825.These architecture-search results support retaining the causal mask and masked mean pooling.
- B.3. Directional Cross-Modal Attention: At seed 42, both unidirectional CMA variants underperformed full bidirectional DSA on test MSE, MAE, and RMSE.The Text→Numerical-only variant degraded less than the Numerical→Text-only variant, but dropping either direction reduced accuracy; this was a single-run result without significance testing.
- B.4. Frozen text encoder: Frozen BioLinkBERT produced a ten-seed mean MSE of 1.217±0.194 versus 1.120±0.096 for full DSA and 1.371 ± 0.177 for TaTS with an iTransformer backbone.Freezing the encoder removes task-specific adaptation and causes measurable loss, indicating that fine-tuning contributes part of DSA’s gain.
B.6. Health_Africa supplementary analysis · C. Additional Statistical Analyses · C.1. Per-seed directional consistency
The Health_Africa analysis provides preliminary evidence that DSA transfers across geography, while per-seed comparisons favor DSA in all matched evaluations against the three core baselines. Supplementary tests also examine TaTS capacity, frozen encoders, horizon stability, and tail risk without establishing a fully grounded external validation claim.
- B.6. Health_Africa supplementary analysis: The Health_Africa experiment uses no retuning and investigates TaTS text-to-numerical dimension mapping as a fair comparative baseline.The mapping dimension was increased from 12 to 32 for three TaTS numerical backbones.
- B.6. Health_Africa supplementary analysis: 0.5793 average per-sample MSE for DSA-Frozen is slightly lower than 0.6070 for TaTS under frozen encoders.The mean per-sample error difference, DSA-Frozen minus TaTS, is -0.1536.
- B.6. Health_Africa supplementary analysis: DSA-Frozen achieves lower and more stable MSE than TaTS across all forecast horizons, especially at longer forecast steps.The horizon-wise error differences are consistently negative.
- B.6. Health_Africa supplementary analysis: Increasing TaTS mapping dimensions from 12 to 32 produced no consistent improvement and was not adopted for external comparison.This does not establish that the chosen dimension is globally optimal; it only indicates that the tested increase did not resolve cross-geography weakness.
- B.6. Health_Africa supplementary analysis: 6.00 mean worst-case MSE for MiniLM-L6-v2 is lowest, followed by 6.21 for DSA and 6.67 for BERT-base-uncased.A separate comparison reports DSA-Frozen at 7.36 versus TaTS at 10.10 for overall mean worst-case single-window MSE.
- B.6. Health_Africa supplementary analysis: DSA has the lowest reported Health_Africa test MSE, MAE, and RMSE among the compared configurations.The evaluation is treated as preliminary transfer evidence rather than a fully grounded external validation claim.
- C.1. Per-seed directional consistency: Ten matched seed-level comparisons favor DSA in all ten evaluations against each of the three core baselines.The sign test assesses directional outcomes rather than error magnitude or temporal dependence, complementing the DM analysis.
C.2. Diebold–Mariano analysis · D. Tail-Risk and Horizon-Wise Supplementary Results · D.1. Worst-window behavior
The supplementary analyses use autocorrelation-robust Diebold–Mariano testing alongside effect sizes, bootstrap ranks, Model Confidence Sets, and worst-window errors to assess DSA’s robustness. Across these lenses, DSA consistently outperforms the baselines, though phase-specific MCS membership is less stable during decline.
- C.2. Diebold–Mariano analysis: Newey–West correction accounts for serial dependence in overlapping multi-horizon forecast-error differentials, producing more conservative inference than a naive lag-0 variance estimate.Negative differentials favor DSA.
- C.2. Diebold–Mariano analysis: Only DSA versus iTransformer crosses the two-sided α = 0.05 threshold; all three DM statistics are negative, consistent with DSA’s 10/10 directional win rate.The DM statistics are averaged over ten seeds and test mean-difference magnitude and precision.
- C.2. Diebold–Mariano analysis: DSA ranks first in every reported bootstrap draw and remains in the Model Confidence Set at every tested threshold, supporting stable descriptive ranking and consistent model selection.Bootstrap rank-one frequency is not equivalent to a formal probability that DSA is universally optimal.
- C.2. Diebold–Mariano analysis: Paired Cohen’s d values use matched differences, while bootstrap ranking resamples ten-seed performance across all four core models.These procedures preserve the paired experiment and answer different questions from the DM test.
- C.2. Diebold–Mariano analysis: MCS membership remains stable for DSA across off-season, rising, and peak phases but becomes less stable during the declining phase.MCS retention does not establish superiority over every excluded model under every loss function.
- D.1. Worst-window behavior: 7.703 is DSA’s largest per-seed single-window MSE, versus 15.643 for iTransformer, 13.748 for TaTS, and 20.407 for GPT4MTS.Lower maximum single-window MSE indicates better worst-window robustness.
- D.1. Worst-window behavior: 6.4036 is DSA’s mean maximum-window error, compared with 10.7543 for iTransformer, 10.1020 for TaTS, and 14.5867 for GPT4MTS.These aggregate means use underlying unrounded evaluation results; per-seed values describe their distribution.
D.2. P90 error · TaTS GPT4MTS · D.3. Horizon-wise behavior
DSA has the lowest reported P90 error among the four core models, consistent with its lowest horizon-averaged error. Across the 12 forecast positions, DSA never trails a baseline, with its advantage generally widening at longer horizons.
- D.2. P90 error: P90 MSE pools per-window squared errors across ten seeds and reports the value below which 90% of errors fall.This measures high-end window error without being dominated by a single outlier.
- TaTS GPT4MTS: DSA has the lowest reported P90 error among the four core models, consistent with its lowest horizon-averaged error.The comparison is reported for test windows and covers DSA, TaTS, GPT4MTS, and iTransformer.
- D.3. Horizon-wise behavior: Across all 12 forecast positions, DSA and iTransformer are close at horizon 1, after which DSA’s advantage increases toward horizon 10.The largest reported gap between DSA and iTransformer occurs around horizon 10.
- D.3. Horizon-wise behavior: The largest reported DSA advantage over TaTS occurs around horizon 7.This is part of the independent horizon-wise evaluation of all 12 forecast positions.
- D.3. Horizon-wise behavior: The largest reported DSA advantage over GPT4MTS occurs around horizon 11.The horizon-wise analysis evaluates each forecast position independently.
- D.3. Horizon-wise behavior: No reported forecast horizon has a baseline with lower average MSE than DSA.The margin grows over the horizon rather than representing a uniformly large improvement at every position.
E. Phase-Stratified Analysis · F. CMA Faithfulness Analysis · F.1. Perturbation protocol
Phase-stratified results show DSA’s aggregate advantage is not uniform across the epidemic cycle, while the CMA faithfulness analysis tests functional informativeness through targeted masking of attended weeks.
- E. Phase-Stratified Analysis: DSA leads by mean MSE in the off-season, rising, and peak phases, but GPT4MTS is substantially better during decline and marginally better off-season.TaTS also outperforms DSA during the rising phase.
- E. Phase-Stratified Analysis: 6.738 is DSA’s peak-phase mean MSE, versus 11.953 for TaTS, 14.405 for iTransformer, and 28.591 for GPT4MTS.The peak phase has the largest reported errors for all four models.
- E. Phase-Stratified Analysis: −0.19 is the reported Cohen’s d for DSA relative to TaTS in the rising phase, where TaTS’s mean MSE is approximately 16.8% lower.This result qualifies DSA’s phase-specific advantage despite its aggregate performance.
- E. Phase-Stratified Analysis: −1.13 is the reported Cohen’s d for GPT4MTS relative to DSA during the declining phase, where GPT4MTS has substantially lower error.This phase-specific result contrasts with DSA’s stronger performance in other regimes.
- F. CMA Faithfulness Analysis: The CMA faithfulness analysis tests whether attention-selected information is functionally used by comparing forecast changes after masking selected weeks.Attention weights alone do not establish that attended information affects the forecasting function.
- F.1. Perturbation protocol: For each test window, weeks are grouped as Top-k, Random-k, or Bottom-k according to their CMA attention weights.Top-k selects the largest weights, Random-k selects random weeks, and Bottom-k selects the smallest weights.
- F.1. Perturbation protocol: k=3 masks 3 of 36 historical weeks, approximately 8.3% of the lookback window, while additional budgets from k=1 through k=12 are evaluated.Forecasts are recomputed after masking, and the MSE change relative to the unperturbed prediction is recorded separately for both CMA pathways.
F.2. Three-week masking results … CRediT authorship contribution statement
Targeted masking supports the functional relevance of highly attended history, especially in the Text→Numerical pathway, while supplementary analyses reinforce DSA’s robustness but bound claims to the tested benchmark. Integrated Gradients provides complementary feature-level attribution, and the authors’ contributions are explicitly documented.
- F.2. Three-week masking results: For k=3, masking top-attended weeks increased test MSE by 1.4436 in Text→Numerical versus 0.0352 in Numerical→Text.The Text→Numerical effect was approximately 41× larger.
- F.2. Three-week masking results: The asymmetry reflects concentrated narrative content in masked text and may partly reflect Text→Numerical attention’s concentration on recent weeks.The test windows cannot fully disentangle recency from content.
- F.3. Masking-budget sensitivity: Across masking budgets k=1 to k=12, the same qualitative ordering held and effects grew as more historical weeks were removed.These budgets represent approximately 2.8% to 33.3% of the 36-week lookback, but simultaneous masking does not establish individual necessity.
- F.4. Interpretation and limitations: The perturbation experiment tests faithfulness rather than causality: larger masked-input error indicates functional relevance under the selected intervention.Masking can also introduce out-of-distribution inputs, so the result does not isolate causal effects.
- F.4. Interpretation and limitations: Text→Numerical was more functionally informative under this protocol, although attention-driven relevance remains confounded with recency and requires broader temporal data to separate.The largest-attended weeks in this direction were overwhelmingly the most recent weeks.
- F.5. Integrated Gradients: Integrated Gradients complemented CMA perturbation by attributing forecasts to individual numerical and text-derived features rather than complete historical weeks.A global attribution visualization was provided in the main manuscript.
- G. Summary of Supplementary Evidence: Supplementary evidence supports DSA as robust and reproducible for the tested 12-week ILI problem, spanning seeds, ablations, and faithfulness tests rather than one aggregate metric.The authors do not claim universal superiority across every epidemic phase, geography, horizon, or dataset outside the tested benchmark.
- CRediT authorship contribution statement: Seyed Mohammad Hossein Hashemi contributed conceptualization, methodology, investigation, software, visualization, and original-draft writing; Mohsen Hooshmand and Parvin Razzaghi contributed supervision and review/editing.Both Hooshmand and Razzaghi also contributed conceptualization, methodology, validation, and resources or related study support.