PaperlayerView exact revision on arXiv

Source-linked AI summary

QiYao-M: Multimodal Time Series Foundation Model with Role-Aware Modeling of Endogenous and Exogenous Modalities

Hanyin Cheng, Linfeng Wang, Zhengbo Qu, Yang Shu, Zhongwen Rao, Meng Wang, Yijie Li, Xin Jiang, Bin Yang, Chenjuan Guo

arXiv:2609.34842v1cs.LG

TL;DR

Existing multimodal TSFMs often use shared mechanisms that overlook the distinct forecasting roles of endogenous and exogenous modalities. QiYao-M models them separately, using explicit endogenous-evolution supervision and exogenous retrieval with proxy training. It reports strong performance across benchmarks with and without exogenous modalities.

  • Problem

    Existing multimodal TSFMs use largely shared mechanisms despite endogenous modalities evolving with the series and exogenous modalities varying across domains and configurations amid scarce pretraining data.

    Source: §1 · pp. 1–2

  • Method

    QiYao-M explicitly predicts and supervises endogenous evolution, while retrieval uses historical exogenous-modality cases and endogenous proxies without updating TSFM parameters.

    Source: §1 · p. 2, §3.2.3 · p. 7

  • Results

    Across unimodal and multimodal benchmarks, QiYao-M reports strong performance; on multimodal metrics it ranks first on 14/18 and in the top two on 17/18.

    Source: §1 · p. 3, §4.2 · p. 8

  • Takeaways & Limitations

    Experiments on five benchmarks show strong cross-domain generalization in scenarios both with and without exogenous modalities.

    Source: §5 · p. 9

Abstract

from arXiv · show

Existing multimodal time series foundation models (TSFMs) typically model heterogeneous modalities through largely shared mechanisms, overlooking the distinct forecasting roles of endogenous and exogenous modalities. In this work, we propose QiYao-M, a role-aware multimodal TSFM that models the two types of modalities separately. For endogenous modalities, to capture how they evolve along with the underlying temporal dynamics, we introduce an Endo-Multimodal Predictor and Endo-Multimodal Supervision to explicitly learn their evolution from history to the future. For exogenous modalities, to generalize across domains and across various modality types and numbers under the scarcity of exo-multimodal pretraining data, we propose an Exo-Multimodal Retrieval Enhancer that enables rapid downstream adaptation without updating the TSFM parameters. We further introduce Endo-Modality Proxy Training to train this retrieval module without exogenous multimodal pretraining data. Extensive experiments across unimodal and multimodal benchmarks demonstrate strong forecasting performance in scenarios both with and without exogenous modalities.

1 INTRODUCTION

QiYao-M addresses the mismatch between shared multimodal modeling and the distinct forecasting roles of endogenous and exogenous modalities. It explicitly models endogenous evolution and adapts to diverse exogenous modalities through retrieval.

1 INTRODUCTION

  • Existing multimodal TSFMs use largely shared mechanisms despite endogenous and exogenous modalities having distinct forecasting roles.Endogenous modalities evolve with the time series, while exogenous modalities vary across domains and modality configurations.Source: §1 · pp. 1–2
  • Endogenous modalities are intrinsically coupled to temporal dynamics, but existing models mainly use them as historical context without explicitly learning their evolution.Numerical forecasting losses evaluate final forecast values but provide no modality-specific supervision for future endogenous information.Source: §1 · pp. 1–2
  • Exogenous modalities pose generalization challenges because their effects can differ across domains and forecasting scenarios vary in modality types and numbers.Scarce exogenous multimodal pretraining data further limits learning transferable knowledge across scenarios.Source: §1 · p. 2
  • QiYao-M uses patch-wise historical fusion and predicts endogenous modalities in the forecast window, with supervision aligning predictions to ground-truth future representations.The Endo-Multimodal Predictor and Endo-Multimodal Supervision explicitly model endogenous evolution from history to the future.Source: §1 · pp. 2–3
  • For exogenous modalities, QiYao-M retrieves historical cases per input modality and uses their future responses as forecasting evidence, without updating TSFM parameters.Endo-Modality Proxy Training uses available endogenous modalities as proxies to train the retrieval module without exogenous multimodal pretraining data.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work adapts multimodal forecasting models to tasks or datasets, while TSFMs learn transferable temporal patterns through large-scale pretraining. Existing multimodal TSFMs still commonly rely on text and role-agnostic modeling.

2.1 MULTIMODAL TIME SERIES FORECASTING

  • Multimodal forecasting approaches use language-model adaptation or auxiliary text, but much of this work relies on task- or dataset-specific adaptation that limits transfer to unseen domains.Examples include bridging temporal and language representations, or incorporating news and contextual reports.Source: §2.1 · p. 3

2.2 TIME SERIES FOUNDATION MODELS

  • TSFMs use large-scale pretraining to learn transferable temporal patterns and support zero-shot cross-domain forecasting.Recent models extend capabilities through scaling, covariate-informed forecasting, streaming, and multiple forecasting tasks.Source: §2.2 · p. 3
  • Recent multimodal TSFMs primarily support exogenous text and typically use shared mechanisms rather than modeling endogenous and exogenous roles separately.The cited models struggle to generalize across different types and numbers of exogenous modalities.Source: §2.2 · p. 3

3 QIYAO-M

QiYao-M assigns endogenous and exogenous modalities distinct modeling and training strategies. It fuses endogenous views patch-wise, predicts their future representations, and uses retrieval to incorporate available exogenous information.

3 QIYAO-M

  • QiYao-M uses dedicated mechanisms for endogenous and exogenous modalities, reflecting their distinct forecasting characteristics.Source: §3 · p. 3, §3.1 · p. 4

3.1.1 ENDO-MULTIMODAL FUSION

  • For each patch, statistical descriptions and multi-order-difference plots form Endo-Text and Endo-Image, which are CLIP-encoded and fused with time-series representations.Source: §3.1.1 · pp. 4–5

3.1.2 EXO-MULTIMODAL RETRIEVAL ENHANCER

  • The exogenous retrieval enhancer analyzes modality contributions, retrieves cases independently per modality, reranks merged candidates, and incorporates selected cases into the forecast representation without updating TSFM parameters.Source: §3.1.2 · pp. 5–6

3.1.3 ENDO-MULTIMODAL PREDICTOR

  • The Endo-Multimodal Predictor jointly forecasts the future series and endogenous-modality representations, avoiding direct generation of text tokens or image pixels.Source: §3.1.3 · p. 6

3.2 TRAINING STRATEGY

  • Training proceeds through numerical-only, endogenous multimodal, and retrieval stages; the first trains the temporal encoder, Transformer backbone, and quantile head without multimodal modules.Source: §3.2 · p. 6, §3.2.1 · p. 6

3.2.2 ENDO-MULTIMODAL TRAINING

  • Endogenous multimodal training freezes CLIP and supervises predicted future modality representations derived from the ground-truth future, alongside the numerical forecasting objective.Source: §3.2.2 · p. 7

3.2.3 RETRIEVAL TRAINING WITH ENDO-MODALITY PROXIES

  • Endo-Modality Proxy Training dynamically samples endogenous modality types and numbers as retrieval proxies, training the enhancer without exogenous multimodal pretraining data.Source: §3.2.3 · p. 7

4 EXPERIMENTS

Across unimodal and multimodal benchmarks, QiYao-M performs strongly; component analyses report improvements from its proposed mechanisms, with the complete model best across variants.

4 EXPERIMENTS

  • Experiments cover GIFT-Eval and TIME unimodal benchmarks, three multimodal datasets, and further model analyses.The multimodal datasets are Time-MMD, MoTime, and FinMultiTime.Source: §4 · p. 7
  • QiYao-M achieves state-of-the-art MASE on unimodal benchmarks and MSE on multimodal benchmarks.Figure 4 summarizes performance in scenarios both with and without exogenous modalities.Source: Figure 4 · p. 7, §4 · p. 7

4.1 PERFORMANCE ON UNIMODAL BENCHMARKS

  • GIFT-Eval MASE falls 1.0% and CRPS falls 1.3% versus TiRex-2-Pretrained; TIME MASE/CRPS are 0.639/0.537 versus Toto-2.0-2.5B’s 0.642/0.539.QiYao-M achieves the best MASE and CRPS on both benchmarks.Source: Figure 5 · p. 8, §4.1 · p. 8

4.2 PERFORMANCE ON MULTIMODAL BENCHMARKS

  • 14/18 metrics rank first and 17/18 rank within the top two against baseline foundation and end-to-end models.Table 1 reports average forecasting results on multimodal datasets.Source: §4.2 · p. 8, Table 1 · p. 9
  • 5.1% average MSE reduction follows adding exo-modalities; average MSE is also 4.3% lower than the strongest unimodal TSFM and 14.6% lower than existing multimodal TSFMs.The 14.6% reduction concerns diverse exogenous modality types and numbers.Source: §4.2 · p. 8, Table 1 · p. 9

4.3 MODEL ANALYSIS

  • The complete QiYao-M performs best across all benchmarks; ablations report gains from endo fusion, endo supervision, retrieval enhancement, and cases analysis.Retrieval-mechanism sensitivity is analyzed separately in Appendix C.1.Source: §4.3 · pp. 8–9, Table 2 · p. 9

5 CONCLUSION

QiYao-M assigns distinct strategies to endogenous and exogenous modalities according to their forecasting roles. Experiments across five benchmarks show strong cross-domain generalization with and without exogenous modalities.

5 CONCLUSION

  • QiYao-M distinguishes endogenous and exogenous modalities according to their forecasting roles.The model is presented as a role-aware multimodal time series foundation model.Source: §5 · p. 9
  • QiYao-M models endogenous temporal evolution and uses historical-case retrieval for diverse exogenous modality types and numbers without updating TSFM parameters.Source: §5 · p. 9
  • Experiments on five benchmarks demonstrate strong cross-domain generalization both with and without exogenous modalities.Source: §5 · p. 9

A EXPERIMENTAL DETAILS

QiYao-M’s pretraining corpus combines broad unimodal time-series sources with multimodal samples tailored to its training objectives. The multimodal data span patch-level endogenous representations and sequence-level exogenous proxies.

A.1 PRETRAINING CORPUS

  • The unimodal pretraining corpus combines GIFT-Eval Pretrain, GIFT-Eval training data, and the Chronos corpus for broad coverage across heterogeneous domains.Only training portions of GIFT-Eval datasets are used, and evaluation portions are kept separate to avoid test-set leakage.Source: §A.1 · p. 14
  • The multimodal corpus contains approximately 5 million patch-level endogenous samples and 10 million sequence-level samples used as exogenous proxies.Source: §A.1 · p. 14
  • Patch-level endogenous samples pair localized time-series patterns with aligned auxiliary text and images to establish fine-grained cross-modal correspondence.They support cross-modal representation learning and recovery of temporal structures from complementary representations.Source: §A.1 · p. 14
  • Sequence-level samples pair text and images with whole time series, providing contextual signals that proxy downstream exogenous information.This design exposes pretraining to heterogeneous context beyond local numerical patterns.Source: §A.1 · p. 14

A.2 MULTIMODAL DATASET GENERATION METHOD

QiYao-M constructs deterministic text and image views of numerical histories at patch and sequence granularity. The aligned artifacts preserve the forecasting information boundary while supporting endogenous supervision and proxy training for retrieval.

A.2.1 PATCH-LEVEL ENDOGENOUS MODALITIES

  • Patch-level text maps mask-aware statistics to fixed linguistic bins and a deterministic template, keeping statements traceable to numerical evidence.Thresholds are fixed before evaluation; data-dependent thresholds are estimated from the training split only.Source: §A.2.1 · p. 15
  • Patch images encode values, first differences, and second differences in RGB channels, while missing or padded positions remain blank and interrupt curves.Differencing is valid only when all required values are valid.Source: §A.2.1 · pp. 15–16

A.2.2 SEQUENCE-LEVEL ENDOGENOUS MODALITIES

  • Sequence-level modalities capture longer-term behavior and serve as retrieval proxies rather than direct numerical-backbone inputs, enabling proxy training without naturally paired exogenous data.They simulate heterogeneous exogenous information associated with historical series.Source: §A.2.2 · pp. 16–17
  • Sequence-level text summarizes global statistics and chronological patch-regime transitions, including changes in trend, variation, or turning behavior.The template also records salient jumps and dominant periodicity.Source: §A.2.2 · p. 17
  • Sequence-level images provide complementary views: line curves show full-window dynamics, heatmaps compare patch levels on a shared scale, and spectra expose periodic and multi-scale behavior.Valid intervals remain separate around missing data; spectrum estimates combine contiguous valid blocks without interpolating across gaps.Source: §A.2.2 · pp. 17–18

A.2 MULTIMODAL DATASET GENERATION METHOD

  • Identical numerical inputs and masks yield reproducible multimodal examples, with generation statistics estimated from training data to preserve the forecasting information boundary.The stored artifacts include patch- and sequence-level text and images, source identifiers, and generation-version identifiers.Source: §A.2 · p. 14, §A.2.2 · p. 18

A.3 BENCHMARKS

The evaluation covers unimodal forecasting benchmarks and multimodal datasets with varied external information. It compares QiYao-M with supervised and foundation-model baselines and uses staged training before tuning-free retrieval at inference.

A.3 BENCHMARKS

  • Unimodal evaluation uses GIFT-Eval and TIME across 73 datasets, while exogenous forecasting covers nine datasets with time series, text, images, and tabular information.The multimodal sets come from Time-MMD, MoTime, and FinMultiTime and use benchmark-specific forecasting horizons.Source: §A.3 · pp. 18–19

A.4 BASELINES

  • Comparisons include four supervised multimodal models, five unimodal TSFMs, and two multimodal foundation models, with leaderboard results used for GIFT-Eval and TIME.The paper identifies the baseline repositories and implementation details in Table 6.Source: §A.4 · p. 19

A.5 PRETRAINING SETTINGS

  • Training proceeds from numerical pretraining to endogenous multimodal training and then exogenous retrieval training; inference selects reference cases without updating model parameters.During retrieval training, the numerical backbone and endogenous modules are frozen, and test targets are excluded from training and calibration.Source: §A.5 · pp. 19–20

A.6 EVALUATION METRICS

The appendix specifies evaluation metrics and model size, then situates QiYao-M among forecasting, multimodal forecasting, and foundation-model research. It identifies role-agnostic modeling and limited modality generalization as gaps addressed by QiYao-M.

A.6 EVALUATION METRICS

  • Point forecasts are evaluated with MSE and MAE, probabilistic forecasts with MASE and CRPS when required, and all metrics exclude invalid or padded targets.Scale-normalized errors use training-interval statistics rather than test-set statistics.Source: §A.6 · pp. 20–22

A.7 MODEL CONFIGURATIONS

  • QiYao-M has 375.63M unique main-model parameters, or 526.90M including the frozen CLIP encoder.The main-model count excludes the frozen encoder.Source: §A.7 · pp. 22–23

B.1 TIME SERIES FORECASTING

  • Time-series forecasting research spans multivariate dependencies, irregular sampling, non-stationarity, exogenous variables, probabilistic forecasts, and benchmark evaluation.The passage also frames forecasting alongside anomaly detection and reasoning-oriented tasks.Source: §B.1 · p. 23

B.2 MULTIMODAL TIME SERIES FORECASTING

  • Multimodal forecasting combines temporal and language representations and auxiliary text, with visual representations also explored; much prior work uses task- or dataset-specific adaptation.The passage notes that this can limit transfer of multimodal knowledge to unseen domains.Source: §B.2 · p. 23

B.3 TIME SERIES FOUNDATION MODELS

  • Existing multimodal TSFMs often use role-agnostic mechanisms and struggle with varied exogenous modality types and numbers; QiYao-M distinguishes modality roles and retrieves reference cases as in-context demonstrations.The stated design supports tuning-free acquisition of domain-specific knowledge and flexible adaptation to arbitrary exogenous modalities.Source: §B.3 · p. 24

C.1 RETRIEVAL ANALYSIS

On GIFT-Eval, QiYao-M receives favorable MASE and CRPS rankings. Retrieval analysis reports strong performance with a small bank and limited sensitivity to bank size and top-k.

C.1 RETRIEVAL ANALYSIS

  • 5% of available training samples suffices for QiYao-M to outperform the internal baseline and strong foundation models; around 20% offers a favorable performance-efficiency trade-off.Forecasting errors remain within a narrow range as top-k varies from 4 to 24.Source: §C.1 · p. 25, Figure 11 · p. 25

D.1 GIFT-EVAL RANKINGS

  • QiYao-M achieves favorable GIFT-Eval rankings in both MASE and CRPS, with the reported comparisons shown in Figures 12 and 13.The authors describe improvements over baseline forecasting models across diverse time-series scenarios.Source: Figure 12 · p. 25, §D.1 · p. 25, Figure 13 · p. 26

D.2 FULL RESULTS

The full results report performance at each forecasting horizon and on average, with consistent effectiveness across horizons and multimodal benchmarks.

D.2 FULL RESULTS

  • The step-wise results demonstrate consistent effectiveness across forecasting horizons and multimodal benchmarks.The reported comparisons include each horizon and average performance.Source: §D.2 · p. 26
  • Tables 8–12 report step-wise results for Agriculture, Climate, Energy, Health, and Social Good using MSE and MAE.Lower values indicate better performance; purple and blue mark the best and second-best results, respectively.Source: Table 8 · p. 26, Table 9 · p. 27, Table 10 · p. 27
  • Tables 13–16 report step-wise results for TAOBAO-fashion, Tianchi, HS300, and SP500 using MSE and MAE.Lower values indicate better performance; purple and blue mark the best and second-best results, respectively.Source: Table 13 · p. 29, Table 14 · p. 29, Table 15 · p. 30
Loading 2609.34842v1…