Source-linked AI summary
Multi-modal Time Series Analysis: A Tutorial and Survey
Yushan Jiang, Kanghui Ning, Zijie Pan, Xuyang Shen, Jingchao Ni, Wenchao Yu, Anderson Schneider, Haifeng Chen, Yuriy Nevmyvaka, Dongjin Song
TL;DR
Multi-modal time series analysis must address heterogeneous modalities, modality gaps, temporal misalignment, and noise. This survey systematically reviews datasets, applications, and methods through a fusion, alignment, and transference framework, while identifying missing and noisy modalities as continuing challenges.
Problem
Multi-modal time series analysis is hindered by heterogeneous data, modality gaps, temporal misalignment, and noisy real-world contexts.
Method
The survey organizes datasets and methods through a unified framework of fusion, alignment, and transference at input, intermediate, and output levels.
Results
The survey provides a comprehensive overview of existing multi-modal time series methods, datasets, real-world applications, and future research directions.
Takeaways & Limitations
Cross-modal interactions provide a systematic perspective for studying multi-modal time series across predictive, analytical, and domain-specific applications.
Takeaways & Limitations
Effective handling of missing and noisy modalities still requires further exploration, including modality-specific imputation, noise reduction, and relevance quantification.
Abstract
from arXiv · showhide
Multi-modal time series analysis has recently emerged as a prominent research area in data mining, driven by the increasing availability of diverse data modalities, such as text, images, and structured tabular data from real-world sources. However, effective analysis of multi-modal time series is hindered by data heterogeneity, modality gap, misalignment, and inherent noise. Recent advancements in multi-modal time series methods have exploited the multi-modal context via cross-modal interactions based on deep learning methods, significantly enhancing various downstream tasks. In this tutorial and survey, we present a systematic and up-to-date overview of multi-modal time series datasets and methods. We first state the existing challenges of multi-modal time series analysis and our motivations, with a brief introduction of preliminaries. Then, we summarize the general pipeline and categorize existing methods through a unified cross-modal interaction framework encompassing fusion, alignment, and transference at different levels (\textit{i.e.}, input, intermediate, output), where key concepts and ideas are highlighted. We also discuss the real-world applications of multi-modal analysis for both standard and spatial time series, tailored to general and specific domains. Finally, we discuss future research directions to help practitioners explore and exploit multi-modal time series. The up-to-date resources are provided in the GitHub repository: https://github.com/UConn-DSIS/Multi-modal-Time-Series-Analysis
1 Introduction
Time series analysis models sequential data with rich temporal dynamics, while multi-modal contexts such as text, images, tables, and graphs add semantic information for understanding real-world systems.
- Time series analysis is a fundamental data-mining task involving sequential data with rich temporal dynamics across diverse real-world systems.
- External contexts—including text, images, tables, and graphs—provide complementary representations for time series analysis.
- Incorporating multi-modal contexts can provide a more comprehensive view of underlying systems and capture subtle dependencies.
Multi-modal Time Series Analysis
Multi-modal time series analysis faces heterogeneity, modality gaps, temporal misalignment, and noisy real-world data. The survey organizes methods around fusion, alignment, and transference and reviews applications and future directions.
- Challenges: Heterogeneous statistical properties, structures, dimensionalities, and semantic meanings make unified cross-modal representations difficult to construct.
- Challenges: Temporal contexts may occur at different timesteps or granularities, hindering meaningful interactions between modalities.
- Applications and motivation: Applications span forecasting, classification, anomaly detection, retrieval, and causal discovery, although existing methods are often task- and domain-specific.
- Survey framework: The survey categorizes cross-modal modeling through fusion, alignment, and transference at input, intermediate, and output levels.
- Contributions: The survey systematically catalogs over 40 multi-modal time series methods with corresponding open-source datasets.
2 Background and Our Scope
The survey frames multi-modal time series analysis as combining temporal data with complementary modalities through cross-modal interactions. It reviews data, interaction mechanisms, and applications while defining a scope centered on real-world contextual information.
- Multi-modal time series analysis combines time series with complementary modalities to support predictive and analytical tasks across real-world contexts.
- The survey reviews multi-modal time series data, analyzes cross-modal interactions, and examines applications across diverse domains.
- Its scope mainly covers standard and spatial time series, with spatial structures paired inherently with temporal data rather than treated as separate modalities.
- The survey focuses on real-world multi-modal inputs that provide complementary information, while generation and retrieval emphasize transforming input modalities into output modalities.
- Compared with narrower surveys on imaging-based transformations or multimodal LLM reasoning, this survey presents a broader structured framework for multi-modal time series.
3 Multi-modal Time Series Data
Multi-modal time series datasets combine temporal measurements with modalities such as tabular records, text, images, audio, and spatial-temporal context. Their characteristics and domains vary substantially.
- Modalities: Multi-modal time series data commonly combines temporal measurements with tabular records, text, images, audio, or other contextual modalities.
- Domains: Representative datasets span healthcare, finance, climate, energy, traffic, environment, speech, and other application domains.
- Healthcare: Healthcare datasets pair physiological signals with clinical notes, demographics, vital signs, laboratory results, and other medical information.
- Finance: Financial datasets combine stock-price histories with tweets, financial news, or reports; FNSPID covers over 4,000 companies from 1999 to 2023.
- Multi-domain datasets: Multi-domain datasets align general-purpose numerical time series with textual information across domains including agriculture, climate, economy, energy, health, security, social good, and traffic.
- Other domains: Other datasets integrate sales with product images and descriptions, audio with visual data, spatial-temporal traffic records with text, or geographic spatial-temporal data with textual descriptions.
4 Cross-modal Interactions with Time Series
The survey organizes cross-modal interactions into fusion, alignment, and transference across input, intermediate, and output stages. These interactions integrate modalities, preserve cross-modal relationships, and map information between modalities for time-series analysis.
- The taxonomy defines fusion, alignment, and transference across input, intermediate, and output stages.
- 4.1 Fusion: Fusion integrates heterogeneous modalities to capture complementary information for time-series modeling.
- 4.1 Fusion: Intermediate fusion commonly adds or concatenates modality representations after separate encoders map raw data into a shared latent space.
- 4.2 Alignment: Alignment addresses imperfect synchronization and uses preprocessing, attention, gating, graph convolution, or learning objectives to preserve cross-modal relationships.
- 4.2 Alignment: Self-attention provides joint undirected alignment, whereas cross-attention uses time series as queries to incorporate contextual information from auxiliary modalities.
- 4.3 Transference: Transference maps one modality to another, supporting augmentation at the input level and task-oriented generation or retrieval at intermediate and output levels.
5 Applications of Multi-modal Time Series Analysis
Multi-modal time-series analysis has been applied across healthcare, finance, transportation, environment, retail, and IoT. The applications combine temporal data with complementary contextual modalities.
- Applications span healthcare, finance, transportation, environment, retail, and the Internet of Things.
Standard Time Series
Standard time-series applications combine temporal data with text, audio, physiological signals, images, or visual information across healthcare, finance, retail, computer vision, and speech. These methods support prediction, classification, retrieval, and generation tasks.
- Healthcare: Healthcare methods combine EHRs, audio, EEG, ECG, wearable recordings, and clinical text for diagnosis, monitoring, classification, and wellness assessment.
- Healthcare: EEG-to-text decoding followed by text classification enables zero-shot EEG-based sentiment classification.
- Finance: Financial methods integrate stock movements, company profiles, and news in structured LLM prompts for stock prediction.
- Finance: Contrastive shared-latent-space alignment supports financial time-series retrieval from textual descriptions or sketched trends and improves retrieval speed and accuracy over SQL search.
- Retail, computer-vision, and speech applications combine temporal data with product descriptions or images, textual structural priors, and correlated audio-visual information.
Spatial Time Series
Spatial time-series applications use contextual information such as weather, geography, and historical flow patterns to support traffic prediction. Environmental applications also use cross-modal information to impute missing stream-image data.
- Transportation: Traffic prediction methods align numerical spatial-temporal data with LLMs and incorporate weather, geographic context, and historical flow patterns.
- Environment: VIMTS imputes missing high-dimensional stream-image modalities from low-dimensional features derived from meteorological time-series records.
6 Future Research Directions
The survey identifies future work spanning reasoning, decision support, domain generalization, robustness to incomplete or noisy modalities, and ethical bias mitigation. These directions target more interpretable, reliable, adaptive, and socially responsible multi-modal time series analysis.
- Reasoning with Multi-modal Time Series: Future systems should integrate temporal reasoning with contextual understanding, external knowledge, and language agents to improve interpretability and reliability.Retrieval-augmented generation and language-agent integration are proposed as possible paths for reasoning beyond immediate data.
- Decision Making: Multi-modal predictive signals and explanations could support more adaptive, interpretable, and reliable decision-support systems for resource allocation and risk management.
- Domain Generalization: Domain generalization must address distribution shifts across time series and other modalities while preserving invariant components and modality-specific variations.Disentangling modality effects is also identified as important for understanding individual contributions and mitigating cross-modal interference.
- Robustness to Missing and Noisy Modalities: Further work is needed on modality-specific imputation, noise reduction, and relevance quantification to improve applicability in incomplete or noisy real-world settings.Existing iterative context refinement can filter less relevant information, but missing and noisy modalities remain open challenges.
- Ethical Considerations and Bias Mitigation: Fairness constraints, counterfactual analysis, adversarial debiasing, and robust bias assessment are proposed to mitigate inequities in multi-modal time series outcomes.
7 Conclusion
The survey comprehensively reviews multi-modal time series methods, organizing them through a cross-modal interaction taxonomy and discussing applications and future directions.
- The survey organizes existing methods by cross-modal interactions between time series and other modalities, while covering datasets, applications, and future research directions.