Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis

1 Introduction

• Most existing TSA models use numerical series alone, although human experts integrate numerical data with multimodal domain knowledge.Source: §1 · p. 1

exogenous textual signals effectively, thereby enhancing current TSA tasks and enabling new applications?

• The potential of exogenous textual signals, including concurrent events and policies, remains untapped in multimodal TSA.Source: §1 · p. 1
• Existing multimodal TS datasets are limited by narrow domains, coarse modality alignment, and contamination from predictions or text potentially included in LLM pretraining.The paper identifies these gaps as obstacles to comprehensive, high-quality multimodal TS data.Source: exogenous textual signals effectively,… · p. 2
• Time-MMD covers nine domains, aligns modalities at fine granularity, and separates facts from predictions while setting cutoffs through May 2024.Source: exogenous textual signals effectively,… · p. 2
• MM-TSFlib provides an end-to-end pipeline for multimodal TSF and supports integrating open-source language models with arbitrary TSF models.Source: exogenous textual signals effectively,… · p. 2

2 Multi-Domain Multimodal Time-Series Dataset: Time-MMD

• The construction pipeline gathers numerical series, collects and filters relevant text, then uses binary timestamps to align modalities for downstream tasks.Text sources are selected by humans and raw text is filtered with LLMs.Source: §2 · p. 3, Figure 1 · p. 3

2.1 Numerical Series Data Construction

• Time-MMD’s numerical data spans from 1950 to the present and exhibits domain-specific patterns such as periodicity and trends.Source: §2.1 · p. 3

2.2 Textual Series Data Construction

• LLM preprocessing filters irrelevant text, separates facts from predictions, and summarizes content to improve usability.In the health example, extracted facts emphasize relevant terms, while discarded text includes research-paper terms and predictions include future-oriented terms.Source: §2.2 · pp. 4–5
• Search text counts rise over time, report counts stabilize in recent years, and textual-data sparsity varies across domains.High-profile fields often have richer text data.Source: Figure 3 · p. 5, §2.2 · p. 5
• Report data has higher relevance but lower coverage than search data, so the two sources complement each other.Relevance is the percentage of texts with relevant content; coverage is the proportion of numerical data covered by at least one fact.Source: §2.2 · p. 5, Table 2 · p. 5

2.3 Binary Time Stamps for Diverse TSA Tasks

• Binary timestamps store manually verified start and end dates, enabling the text data to serve different tasks.Source: §2.3 · p. 5

2.4 Considerations for Fairness and Data Release

• The Health domain includes US and African data, but African data has weaker numerical periodicity and considerably fewer reports.Source: §2.4 · p. 6

3.1 Problem Formulation

• Multimodal TSF maps numerical and textual series inputs to future values, with the text lookback length independent of the numerical lookback length.The formulation denotes the numerical input as X and textual input as S, and maps their combination to forecasts Y.Source: §3.1 · p. 6

3.2 Pioneering Solution for Multimodal TSF

• The framework independently models numerical and textual series, then combines their outputs with learnable linear weights in end-to-end training.LLM parameters are frozen while projection layers are trained; pooling addresses inconsistent text-variable dimensions, with minimal trainable-parameter overhead.Source: §3.2 · pp. 6–7, Figure 5 · p. 7
• MM-TSFlib extends over 20 unimodal TSF algorithms through 7 open-source language models.The supported models include BERT, GPT-2, Llama-2-7B, and Llama-3-8B.Source: §3.2 · p. 7

4.1 Experimental Setup

• Experiments covered all 9 Time-MMD domains, four horizon lengths per dataset, and 12 TSF methods in both unimodal and multimodal versions.GPT-2-Small was the default LLM backbone; MSE was the evaluation metric.Source: §4.1 · p. 7

4.2 Experimental Results

• Multimodal versions beat their unimodal counterparts in 95% of over 1,000 experiments, reducing mean squared error by over 15% on average and up to 40% in text-rich domains.Figure 6 summarizes average normalized MSE by TSF backbone; the authors report that multimodal gains varied by backbone.Source: Figure 6 · p. 7, §4.2 · p. 8
• Leading unimodal performance from models such as iTransformer and PatchTST supports the quality of Time-MMD’s numerical data, while multimodal gains support its textual data.The authors use Figure 6 to support both observations.Source: Figure 6 · p. 7, §4.2 · p. 8
• Across domains, relevant-fact count generally correlated positively with MSE reduction, while domain characteristics also mattered; the security domain benefited less amid future unpredictability.Figure 7a presents the relationship between relevant fact count and reduced MSE.Source: §4.2 · p. 8, Figure 7 · p. 8
• Multimodal MSE reductions remained stable and promising across short- and long-term forecast horizons.Figure 7c relates horizon window size to average MSE reduction by domain.Source: §4.2 · p. 8, Figure 7 · p. 8
• LLM backbone choice showed no significant correlation with multimodal performance, and GPT-2 scaling showed no clear positive performance relationship.Performance was relatively similar across LLMs, including Llama-3-8B and BERT; the authors suggest projection-only tuning may not fully use LLM capabilities.Source: §4.2 · pp. 8–9, Figure 7 · p. 8
• Doc2Vec was effective but generally performed worse than BERT on three datasets across four horizons, and was included to broaden applicability to underrepresented languages and domains.The authors position Doc2Vec as an alternative for unfamiliar domains or unsupported languages.Source: §4.2 · p. 9
• The authors conclude that multimodal TSF methodology still has significant room for improvement.Source: §4.2 · p. 9

• Time-MMD can serve as a multimodal imputation dataset by pairing numerical series with textual context such as incident reports, weather, and special events.The authors suggest this integration may enhance imputation accuracy.Source: §5 · p. 9
• Time-MMD can support multimodal anomaly detection, including labeling influenza data as anomalous when aligned text mentions a flu outbreak.Source: §5 · p. 9
• The authors expect Time-MMD to support multimodal foundation time-series models, more informative prompts, and research on models tailored to time-series analysis.They describe this area as less explored than vision and video generation.Source: §5 · p. 9

• Collection was restricted to content without premium access or subscription requirements, and web searches used Google’s official API.The authors state these measures were intended to comply with ethical standards and website policies.Source: §6 · pp. 9–10

7 Conclusion

• Time-MMD is presented as the first multi-domain multimodal time-series dataset, alongside MM-TSFlib, a library for multimodal forecasting experiments.The library facilitates a pilot study of multimodal time-series analysis on Time-MMD.Source: §7 · p. 10
• Experiments demonstrate the high quality of Time-MMD and the effectiveness of MM-TSFlib.The paper also reports benefits from integrating textual information into time-series analysis.Source: §7 · p. 10
• The authors envision the dataset and library helping shift time-series analysis from unimodal to multimodal approaches using natural language.Source: §7 · p. 10

A Additional Related Dataset Work

• Existing datasets primarily address stock analysis in finance, while agriculture and security remain challenging domains with substantial real-world impact.Source: A Additional Related Dataset Work · p. 16
• Other multimodal datasets address traffic demand prediction, news impact prediction, and electrocardiogram classification, but are described as not addressing the identified gaps.Source: A Additional Related Dataset Work · p. 16
• Image-plus-series datasets typically use spatial imagery and focus on spatiotemporal analysis, unlike this work’s focus on time-series analysis.Source: A Additional Related Dataset Work · p. 16

B Limitations

• All text in Time-MMD is English, and the work focuses on forecasting rather than other tasks such as classification or equation discovery.Multilingual extensions, additional task curation, and datasets combining text and numerical data with images or audio are identified as future directions.Source: B Limitations · p. 16
• MM-TSFlib uses a simple projection-layer framework for LLM fine-tuning, and the authors describe it as a first-cut rather than an optimal approach.They identify more efficient fine-tuning, text distribution shift, and alternatives such as retrieval-augmented generation as future-work directions.Source: B Limitations · p. 16
• The authors hope the dataset and library will facilitate broader multimodal time-series research and applications, including benchmarks and agents.Source: B Limitations · p. 16

C Considerations for Domain Selection

• Domain selection balances comparability in established domains with coverage of overlooked social-good, agriculture, and security domains, including underrepresented groups.The Health (Africa) dataset is considered alongside Health (U.S.).Source: C Considerations for Domain Selection · p. 16
• The selected domains differ in their numerical data, textual data, and model performances; the authors describe Time-MMD as the most domain-inclusive multimodal time-series dataset to their knowledge.Source: C Considerations for Domain Selection · p. 17

D.1.1 Numeric Data

• The broiler component uses USDA retail broiler composite data and reports describing national market conditions, including price and supply information.Source: §D.1.1 · p. 17, Report: Source 1 · p. 17

D.2 Climate Domain Dataset Sources

• The climate domain uses historical NOAA data, while its listed monthly text sources include drought and national climate reports.Source: §D.2 · p. 17, Report: Source 1 · p. 18, Report: Source 2 · p. 18

D.4 Energy Domain Dataset Sources

• Energy, social-good, and public-health examples pair gasoline, race-disaggregated unemployment, and influenza-like-illness series with reports or related news.Source: §D.4.1 · p. 18, §D.5.1 · p. 19, §D.6.1 · p. 20

E Statistics of Textual Data

• The report and search-data tables cover per-domain relevance, coverage, and token counts; security reports with hand-written summaries bypassed LLM preprocessing.The provided passages identify the tables’ scope but do not give their values.Source: Table 3 · p. 23, Table 4 · p. 23, E Statistics of Textual Data · p. 23

F Prompt designed for LLM preprocessing; G Showcase of Raw and Preprocessed Text

• The appendix presents a domain-instructed LLM prompt and a climate-report example, with the authors observing accurate factual summarization after preprocessing.Figure 8 contains the prompt template; Figure 9 shows text before and after preprocessing.Source: F Prompt designed for LLM preprocessing · p. 24, Figure 8 · p. 24, G Showcase of Raw and Preprocessed Text · p. 25

H Manual Verification of LLM Preprocessing

• Manual inspection found a 3/127 true hallucination rate and a 4/52 error discard rate in LLM preprocessing.Of 8 fabricated facts, 5 were labeled with LLM as the data source and could be filtered out.Source: H Manual Verification of LLM… · p. 25

I Case Study of Semantic Alignment

• A February 2008 influenza report aligns an upward health-data trend with a report noting deaths above the epidemic threshold for a fourth consecutive week.The authors say such explicit alignments can be readily found in the dataset.Source: I Case Study of Semantic Alignment · p. 26

J Details of MM-TSFlib

• MM-TSFlib extends TSlib, uses Hugging Face for LLM invocation, and supports multimodal extensions of over 20 forecasting algorithms with 7 open-source LLMs.The library constrains the latest end date of input text to precede that of the numerical input to avoid information leakage.Source: J Details of MM-TSFlib · p. 26

K.1 Time-Series Forecasting Backbones

• The documented forecasting backbones include Transformer-based models, DLinear, FiLM, and Time-LLM, spanning sequence, linear, model-agnostic, and LLM-based approaches.The supplied descriptions cover Transformer, Reformer, Informer, Autoformer, Crossformer, Non-stationary Transformer, FEDformer, iTransformer, DLinear, FiLM, and Time-LLM.Source: §K.1 · p. 27

J.2 List of Supported LLM; K.2 LLM Backbones

• Most experiments use GPT-2 Small, while the LLM ablation includes other GPT-2 sizes, BERT, Llama-2, and Llama-3.The appendix also describes BERT, GPT-2, Llama-2, and Llama-3.Source: §K.2 · p. 27

K.3 Evaluation Metrics; K.4 Details of Implementation

• Forecasting uses MSE across daily, weekly, and monthly horizons; text pairing traces back the latest k samples and is described as a preliminary approach.The horizon windows vary by reporting frequency, with fixed lookback and decoder label windows; the authors anticipate more effective matching approaches.Source: §K.3 · p. 28, §K.4 · p. 28

L Additional Experiments on Text Integration Approaches

• The Time-LLM “prompt as prefix” approach is reported as unsuitable for textual series, while the authors’ framework is described as effective.The authors suggest longer text series may overwhelm numerical-series information as one possible reason.Source: L Additional Experiments on Text… · p. 28

M Additional Experiments on Text Modeling Approaches

• Using only an LLM’s embedding layer often yields results close to or better than using its last layer.The authors say this leaves room for improvement in LLM use for time-series forecasting.Source: L Additional Experiments on Text… · p. 28

N.1 Using attention mechanisms

• A basic attention-based weighting of LLM-output tokens has not demonstrated a clear advantage over the current simple combining approach.The method calculates token attention scores from the time-series model output and aggregates the tokens with those weights.Source: §N.1 · p. 29

N.2 Using closed-source LLMs

• Fully frozen closed-source LLMs were initially observed to be less effective than fine-tuned open-source LLMs.The experiment generated text-based predictions with closed-source LLMs such as GPT-3.5 and then encoded them using BERT; results are referenced in Table 8.Source: §N.2 · p. 29, Table 8 · p. 29
• Using LLM encoder layers usually yields better performance.Source: §N.2 · p. 29

O Detailed Results

• Detailed-result tables are listed for Agriculture, Climate, Economy, Energy, Environment, Health (US), Security, SocialGood, and Traffic.The provided table passages give their domain labels, not cell values or comparative outcomes.Source: Table 9 · p. 30, Table 10 · p. 31, Table 11 · p. 32

P Dataset Documentations

• Numerical records include time bounds, target and other variable values, while text records include time bounds and extracted fact and prediction text with source information.The dataset is provided in CSV format.Source: P Dataset Documentations · p. 39

Q Intended Uses

• Time-MMD is intended for researchers developing multimodal time-series analysis methods and developers training or fine-tuning multimodal foundation time-series models.Source: Q Intended Uses · p. 39

T Hosting and Maintenance Plan; U Time-MMD Community

• The authors plan six-month cutoff-time updates and future expansion to more languages, domains, and multimodal analysis tasks, with external contributions welcomed.They describe Time-MMD as community-driven and open source.Source: T Hosting and Maintenance Plan · p. 39, U Time-MMD Community · p. 39

V Licensing

• The dataset uses ODC-By v1.0, and the authors state that sources are public domain and authorized for non-commercial distribution.Source: V Licensing · pp. 39–40