Source-linked AI summary
ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data
Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, Jianxin Liao
TL;DR
Time-series forecasting systems often lack adaptability, textual processing, efficient training, or zero-shot capability. ChatTime treats time series as a foreign language within an existing LLM, and the paper reports broad multimodal task support with strong zero-shot efficiency and performance.
Problem
Existing time-series predictors are often unimodal and fixed-window, while LLM-based approaches can lack efficient training, textual processing, or zero-shot forecasting.
Method
ChatTime converts time series into a foreign-language representation and adapts a pretrained LLM through vocabulary expansion, continuous pretraining, and instruction fine-tuning.
Results
ChatTime supports zero-shot forecasting and bimodal time-series/text input and output across three tasks, achieving 99.9% of Chronos’s zero-shot accuracy with 4% of the data.
Takeaways & Limitations
The results demonstrate the potential of an efficient unified framework for multimodal time-series analysis across forecasting and question answering.
Abstract
from arXiv · showhide
Human experts typically integrate numerical and textual multimodal information to analyze time series. However, most traditional deep learning predictors rely solely on unimodal numerical data, using a fixed-length window for training and prediction on a single dataset, and cannot adapt to different scenarios. The powered pre-trained large language model has introduced new opportunities for time series analysis. Yet, existing methods are either inefficient in training, incapable of handling textual information, or lack zero-shot forecasting capability. In this paper, we innovatively model time series as a foreign language and construct ChatTime, a unified framework for time series and text processing. As an out-of-the-box multimodal time series foundation model, ChatTime provides zero-shot forecasting capability and supports bimodal input/output for both time series and text. We design a series of experiments to verify the superior performance of ChatTime across multiple tasks and scenarios, and create four multimodal datasets to address data gaps. The experimental results demonstrate the potential and utility of ChatTime.
1 Introduction
Time-series forecasting remains constrained by unimodal, fixed-window approaches, while existing LLM-based methods do not consistently combine textual information, efficient training, and zero-shot capability. ChatTime addresses these gaps by treating time series as a foreign language and supporting multimodal inference across forecasting and question answering tasks.
- Motivation and contribution: Existing deep-learning predictors commonly train on a single dataset with fixed history and prediction windows, limiting adaptation across scenarios.The introduction identifies this fixed-window, single-dataset setup as a limitation of current deep paradigms.
- Motivation and contribution: ChatTime models time series as a foreign language, enabling zero-shot inference and bimodal time-series/text inputs and outputs.The framework converts time-series values into discrete foreign-language words and applies vocabulary expansion to an existing LLM.
- Evaluation scope: ChatTime evaluates time-series-to-time-series, text-to-time-series, and time-series-to-text capabilities through zero-shot forecasting, context-guided forecasting, and time-series question answering.Zero-shot forecasting uses eight real-world benchmark datasets across four domains, while the broader task suite tests multimodal translation.
2 Related Work
Related work applies pretrained LLMs to time series through prompting, architectural adaptation, or new foundation-model training, but these approaches leave gaps in accuracy, efficiency, multimodal processing, and fine-grained forecasting. ChatTime is positioned as an existing-LLM-based alternative that uses time-series language.
- LLM-based time-series analysis: LLM-based time-series research comprises direct use of pretrained weights, integration of those weights into new frameworks, and new models trained with LLM architectures.These three paradigms differ in how much they retain from pretrained models and how they adapt to numerical sequences.
- LLM-based time-series analysis: Direct prompting methods often have low time-series understanding and token utilization, while instruction fine-tuning does not resolve all limitations.The cited discussion attributes the token inefficiency to bit-by-bit tokenization.
- LLM-based time-series analysis: Frameworks that add neural input and output layers can improve forecasting or incorporate context, but most do not provide the full capabilities sought for general time-series analysis.The passage distinguishes forecasting-focused adaptations from methods that use pretrained models as embedding modules for contextual inputs.
- Time-series foundation models: Training new time-series foundation models can yield promising results but is highly inefficient and generally supports only unimodal numerical data.This limitation motivates reusing pretrained LLMs rather than training comparable models from scratch.
- Multimodal approaches: Existing multimodal approaches are limited in domains and tasks, and chart-based methods do not support fine-grained time-series forecasting.The related-work discussion identifies fine-grained forecasting as a central task that these approaches omit.
3 Methodology
ChatTime converts normalized and discretized time series into a tokenizer-compatible foreign language, processes them with an expanded pretrained LLM, and decodes outputs back into text or numerical sequences. Its training combines continuous autoregressive pretraining with instruction fine-tuning on unimodal and multimodal task data.
- 3.1 Overview: ChatTime translates time series into foreign-language words through normalization, discretization, mark characters, vocabulary expansion, and inverse decoding.The tokenizer maps text and foreign words to indexes, while the de-tokenizer and inverse normalization recover textual or time-series outputs.
- 3.2 Model Architecture: History-based min-max scaling maps values into a buffered range, reserving space for predictions that may exceed the observed history.The history is scaled to -0.5 to 0.5 within an overall -1 to 1 range because future values are unavailable at inference.
- 3.2 Model Architecture: Uniformly partitioning -1 to 1 into 10K bins converts continuous values into discrete tokens, while mark characters allow one vocabulary token per value regardless of precision.The method fixes discretized precision to 4 and adds a dedicated token for missing values.
- 3.3 Continuous Pre-Training: Continuous pretraining uses 1M high-quality time-series slices and autoregressive forecasting, with data drawn from roughly 100 sub-datasets while excluding evaluation data.Sliding slices with five window and step-size settings support history and prediction windows of any size.
- 3.4 Instruction Fine-Tuning: Instruction fine-tuning uses 100K instances across four tasks, including text question answering, unimodal forecasting, context-guided forecasting, and time-series question answering.The multimodal datasets combine time series with background, weather, date, or generated question-answer information while restricting auxiliary context to avoid future leakage.
4 Experiment
Across unimodal forecasting, context-guided forecasting, and time series question answering, ChatTime is evaluated against specialized and foundation-model baselines, with ablations testing its design choices.
- 4.2 Zero-Shot Time Series Forecasting: ChatTime achieves 99.9% of Chronos’s zero-shot prediction accuracy using 4% of the data and 90.9% of GPT4TS’s full-shot accuracy.The comparison uses average MAE and average Rank across eight datasets.
- 4.3 Context-Guided Time Series Forecasting: ChatTime surpasses TGForecaster in context-guided forecasting and significantly outperforms its unimodal variant without textual input.The reported comparison attributes the advantage to integrating textual and numerical modalities.
- 4.4 Time Series Question Answering: ChatTime outperforms generic large language models in time series question answering while preserving their text inference capabilities.Results compare average accuracy and average rank across four time series features.
- 4.5 Ablation Study: Removing continuous pre-training sharply reduces zero-shot inference capability, although CGTSF and TSQA improve slightly in this ablation.Replacing the 1M pre-training dataset with the 100K fine-tuning dataset also produces overfitting.
- 4.5 Ablation Study: Replacing clustered time-series slices with randomly sampled data causes degradation across all three tasks and increases overfitting.The ablation indicates that randomly sampled data is less challenging to model.
- 4.5 Ablation Study: Removing the time series question answering dataset degrades performance across all three tasks, especially multimodal forecasting and question answering.This ablation excludes the text question answering dataset during instruction fine-tuning.
5 Conclusion
ChatTime is presented as a unified framework that processes time series and text by representing time series as a foreign language. The paper reports multimodal capabilities and identifies resource limits as a boundary for future extension.
- 5 Conclusion: ChatTime characterizes time series as a foreign language to unify time series and text processing in a multimodal foundation model.The framework supports zero-shot inference and bimodal inputs and outputs.
- 5 Conclusion: Resource constraints prevent ChatTime from reaching saturation, motivating future work with more data and computation across additional tasks.The stated examples include anomaly detection, classification, and summarization.
- 5 Conclusion: The paper uses prompts for conventional forecasting, context-guided forecasting, and time series question answering to structure multimodal interactions.The prompts combine system instructions, task descriptions, inputs, and responses, with contextual information added where required.
B.1 Zero-Shot Time Series Forecasting
The zero-shot time series forecasting materials define prompt-based analyses across multiple time series features and benchmark datasets. The prompts translate sequences into the paper’s foreign-language representation and request predictions or feature answers.
- B.1 Zero-Shot Time Series Forecasting: The forecasting benchmark includes Electric, Exchange, Traffic, Weather, and four ETT datasets, with dataset statistics summarized in Table 7.The datasets cover electricity consumption, exchange rates, and road occupancy among other real-world series.
- B.1 Zero-Shot Time Series Forecasting: Forecasting prompts instruct ChatTime to predict a sequence from a supplied sequence, while context-guided prompts can include additional background information.The prompt examples distinguish sequence prediction from feature-oriented question answering.
- B.1 Zero-Shot Time Series Forecasting: The time series question-answering prompts define trend, volatility, seasonality, and outlier categories for sequence analysis.Separate prompts address trend, volatility, season, and outlier features.
- B.1 Zero-Shot Time Series Forecasting: Tables 7 and 8 provide dataset statistics for the traditional forecasting and context-guided forecasting settings.The supplied materials identify the tables as summaries of the respective dataset collections.
B.2 Context-Guided Time Series Forecasting
The context-guided forecasting setting addresses limited multimodal resources by collecting three datasets, while the question-answering dataset organizes generic time series features into labeled categories.
- B.2 Context-Guided Time Series Forecasting: Three collected multimodal datasets support context-guided forecasting, addressing limited relevant resources; their statistics are summarized in Table 8.MSPG contains solar generation data from 27 photovoltaic sites, and LEU contains electricity usage data from 16 houses.
- B.2 Context-Guided Time Series Forecasting: The time series question-answering dataset organizes questions around trend, volatility, season, and outlier features.These generic features are intended to help ChatTime comprehend fundamental time series principles.
C.1 Zero-Shot Time Series Forecasting
ChatTime achieves zero-shot forecasting accuracy comparable to Chronos while using only 4% of its pre-training data, and often performs better in non-stationary settings.
- 4% of the pre-training data enables ChatTime to achieve accuracy similar to Chronos and surpass it in some zero-shot scenarios.
- ChatTime often forecasts more accurate trends than single-dataset models in non-stationary scenarios such as ETTh2 and Electric.
- The showcases cover eight real-world benchmark datasets, with history windows set to twice the prediction-window length.
C.2 Context-Guided Time Series Forecasting
Context-guided information significantly improves ChatTime's forecasting accuracy across three real-world scenarios by incorporating external events that affect demand, generation, and traffic.
- Adding context-guided information significantly improves ChatTime's prediction accuracy on all three benchmark datasets.
- Rain with cloud cover reduces solar generation, weekends lower electricity consumption, and weekday travel increases traffic flow.
- Without external context, ChatTime relies on historical series alone and cannot perceive the effects of events on forecasts.
C.3 Time Series Question Answering
ChatTime improves time series comprehension and supports question answering and summarization through seamless bimodal processing of time series and text.
- ChatTime provides an excellent understanding of time series features while supporting seamless bimodal input and output of time series and text.
- Its capabilities enable both time series question answering and summarization.
- The evaluation uses question answering examples covering four typical time series features.