PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data

Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, Jianxin Liao

arXiv:2412.11376v1cs.CLcs.LG

TL;DR

Many deep-learning time-series predictors use single-dataset, fixed-window settings, while human experts integrate numerical and textual information. ChatTime models time series as a foreign language for unified time-series and text processing, and reports advantages across tasks, including 99.9% of Chronos’s zero-shot prediction accuracy using 4% of the data.

  • Problem

    Deep-learning predictors often train and forecast on a single dataset with fixed history and prediction windows, while human experts integrate multimodal information.

    Source: §1 · p. 1

  • Method

    ChatTime represents time series as a foreign language and uses vocabulary expansion to unify time-series and text processing with a pre-trained large language model.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    ChatTime shows considerable advantages across multiple time-series tasks; for zero-shot forecasting, it achieves 99.9% of Chronos’s zero-shot prediction accuracy using only 4% of the data.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The four constructed multimodal datasets offer resources for future research in related multimodal domains.

    Source: §1 · p. 2

    The authors report that resource constraints have kept ChatTime from reaching saturation.

    Source: §5 · p. 7

Abstract

from arXiv · show

Human experts typically integrate numerical and textual multimodal information to analyze time series. However, most traditional deep learning predictors rely solely on unimodal numerical data, using a fixed-length window for training and prediction on a single dataset, and cannot adapt to different scenarios. The powered pre-trained large language model has introduced new opportunities for time series analysis. Yet, existing methods are either inefficient in training, incapable of handling textual information, or lack zero-shot forecasting capability. In this paper, we innovatively model time series as a foreign language and construct ChatTime, a unified framework for time series and text processing. As an out-of-the-box multimodal time series foundation model, ChatTime provides zero-shot forecasting capability and supports bimodal input/output for both time series and text. We design a series of experiments to verify the superior performance of ChatTime across multiple tasks and scenarios, and create four multimodal datasets to address data gaps. The experimental results demonstrate the potential and utility of ChatTime.

1 Introduction

ChatTime treats time series as a foreign language to unify time-series and text processing in a multimodal foundation model. It is designed to support zero-shot inference and bimodal inputs and outputs, addressing limitations of existing approaches.

1 Introduction

  • Most existing predictors use only numerical data and fixed history and prediction windows on a single dataset, while simple linear models often rival complex state-of-the-art models.The authors frame these limitations as reducing adaptability across scenarios and suggest unimodal approaches may be nearing saturation.Source: §1 · p. 1
  • ChatTime supports zero-shot forecasting and time-series and text inputs and outputs, addressing prior methods that require dataset-specific fine-tuning or cannot produce text.Earlier approaches either trained from scratch without text processing or added layers that required re-fine-tuning for each dataset and could not output text.Source: §1 · pp. 1–2
  • ChatTime discretizes normalized time series into marked symbolic values and adapts a pre-trained LLM through continuous pre-training and instruction fine-tuning, without training from scratch or changing its architecture.The authors report lower training cost than other foundation models and the ability to process textual information.Source: §1 · p. 2
  • Experiments cover zero-shot forecasting, context-guided forecasting, and time-series question answering, with results reported as superior across multiple tasks and scenarios.The evaluation uses eight forecasting benchmarks across four domains, context-guided data from three scenarios, and a variable-length question-answering dataset; the paper also establishes three context-guided datasets and one question-answering dataset.Source: §1 · p. 2

2 Related Work

Prior forecasting methods progressed from statistical models to deep learning, but remain limited by adaptability and reliance on numerical data alone. LLM-based time-series work spans three approaches with distinct tradeoffs in accuracy, inference cost, zero-shot ability, and modality support.

2.1 Long-Term Time Series Forecasting

  • Forecasting progressed from ARIMA to recurrent, convolutional, and attention-based models, but real-world adaptation remains difficult and most deep methods use only numerical data.RNNs capture temporal dependencies but can suffer vanishing or exploding gradients and forget information; convolutional and self-attention methods target long-range dependencies.Source: §2.1 · p. 2

2.2 LLM-Based Time Series Analysis

3 Methodology

ChatTime converts numerical time series into a tokenizable foreign language that an LLM can process, then trains the model in continuous pre-training and instruction fine-tuning phases. Its methodology combines this representation with datasets for forecasting and multimodal text–series tasks.

3.1 Overview

  • The two training phases use 4-bit quantized models with LoRA; continuous pre-training trains LLaMA-2-7B-Base on 1M time-series slices with autoregressive forecasting.The expanded tokenizer’s embedding layer and output head are trained alongside the Transformer during continuous pre-training.Source: §3.1 · p. 3, §3.3 · p. 4

3.2 Model Architecture

  • ChatTime scales series using history only into -0.5 to 0.5, quantizes values into 10K bins, wraps them as ###-marked vocabulary tokens, and decodes LLM outputs back into time series.An additional ###Nan### token represents missing values; one token represents each value regardless of precision.Source: §3.2 · p. 3, §3.1 · p. 3

3.3 Continuous Pre-Training

  • Continuous-pretraining data come from Monash and TFB, exclude 11 evaluation sub-datasets, and are sampled from 10M clustered slices into 1M and 25K sets; autoregressive forecasting supports windows of any size.Sliding slices use five window and step-size settings, prioritizing larger segments.Source: §3.3 · p. 4

3.4 Instruction Fine-Tuning

  • Instruction fine-tuning uses 25K examples each for text QA, unimodal forecasting, context-guided forecasting, and time-series QA (100K total), updating only the Transformer layer.Text QA samples come from Alpaca; forecasting uses high-quality slices, while the multimodal tasks use collected CGTSF datasets and synthesized TSQA data.Source: §3.4 · p. 4
  • CGTSF datasets cover Melbourne solar, London electricity, and Paris traffic with background, weather, and date text; TSQA uses KernelSynth to generate variable-length questions from four typical time-series features.The three CGTSF datasets are chronologically split 6:2:2; aligning time-series features with text can also improve context-guided forecasting.Source: §3.4 · p. 4

4 Experiment

ChatTime is evaluated on zero-shot and context-guided forecasting and time-series question answering, with results favoring its multimodal approach; ablations indicate that its pre-training, clustered slices, and text-QA fine-tuning each contribute.

4.1 Implementation Setting

  • Both training stages use 4-bit LoRA; pre-training runs 8K steps and fine-tuning 1.6K steps, with the full process executable on a single RTX 4090.The global batch size is 256.Source: §4.1 · pp. 4–5

4.2 Zero-Shot Time Series Forecasting

  • 99.9% of Chronos’s zero-shot prediction accuracy with only 4% of the data, and 90.9% of GPT4TS’s prediction accuracy, are reported for ChatTime.GPT4TS and TimeLLM, despite using pretrained LLMs, do not significantly outperform DLinear.Source: Table 4 · p. 5, §4.2 · p. 6
  • Zero-shot tests use eight held-out datasets across four domains plus four ETT datasets, chronological 6:2:2 splits, period-based horizons, 2–5-period histories, and MAE.The datasets were excluded from ChatTime training to prevent information leakage.Source: §4.2 · p. 5

4.3 Context-Guided Time Series Forecasting

  • With textual context, ChatTime and TGForecaster outperform other baselines; ChatTime surpasses dataset-specific TGForecaster and ChatTime− without text.ChatTime is instruction-fine-tuned on partial training sets of three multimodal datasets but uses shared weights rather than separate scenario-specific training.Source: Table 5 · p. 6, §4.3 · p. 6

4.4 Time Series Question Answering

  • ChatTime preserves LLM inference capabilities while showing better understanding of time-series features than generic pretrained LLMs on question answering.The evaluation compares average accuracy and rank across four features.Source: Table 6 · p. 7, §4.4 · p. 7

4.5 Ablation Study

  • Ablations show that autoregressive pre-training, clustered slices, and text-QA fine-tuning each matter, with removing pre-training especially reducing zero-shot capability.Replacing clustered slices with random samples degrades all three tasks and encourages overfitting; omitting text-QA fine-tuning degrades all three, particularly context-guided forecasting and question answering.Source: §4.5 · p. 7, Figure 3 · p. 7

5 Conclusion · A Prompt

ChatTime unifies time-series and text processing by treating time series as a foreign language, with zero-shot inference and bimodal inputs and outputs. The conclusion notes its current resource-limited scope and outlines future expansion, while the prompt appendix describes three task designs.

5 Conclusion

  • ChatTime frames time series as a foreign language to unify time-series and text processing, supporting zero-shot inference and bimodal inputs and outputs.The study also reports experiments and four multimodal datasets created to address data gaps.Source: §5 · p. 7
  • Because of resource constraints, ChatTime has not reached saturation; future work will use more data and computation to extend it to anomaly detection, classification, and summarization.Source: §5 · p. 7

A.1 Zero-Shot Time Series Forecasting

  • Zero-shot forecasting prompts combine a system prompt, task introduction, history input, and prediction response, with the series represented as foreign-language words.The system prompt and introduction remain consistent; history and prediction series vary.Source: §A.1 · p. 10
  • During zero-shot forecasting inference, the prediction series is concealed so ChatTime generates it; it is supplied only during instruction fine-tuning.Source: §A.1 · p. 10

A.2 Context-Guided Time Series Forecasting

  • Context-guided forecasting adds contextual knowledge and supplementary information to the introduction, while history and prediction series remain the input and response.The system prompt and task description remain consistent, while context and the series vary; the prediction is concealed at inference and supplied during fine-tuning.Source: §A.2 · p. 10

A.3 Time Series Question Answering

  • Time-series question-answering prompts provide background knowledge, a foreign-language time series as input, and the correct answer as the response.The answer is supplied only during instruction fine-tuning and concealed at inference; the background knowledge, analyzed series, and answer are variable components.Source: §A.3 · p. 10

B Dataset · B.1 Zero-Shot Time Series Forecasting

B.1 evaluates zero-shot forecasting on eight publicly available, real-world datasets spanning five dataset groups, with dataset statistics summarized in Table 7.

B.1 Zero-Shot Time Series Forecasting

  • Experiments cover eight real-world datasets across Electric, Exchange, Traffic, Weather, and four ETT datasets.The datasets are described as widely used for benchmarking and publicly available.Source: §B.1 · pp. 10, 12
  • Table 7 summarizes the statistics of each dataset used in the traditional unimodal forecasting task.Source: Table 7 · p. 12, §B.1 · p. 12
  • Electric records hourly electricity consumption for 321 customers from 2012 to 2014, while Exchange contains daily exchange rates for eight countries from 1990 to 2019.Source: §B.1 · p. 12
  • Traffic aggregates hourly road occupancy from 862 San Francisco Bay Area freeway sensors during 2015–2016, while Weather records 21 German weather parameters every 10 minutes in 2020.Source: §B.1 · p. 12
  • The four ETT datasets, ETTm1, ETTm2, ETTh1, and ETTh2, cover oil temperature and load characteristics of two power transformers from 2016 to 2018.Source: §B.1 · p. 12
  • The datasets are widely used for benchmarking and publicly available.Source: §B.1 · p. 12

B.2 Context-Guided Time Series Forecasting · B.3 Time Series Question Answering · C Showcase

ChatTime’s context-guided forecasting uses text aligned with time series, while its question-answering task pairs time series features with textual representations. The section describes three forecasting datasets and a synthetic dataset for question answering.

B.2 Context-Guided Time Series Forecasting

  • Context-guided forecasting transforms text into time series data, addressing limited multimodal dataset availability with three collected datasets.Raw time series records are gathered from Kaggle.Source: §B.2 · p. 12
  • MSPG contains 13 months of solar power generation data from 27 photovoltaic sites in Melbourne during 2021–2022.Source: §B.2 · p. 12
  • LEU covers 24 months of electricity use across 16 London households during 2012–2013, while PTF covers 12 months of traffic flow at 32 Paris detectors in 2012.Source: §B.2 · p. 12
  • To prevent future data leakage, auxiliary text uses background, weather, and date information strictly aligned with time series records by day.Weather forecasts from Open-Meteo include weather codes, temperatures, and sunrise and sunset times; date information includes the raw date, weekday, and holidays.Source: §B.2 · p. 12

B.3 Time Series Question Answering

  • Question-answer pairs target four generic time-series features: trend, volatility, seasonality, and outliers, each represented through three categories.The categories distinguish upward, downward, or constant trends; increased, decreased, or constant volatility; fixed, shifting, or absent seasonality; and sudden spikes, level shifts, or stable absence of outliers.Source: §B.3 · p. 12
  • KernelSynth generates time-series slices of lengths 64, 128, 256, and 512 to enhance robustness; aligning features with text can also improve downstream task performance.Source: §B.3 · p. 12

C.1 Zero-Shot Time Series Forecasting

ChatTime uses 4% of the pre-training data to achieve accuracy similar to Chronos and surpass it in some scenarios. It often forecasts more accurate trends than single-dataset models in non-stationary settings.

C.1 Zero-Shot Time Series Forecasting

  • Using only 4% of pre-training data, ChatTime achieves accuracy similar to Chronos and surpasses it in some scenarios across eight real-world benchmark datasets.Figure 10 compares forecasting showcases; each dataset uses a history window twice the prediction-window length.Source: §C.1 · p. 12
  • In non-stationary scenarios such as ETTh2 and Electric, ChatTime often forecasts trends more accurately than models trained on a single dataset.The authors present this as evidence of the potential and utility of a generic foundation model.Source: §C.1 · p. 12
  • For full-shot forecasting, DLinear performs comparably to GPT4TS, suggesting current unimodal approaches may be nearing saturation.Source: §C.1 · p. 12

C.2 Context-Guided Time Series Forecasting

ChatTime’s context-guided forecasts improve on predictions made from time-series history alone across three real-world datasets. The paper presents this as evidence that external context is necessary for forecasting.

C.2 Context-Guided Time Series Forecasting

  • Because simple linear models can rival state-of-the-art complex models, the paper argues that unimodal forecasting may be nearing its performance limits and needs auxiliary information.Source: §C.2 · p. 12
  • With context, ChatTime significantly improves prediction accuracy on MSPG, LEU, and PTF; without it, the model cannot perceive external-event effects.Rain with cloud cover reduces solar generation in MSPG, weekend breaks lower electricity consumption in LEU, and weekday travel raises traffic flow in PTF.Source: Figure 11 · p. 14, §C.2 · p. 15
  • ChatTime expands the vocabulary of pre-trained LLaMA to accept time-series and textual inputs together while retaining LLaMA’s inference capability.Source: §C.2 · p. 15

C.3 Time Series Question Answering

ChatTime is presented as a time-series comprehension model that supports question answering and summarization across time series and text. Examples cover four typical time-series features, while the paper notes that generic pretrained LLMs are less satisfactory at time-series comprehension.

C.3 Time Series Question Answering

  • ChatTime’s question-answering examples cover four typical time-series features to validate its time-series comprehension.The examples are presented in Figure 12.Source: §C.3 · p. 15
  • Generic pretrained LLMs perform less satisfactorily on time-series comprehension than on textual tasks, according to the paper.Source: §C.3 · p. 15
  • Fine-tuned from LLaMA with an expanded vocabulary, ChatTime supports bimodal time-series and text input and output for question answering and summarization.Source: §C.3 · p. 15

Related papers

Loading 2412.11376v1…