PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning

Zhe Xie, Zeyan Li, Xiao He, Longlong Xu, Xidao Wen, Tieying Zhang, Jianjun Chen, Rui Shi, Dan Pei

arXiv:2412.03104v3cs.AI

TL;DR

Time-series understanding and reasoning with MLLMs is limited by scarce high-quality time-series/text alignment data. ChatTS uses attribute-based synthetic data and Time Series Evol-Instruct to support multivariate time-series understanding and reasoning. It outperforms baselines across nearly all alignment tasks and all reasoning tasks in the reported evaluations.

  • Problem

    Research on MLLMs for time-series understanding and reasoning remains limited, primarily because high-quality datasets aligning time series with text are scarce.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    ChatTS is fine-tuned on synthetic time-series/text data with detailed attribute descriptions, while Time Series Evol-Instruct generates diverse question-answer data from attribute combinations.

    Source: §3.1 · p. 3, §8 · p. 12

  • Results

    ChatTS outperforms baseline models across nearly all alignment tasks and all reasoning tasks in the reported evaluations.

    Source: §4.2 · p. 7, §4.3 · p. 8

  • Takeaways & Limitations

    The results show that synthetic training data can enable strong alignment with real-world time series and support understanding of their physical patterns.

    Source: §4.2 · p. 7, §4.3 · p. 8

    The authors consider real-world data essential for further enhancing TS-MLLM capabilities, despite satisfactory results from synthetic data.

    Source: §7 · p. 12

Abstract

from arXiv · show

Understanding time series is crucial for its application in real-world scenarios. Recently, large language models (LLMs) have been increasingly applied to time series tasks, leveraging their strong language capabilities to enhance various applications. However, research on multimodal LLMs (MLLMs) for time series understanding and reasoning remains limited, primarily due to the scarcity of high-quality datasets that align time series with textual information. This paper introduces ChatTS, a novel MLLM designed for time series analysis. ChatTS treats time series as a modality, similar to how vision MLLMs process images, enabling it to perform both understanding and reasoning with time series. To address the scarcity of training data, we propose an attribute-based method for generating synthetic time series with detailed attribute descriptions. We further introduce Time Series Evol-Instruct, a novel approach that generates diverse time series Q&As, enhancing the model's reasoning capabilities. To the best of our knowledge, ChatTS is the first TS-MLLM that takes multivariate time series as input for understanding and reasoning, which is fine-tuned exclusively on synthetic datasets. We evaluate its performance using benchmark datasets with real-world data, including six alignment tasks and four reasoning tasks. Our results show that ChatTS significantly outperforms existing vision-based MLLMs (e.g., GPT-4o) and text/agent-based LLMs, achieving a 46.0% improvement in alignment tasks and a 25.8% improvement in reasoning tasks. We have open-sourced the source code, model checkpoint and datasets at https://github.com/NetManAIOps/ChatTS.

1 INTRODUCTION

The introduction argues that time-series reasoning matters in real-world applications, while current LLM approaches and scarce language-aligned data limit time-series understanding. ChatTS combines synthetic aligned data with a model for multivariate time-series reasoning and is reported to outperform baseline models.

1 INTRODUCTION

  • In AIOps, natural-language dialogue about multivariate monitoring time series can improve diagnosis and troubleshooting.The introduction presents time-series reasoning as relevant to human decision-making across domains including electricity, healthcare, traffic, weather, and finance.Source: §1 · p. 1
  • Text-based methods face prompt-length and global-feature limits, vision-based methods struggle with plotted detail, and agent-based methods depend on tool functionality.These approaches are described as distinct ways of giving LLMs time-series information.Source: §1 · p. 1
  • TS-MLLM development is challenged by scarce time-series/text pairs, the need for diverse accurate attribute descriptions, multivariate variable-length data, and limited evaluation resources.The introduction identifies relationships among multiple time series as important for understanding and reasoning.Source: §1 · p. 2
  • ChatTS is trained on synthetic time-series/text data generated with detailed attributes, while Time Series Evol-Instruct creates diverse question-answer data through attribute and task combinations.The proposed context-aware encoder is designed for variable-length, multivariate inputs while retaining numerical information.Source: §1 · p. 2
  • ChatTS significantly outperforms baseline models, including GPT-4o, across multiple datasets in both alignment and reasoning tasks.The evaluation includes six alignment tasks and four reasoning tasks.Source: §1 · p. 2

2 PRELIMINARY AND MOTIVATION

The section defines time-series question answering as generating a text response from a query and one or more time series. It contrasts text, vision, and tool-based approaches with TS-MLLMs, which natively combine time-series and textual inputs.

2.1 Problem Definition

  • A TS-MLLM takes a set of time series and a natural-language query, then generates a text answer describing relevant patterns or relationships.Each series may have its own number of observed time points.Source: §2.1 · p. 3

2.2 Existing Methods

  • Text-based methods are limited by prompt length, vision-based methods may miss detailed information and incur computational overhead, and agent-based methods depend on tools and can be token-intensive.The section also notes that longer agent reasoning chains can increase hallucination concerns in complex analysis.Source: §2.2 · p. 3

2.3 Time Series Multimodal LLM

  • TS-MLLMs natively process multiple time series and textual descriptions, supporting unified analysis of multivariate relationships without visual representations or lengthy token chains.The section associates time-series/text alignment with global and local analysis of shape and numerical information.Source: §2.3 · p. 3

3 METHODOLOGY

ChatTS builds a time-series/text training pipeline from synthetic data, combining attribute-based series generation, Time Series Evol-Instruct, a context-aware multimodal model, and two-stage training. Its design targets time-series alignment and reasoning, including multivariate inputs and numerical queries.

3.1 Overview

  • ChatTS’s pipeline generates synthetic time-series/text data, then uses alignment training and supervised fine-tuning to align modalities and strengthen reasoning.The overview combines attribute-based generation, TSEvol, and model training.Source: §3.1 · pp. 3–4

3.2 Attribute-Based Time Series Generator

  • The generator selects metric-relevant attributes, samples values and positions, and constructs matching series from an attribute pool.Its attribute set spans trends, seasonality, noise, and local fluctuations, with 4, 7, 3, and 19 attribute types respectively.Source: §3.2 · p. 4

3.3 Time Series Evol-Instruct

  • TSEvol dynamically adds sampled time-series attributes to evolving prompts and filters Q&As for attribute consistency.It uses multivariate attribute and correlation pools and adds reasoning- and situation-based evolution types.Source: §3.3 · pp. 4–5

3.4 Time Series Multimodal LLM

  • ChatTS encodes fixed-size time-series patches with a five-layer MLP and maps them alongside tokenized text into a shared embedding space.The model accepts multivariate time series, text, and contextual information as inputs.Source: §3.4.1 · p. 5, §3.4 · p. 5

3.4.1 Context-Aware Time-Series Multimodal LLM.

  • Context-aware encoding inserts each series’ patches among surrounding text tokens, preserving multivariate references for the LLM’s answer generation.Min-max normalization is paired with Value Scaling and Value Offset in the prompt to preserve original numerical information.Source: §3.4.1 · p. 5, §3.4.2 · p. 5

3.5 Model Training

  • Training uses two stages: large-scale alignment training followed by supervised fine-tuning.ChatTS is based on QWen2.5-14B-Instruct.Source: §3.5 · p. 5

3.5.1 Large-Scale Alignment Training.

  • Alignment training uses synthetic univariate and multivariate datasets covering global/local attributes, trends, and correlations.MTS and UTS training data are used at an approximately 2:1 ratio.Source: §3.5.1 · p. 6

3.5.2 Supervised Fine-Tuning.

  • Supervised fine-tuning uses TSEvol Q&As and instruction-following data to train question answering, reasoning, and response-format adherence.It mixes 30% of the alignment dataset into SFT to reduce overfitting and includes numerical tasks such as extrema, averages, and point values.Source: §3.5.2 · p. 6, §3.5.3 · p. 6

4 EVALUATION

The evaluation tests time-series alignment and reasoning against text-, vision-, and agent-based baselines. It uses manually collected real-world data, generated data, and an open-source comparison dataset, with selected tasks and metrics suited to each task type.

4 EVALUATION

  • The evaluation frames six questions around alignment, reasoning, data and TSEvol effectiveness, training-set size, modality usefulness, and agent comparisons.The evaluation covers both alignment and reasoning performance.Source: §4 · p. 6, §4.1.1 · p. 6

4.1.1 Evaluation Tasks.

  • Alignment tasks assess univariate trends, seasonality, noise, and local fluctuations, plus multivariate correlation and clustering.Univariate tasks include categorical and numerical subtasks; multivariate tasks are categorical.Source: §4.1.1 · p. 6
  • Reasoning tasks cover inductive, deductive, causal, and two-series comparison questions; tasks requiring substantial domain knowledge are excluded.Dataset B includes only inductive reasoning questions to maintain question quality.Source: §4.1.1 · pp. 6–7, §4.1.3 · p. 7

4.1.2 Evaluation Metrics.

  • Evaluation uses F1-Score for categorical alignment, relative accuracy for numerical alignment, RAGAS for inductive Q&A, and accuracy for choice tasks.Relative accuracy is floored at 0.0 to mitigate outlier effects.Source: §4.1.2 · p. 7

4.1.3 Evaluation Datasets.

  • Dataset A contains 525 manually labeled questions from real-world domains, Dataset B has 1,616 generated questions, and 100 MCQ2 questions are sampled from an LLM-generated dataset.MCQ2 supplies comparison reasoning; its questions, answers, and series are generated by LLMs.Source: §4.1.3 · p. 7

4.1.4 Baselines.

  • Baselines include text-prompt LLMs, vision MLLMs given plots, and GPT-4o or GPT-4o-mini agents using time-series analysis tools.GPT-based models use OpenAI’s API, while ChatTS and QWen-based models run locally on 8×A800 GPUs.Source: §4.1.4 · p. 7, §4.1.5 · p. 7

4.2 RQ1. Alignment Tasks

On alignment tasks, ChatTS outperforms the baselines across nearly all tasks and datasets, with especially pronounced advantages on multivariate inputs. The paper also reports lower input-token costs than the baselines.

4.2 RQ1. Alignment Tasks

  • 46.0%–75.9% categorical-metric and 80.7%–112.7% numerical-metric improvements over industry-leading models such as GPT-4o are reported across nearly all tasks and datasets.Table 3 compares model performance and input-token cost; categorical tasks use F1-Score and numerical tasks use relative accuracy.Source: §4.2 · p. 7, Table 3 · p. 8
  • GPT-4o Vision is the strongest baseline, while text-based methods face prompt-length constraints and agents perform below expectations.The paper attributes vision models’ remaining detail limitations to plot resolution.Source: §4.2 · p. 7
  • ChatTS’s advantage is especially pronounced on multivariate tasks, where its context-aware encoding analyzes referenced series using textual context.Text baselines face long prompts, while vision baselines struggle to distinguish multiple plotted series.Source: §4.2 · p. 7
  • ChatTS’s native multimodal encoding represents time series with significantly fewer input tokens and lower costs than the baselines.The cost comparison is reported in Table 3.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 RQ2. Reasoning Tasks

ChatTS improves reasoning-task performance, while ablations indicate that attribute-based synthetic data and TSEvol contribute to its performance across alignment and reasoning tasks.

4.3 RQ2. Reasoning Tasks

  • ChatTS achieves a 34.5% improvement over baseline models on Inductive Reasoning and consistent improvements across all reasoning tasks.The authors associate the Inductive Reasoning result with linking time-series attributes to their physical meanings.Source: §4.3 · p. 8

4.4 RQ3. Studies of Synthetic Training Data

  • Models trained on GPT-generated data perform worse across alignment tasks, particularly in local fluctuation detection and numerical analysis.The ablation replaces the attribute-based training datasets with GPT-generated time series and Q&As.Source: §4.4 · p. 8
  • TSEvol improves reasoning substantially and alignment modestly, with results attributed to more diverse question formats and tailored Q&As.The comparison uses data generated without the evolutionary approach as the TSEvol ablation.Source: §4.4 · p. 8

4.5 RQ4. Scaling of Training Dataset

Increasing Phase 1 training data from 10% to 100% improves performance, while further expansion yields minimal gains; modality ablations favor native time-series processing, especially for multivariate tasks.

4.5 RQ4. Scaling of Training Dataset

  • Increasing Phase 1 training data from 10% to 100% significantly improves performance, but further expansion yields minimal gains.The authors describe the selected training-set size as balancing alignment quality against training resource consumption.Source: §4.5 · pp. 8–9, Figure 11 · p. 9

4.6 RQ5. Study of Time Series Modality

  • The text-only model performs significantly worse overall than ChatTS, though it outperforms ChatTS on some noise metrics.The text-only version removes the time-series encoder and encodes the time-series arrays as text; the comparison covers alignment and reasoning tasks.Source: §4.6 · p. 9, Figure 9 · p. 9, Figure 10 · p. 9
  • On multivariate time-series tasks, the text-only model is nearly incapable of answering questions, with long contexts, hallucinations, and inaccurate responses identified as difficulties.The authors attribute the difficulty to excessively long context lengths.Source: §4.6 · p. 9
  • Fine-tuned Qwen2.5-32B does not outperform ChatTS-14B, which has native multimodal capabilities.The comparison uses the text version of ChatTS's training dataset; the authors also point to accuracy and cost efficiency.Source: §4.6 · p. 9, Figure 11 · p. 9

4.7 RQ6. Study of Agent-Based Methods

Agent-based methods face tool-accuracy and tool-selection challenges; even with perfect tools, tool-use errors and other reasoning failures remain, particularly for multivariate tasks.

4.7 RQ6. Study of Agent-Based Methods

  • Agent performance is highly sensitive to tool accuracy, especially between 0.9 and 1.0; perfect-tool agents slightly outperform ChatTS on univariate tasks but lag on multivariate tasks.Multivariate tasks typically require more tool calls and reasoning, while ChatTS processes multiple series natively.Source: Figure 12 · p. 9, §4.7 · p. 10
  • Agents depend on both tool precision and completeness, yet may invoke the wrong tool even when all tools are available.The authors give selecting a classification tool instead of an anomaly-detection tool to locate a spike as an example.Source: §4.7 · p. 10
  • Error Tool Using is the largest source of agent errors; excluding those cases raises agent accuracy above 95%, while tool-selection and reasoning remain limitations.The error analysis also categorizes misunderstanding and hallucination.Source: §4.7 · p. 10

5 CASE STUDIES AND APPLICATIONS

Real-world cases show ChatTS identifying time-series patterns, statistics, and anomalies, then using textual context for analysis; comparisons highlight challenges faced by agent-, vision-, and text-based models.

5.1.1 Shape and Statistical Analysis.

  • ChatTS identifies trend segments, periodicity, a local spike and its amplitude in NYC taxi data, and max/min values and positions in advertisement CPC data with misleading scaling.The cases illustrate both global-pattern and localized-feature analysis.Source: §5.1.1 · p. 10, §5.1 · p. 10

5.1.2 OOD Fluctuation Recognition.

  • For a traffic-occupancy fluctuation absent from training, ChatTS describes a gradual rise and sharper decline as a convex-shaped elevation.The authors present this as evidence of some capacity to generalize to real-world data despite synthetic-only training.Source: §5.1.2 · p. 10, Figure 13 · p. 10

5.2 Real-World Application: DB Operation

  • In an Oracle database case, ChatTS identifies anomalous metrics and uses an unstructured troubleshooting rulebook to reason step-by-step about root cause and failure propagation.The operator supplies the rulebook as text alongside multivariate time-series metrics.Source: §5.2 · pp. 10–11, Figure 14 · p. 11

5.3 Real-World Application: Detailed Analysis

  • For AAPL-related Twitter discussion intensity, ChatTS infers three hot-event timestamps from vague input, identifies the highest peak, and analyzes events using the series' physical meaning.The time-series data comes from NAB.Source: §5.3 · p. 11, Figure 15 · p. 11

5.4 Baseline Comparison

  • In a trend case, tool inaccuracy leads an agent to miss periodicity and misread trends, while the vision-based model also errs on local fluctuations and periodicity.ChatTS accurately captures periodicity and trend transitions in this example.Source: §5.4.1 · p. 11, Figure 16 · p. 11

5.4.2 Detailed Trend Analysis.

  • When two spikes obscure a decreasing trend in the original plot, the vision-based model calls it steady and text-based models miss the global shape; ChatTS captures the trend and numerical details.The comparison concerns the cropped plot with the two spikes ignored.Source: Figure 16 · p. 11, §5.4.2 · p. 12

6 RELATED WORK

Time-series modality alignment in MLLMs remains limited, with high-quality time-series–text datasets scarce. Existing time-series question-answering approaches often lack comprehensive understanding and reasoning or rely on task-specific tools and corpora.

6 RELATED WORK

  • Research aligning LLMs with time-series modalities remains limited, primarily because high-quality multimodal datasets combining time series and text are scarce [15] [27] [43].The paper notes that time-series-specific MLLMs for question answering and reasoning have lagged behind other modalities.Source: §6 · p. 12
  • TSQA methods often rely on agents or RAG and lack comprehensive understanding and sufficient reasoning; task-specific temporal multimodal systems also lack multivariate analysis [57] [30] [15].The paper proposes comprehensive time-series alignment and fine-tuning evaluated across multiple alignment and reasoning tasks.Source: §6 · p. 12
  • LLM–time-series studies target specific downstream tasks and lack language alignment [12] [13] [23] [26] [38] [51] [67], while vision-based approaches are significantly constrained in time-series analysis [14] [69].The cited vision-based methods use multimodal LLMs for time-series prediction and anomaly detection.Source: §6 · p. 12

7 LIMITATION AND FUTURE WORK

The authors identify limitations in the data, encoder exploration, evaluation coverage, and task scope, and outline corresponding directions for future work.

7 LIMITATION AND FUTURE WORK

  • Real-world data is considered essential for further enhancing TS-MLLM capabilities, although synthetic data achieved satisfactory alignment and reasoning performance.The authors hope more relevant datasets will emerge.Source: §7 · p. 12
  • Exploring more effective multimodal encoding and integration remains open, despite a simple MLP encoder performing well on relatively simple time-series structures.Source: §7 · p. 12
  • Hundreds of labeled real-world time series and 14 evaluation metrics remain insufficient for comprehensive TS-MLLM evaluation, motivating more labeled real-world data.Source: §7 · p. 12
  • The work focuses on understanding tasks such as language alignment and reasoning; generating time series from textual input remains a future research direction.Source: §7 · p. 12

8 CONCLUSION

ChatTS addresses limited time-series understanding and reasoning research with synthetic-data training and evaluation on real-world datasets. It reports gains over baselines in alignment and reasoning tasks.

8 CONCLUSION

  • 46.0% improvement in alignment tasks and 25.8% improvement in reasoning tasks over baselines are reported for ChatTS.Source: §8 · p. 12
  • ChatTS is presented as the first TS-MLLM taking multivariate time series as input for complex time-series QA and reasoning, fine-tuned on synthetic data.Source: §8 · p. 12
  • Attribute-based generation produces diverse time series with complete, precise attribute descriptions, while TSEvol combines attribute combinations and Evol-Instruct to generate diverse, accurate QAs.Source: §8 · p. 12
  • Collected datasets with real-world time series support evaluation of alignment and reasoning tasks, and the source code, trained weights, and evaluation datasets are open-sourced.Source: §8 · p. 12
Loading 2412.03104v3…