PaperlayerView exact revision on arXiv

Source-linked AI summary

TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop

Yushan Jiang, Wenchao Yu, Geon Lee, Dongjin Song, Kijung Shin, Wei Cheng, Yanchi Liu, Haifeng Chen

arXiv:2503.01013v4cs.LG

TL;DR

Multimodal time-series methods can use contextual signals, but often lack explicit explanations of how those signals affect predictions. TimeXL combines a prototype-based encoder with prediction, reflection, and refinement LLMs; across four real-world benchmarks, it outperforms baselines by up to an 8.9% improvement in AUC while producing human-centric multimodal explanations.

  • Problem

    Multimodal time-series methods often use auxiliary context to improve prediction but lack explicit mechanisms to explain how contextual signals affect outcomes, a barrier to trust in high-stakes applications.

    Source: §1 · p. 1

  • Method

    TimeXL combines a prototype-based multimodal encoder that produces case-based rationales with LLMs that refine forecasts, critique them against ground truth, and iteratively refine text.

    Source: §1 · p. 2

  • Results

    Across four real-world benchmarks, TimeXL consistently outperforms baselines, achieving up to an 8.9% improvement in AUC while providing faithful, human-centric multimodal explanations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TimeXL presents prototype-based inference coupled with LLM reasoning as an approach to explainable multimodal time-series analysis.

    Source: §1 · p. 2

Abstract

from arXiv · show

Time series analysis provides essential insights for real-world system dynamics and informs downstream decision-making, yet most existing methods often overlook the rich contextual signals present in auxiliary modalities. To bridge this gap, we introduce TimeXL, a multi-modal prediction framework that integrates a prototype-based time series encoder with three collaborating Large Language Models (LLMs) to deliver more accurate predictions and interpretable explanations. First, a multi-modal prototype-based encoder processes both time series and textual inputs to generate preliminary forecasts alongside case-based rationales. These outputs then feed into a prediction LLM, which refines the forecasts by reasoning over the encoder's predictions and explanations. Next, a reflection LLM compares the predicted values against the ground truth, identifying textual inconsistencies or noise. Guided by this feedback, a refinement LLM iteratively enhances text quality and triggers encoder retraining. This closed-loop workflow-prediction, critique (reflect), and refinement-continuously boosts the framework's performance and interpretability. Empirical evaluations on four real-world datasets demonstrate that TimeXL achieves up to 8.9% improvement in AUC and produces human-centric, multi-modal explanations, highlighting the power of LLM-driven reasoning for time series prediction.

1 Introduction

TimeXL addresses the interpretability gap in multimodal time-series methods with a prototype-based encoder and collaborating LLMs that refine forecasts and explanations. Across four real-world benchmarks, it outperforms baselines by up to 8.9% in AUC while providing human-centric multimodal explanations.

1 Introduction

  • Multimodal time-series approaches often use contextual signals to improve prediction but lack explicit reasoning about how those signals affect outcomes, limiting trust in high-stakes settings.Textual narratives can provide context relevant to forecasting and explainability.Source: §1 · p. 1
  • TimeXL combines a prototype-based multimodal encoder with LLM reasoning in a closed loop of prediction, critique, and refinement.The encoder produces case-based explanations from temporal and textual modalities, which guide forecast refinement.Source: §1 · p. 2
  • A reflection LLM compares predictions with ground truth to identify textual noise, and a refinement LLM updates text and prototypes iteratively.The paper describes these iterations as improving predictive and explanatory capabilities.Source: §1 · p. 2
  • Across four real-world benchmarks, TimeXL consistently outperforms baselines, with up to an 8.9% improvement in AUC and faithful, human-centric multimodal explanations.The paper presents this as the overall result of its framework.Source: §1 · p. 2

2 Related Work

Related work spans multimodal time-series analysis, time-series explanation, and LLMs for time-series tasks. TimeXL distinguishes itself by pairing prototype-based explanations with iterative, grounded interaction with LLMs.

Multi-modal Time Series Analysis

  • Multimodal time-series methods use fusion, attention, gating, or contrastive learning across modalities, but tractable rationales for how context affects outcomes remain underexplored.The surveyed work also includes retrieval, imputation, and causal discovery tasks [14].Source: §2 · pp. 2–3

Time Series Explanation

  • Time-series explanations include saliency and surrogate methods, whereas TimeXL uses learned prototypes for case-based predictions and human-readable explanations across temporal and contextual modalities.The cited paradigms include Shapley values [48] and self-supervised consistency [49].Source: §2 · p. 3

LLMs for Time Series Analysis

  • LLM time-series work includes fine-tuning, prompting, and knowledge inference; TimeXL instead combines an explainable model with iterative, grounded, reflective interaction.Prior directions include time-series foundation models [55] [60] [61] [62] and language-guided inference [67] [69].Source: §2 · p. 3

3 Methodology

TimeXL frames multimodal time-series prediction as combining time series and contextual text, then coordinates a prototype encoder with LLM agents for prediction, reflection, and refinement. Its method builds case-based explanations, enriches LLM predictions with relevant prototypes, and iteratively refines text and retrains the encoder.

3.1 Problem Statement

  • TimeXL predicts discrete outcomes from historical time series and corresponding text, using an encoder and three LLM agents for prediction, reflection, and refinement.The paper focuses on classification and also evaluates regression in the appendix.Source: §3.1 · p. 3

3.2 Multi-modal Prototype-based Encoder

  • The multimodal encoder processes time series and text to produce predictions and explanations, using separate encoders to represent temporal and semantic patterns.Convolution-based encoders capture fine-grained subsequence patterns in both modalities.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.1 Multi-modal Sequence Modeling with Prototypes

  • Class-specific time-series and text prototypes are matched to relevant segments, and their similarity scores are fused into class probabilities.A non-negative fusion weight matrix combines the cross-modal relevance scores.Source: §3.2.1 · p. 4

3.2.2 Learning Prototypes toward Better Explanation

  • The encoder objective combines classification loss with clustering, evidence-segment proximity, and prototype-diversity regularizers, then projects prototypes onto training segments.Projection associates each prototype with a same-class training segment, supporting its interpretation through an exemplar.Source: §3.2.2 · p. 5

3.3.1 Model Synergy for Augmented Prediction

  • The prediction LLM receives text augmented with the most relevant prototype-segment pairs, and its output is linearly combined with the encoder prediction.The fusion weight is selected using validation data.Source: §3.3.1 · pp. 5–6

3.3.2 Iterative Context Refinement via Reflective Feedback

  • A reflection LLM compares the prediction with ground truth and input text, after which a refinement LLM updates the text and the encoder is retrained for the next iteration.The system retains reflections and models when validation evaluation improves, then uses the best-performing reflections to guide test-time refinement.Source: §3.3.2 · p. 6

4 Experiments

Experiments compare multimodal prediction across four datasets and examine learned prototypes, case-based reasoning, iterative refinement, and component contributions.

4.1 Experimental Setup

  • The evaluation covers four multimodal datasets from weather, finance, and healthcare, using F1 and AUROC across five-run averages and 6:2:2 data splits.The datasets include Weather, Finance, Healthcare Test-Positive, and Healthcare Mortality.Source: §4.1 · pp. 6–7

4.2 Performance Evaluation

  • Multimodal methods generally outperform time-series methods across all datasets in the predictive-performance evaluation.The comparison includes F1 and AUROC (AUC) scores.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 Explainable Multi-modal Prototypes

  • Weather prototypes capture class-associated temporal patterns and semantic descriptions, such as humidity, pressure, and wind cues for rain or non-rain conditions.The figure shows time-series prototypes by channel alongside text prototypes.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 Multi-modal Case-based Reasoning

  • In the Weather case study, refinement shifts text matches toward rain-indicative humidity and wind cues, while time-series prototypes provide complementary evidence.Explanations are generated at half-sentence granularity.Source: §4.4 · pp. 8–9, Figure 4 · p. 9

4.5 Iterative Analysis

  • Text quality mostly saturates after one or two iterations, while TimeXL performance improves quickly and then stabilizes with minor fluctuations.The analysis evaluates text quality using an LLM's zero-shot predictive accuracy.Source: §4.5 · p. 9, Figure 5 · p. 9

4.6 Ablation Studies

  • Across datasets, the multimodal encoder beats single-modality variants, text prototypes improve LLM prediction, and combining the encoder with the prediction LLM beats either component alone.The ablation texts are refined using reflections selected for best validation performance.Source: §4.6 · p. 9, Table 2 · p. 10

5 Conclusions

TimeXL combines a prototype-based encoder with prediction, reflection, and refinement LLM agents. Experiments on four multimodal time-series datasets show advantages over state-of-the-art baselines and strong explanation capabilities.

5 Conclusions

  • TimeXL combines a prototype-based encoder with three collaborative LLM agents to improve predictions and explanations, with experiments showing advantages over state-of-the-art baselines on four multimodal datasets.The agents perform prediction, reflection, and refinement.Source: §5 · p. 10

A.1 Dataset Statistics

The appendix describes the real-world dataset scope and summarizes implementation settings, including hyperparameters, LLM configurations, and computing environment.

A.1 Dataset Statistics

  • The four datasets cover weather, finance, and healthcare, pairing time series with weather summaries, financial news, or healthcare reports.Weather and healthcare data are used from TimeCAP, while the Finance dataset is extended from prior work.Source: §A.1 · p. 16

A.2 Hyperparameters

  • Hyperparameter searches vary prototype counts and lengths, regularization strengths, encoder learning rate, and the number of case-based explanations.Results are reported as averages over five experimental runs.Source: §A.2 · p. 16

A.3 Large Language Model

  • GPT-4o-2024-08-06 is the default GPT-4o version, and the same settings are used for Gemini-2.0-Flash and GPT-4o-mini.Generation uses temperature 0.7 and prediction uses 0.3.Source: §A.3 · p. 17

A.4 Environment

  • Experiments ran on a TensorEX server with two 20-core Intel Xeon Gold 5218R processors, 512 GB memory, and four RTX A6000 GPUs.Source: §A.4 · p. 17

B More Ablation Studies and Model Analysis

Additional analyses examine encoder objectives, regularization and prototype-length sensitivity, case-based explanations, and the effects of different base LLMs.

Ablations of learning objectives

  • The full encoder objective achieves the best prediction performance, while clustering and evidencing objectives support distinguishable prototypes and accurate projection onto training data.The study identifies both objectives as important to accurate prediction.Source: Figure 6 · p. 17, B More Ablation Studies and Model… · p. 17

Sensitivity of prototype lengths

Sensitivity of regularization strengths

  • Weather encoder performance remains relatively stable across regularization values, with moderate clustering and evidencing strengths and slight diversity constraint tending to help.The analysis reports that these objectives guide meaningful prototype learning and a compact interpretation prototype space.Source: B More Ablation Studies and Model… · p. 17, Table 4 · p. 18

Effect of different base LLMs

Effect of case-based explanations

  • On Weather and Finance, adding more relevant case-based explanations consistently improves prediction performance and serves as a proxy for explanation quality.The paper also notes LLM-as-a-judge as an alternative strategy for assessing explanation relevance and helpfulness.Source: B More Ablation Studies and Model… · p. 18, Figure 7 · p. 19

C Computation Cost for Iterative Optimization and Inference

The appendix quantifies the iterative workflow’s compute demands and describes ways to reduce them. It also presents learned prototypes across datasets as aligned with real-world scenarios.

C Computation Cost for Iterative Optimization and Inference

  • Each iteration over 4,000 training and validation samples uses about 9.74 M input tokens and 1.64 M output tokens, taking GPT-4o about 3.9 hours serially.With 20 concurrent calls, the iteration takes about 12 minutes; Gemini-Flash-2.0 reduces it to 8 minutes.Source: C Computation Cost for Iterative… · p. 19
  • Test-time refinement and prediction take 3.46 seconds per sample with GPT-4o or 2.29 seconds with Gemini-2.0-Flash.The authors describe these times as reasonable for real-world deployment.Source: C Computation Cost for Iterative… · p. 19
  • One or two iterations often yield clear gains; the loop can stop when validation improvements fall below a threshold or skip previously accurate training examples.The authors present these as strategies to make the method more computationally tractable.Source: C Computation Cost for Iterative… · p. 20

D.1 Multi-modal Prototypes for All Datasets

  • Across Weather, Finance, and two Healthcare datasets, learned prototypes from both modalities align with real-world ground-truth scenarios.The authors state that this alignment supports faithful explanations and enhances LLM predictions.Source: §D.1 · p. 20

D.2 Case-based Reasoning Example on the Finance Dataset

A Finance case study shows how refining text changes the case-based explanations toward market conditions consistent with a declining raw-material price trend. Time-series explanations provide a complementary view through declining movements across multiple indices.

D.2 Case-based Reasoning Example on the Finance Dataset

  • Refinement shifts the Finance example’s text from a neutral misprediction toward emphasis on subdued demand, thin margins, and bearish sentiment, consistent with a declining raw-material price trend.The refined explanation also highlights demand contraction, production constraints, and macroeconomic uncertainty rather than the original focus on inventory management and short-term stability.Source: §D.2 · p. 20
  • Time-series explanations identify declining price movements across multiple indices, complementing the text-based reasoning about raw-material prices.The example’s original text was predicted as neutral rather than as a decreasing trend.Source: §D.2 · p. 20
  • The appendix shows key learned prototypes for Weather and three other datasets; each figure row represents a time-series prototype.A separate Finance figure is identified as a multi-modal case-based reasoning example.Source: Figure 9 · p. 20, Figure 10 · p. 21, Figure 11 · p. 21

E Designed Prompts for Experiments

The appendix provides prompts for prediction, reflection, and refinement LLMs, alongside a structured generate-update-summarize reflection strategy. It also presents first-iteration reflection reports for four datasets.

E Designed Prompts for Experiments

F Reflection Reports for Text Refinement

G TimeXL for Regression-based Prediction: A Finance Demonstration

For Finance stock-price regression, TimeXL adds a regression branch and modifies the prediction prompt for continuous forecasts. The appendix reports that it outperforms the compared variants and illustrates prototype-based reasoning.

G TimeXL for Regression-based Prediction: A Finance Demonstration

  • The regression adaptation adds a branch that ensembles time-series segment representations as weighted prototype sums, uses a non-negative-weight regression head, and adds MSE and MAE losses.The prediction LLM prompt is adjusted to include time-series inputs and request continuous forecast values.Source: G TimeXL for Regression-based… · p. 26, Figure 24 · p. 26, Figure 25 · p. 27
  • TimeXL achieves the best results among the compared regression methods on Finance raw-material stock-price prediction.The authors also report that multi-modal MM-iTransformer and MM-PatchTST benefit from additional text, while Table 9 supports the ablation analysis.Source: Table 8 · p. 27, G TimeXL for Regression-based… · p. 27
  • Regression performance quickly improves and stabilizes over iterations, as reported in the iterative analysis.The authors link this pattern to alternating prediction and reflective refinement.Source: G TimeXL for Regression-based… · p. 27, Table 10 · p. 28
  • In the Finance case study, refined-text prototypes capture bearish sentiment, while similar time-series prototypes provide reference trends and correlate strongly with the input series.The original text also allocates neutral-class prototypes, indicating mixed sentiment and potential fluctuations that the authors associate with lower numerical prediction accuracy.Source: G TimeXL for Regression-based… · pp. 28–29, Figure 26 · p. 28

H More Detailed Discussions of Related Work

H.1 Multi-modal Time Series Analysis.

Prior work combines modalities to improve time-series tasks, while explanations range from saliency and surrogate methods to prototype-based reasoning. The authors position their approach as extending case-based explanations across temporal and contextual modalities, with iterative LLM interaction, amid underexplored rationales for contextual signals.

H.1 Multi-modal Time Series Analysis.

  • Multi-modal fusion and cross-modal alignment model interactions between time series and other modalities, but the rationale for contextual signals’ influence remains underexplored.Prior work also represents time series as images, text, tables, or graphs, or generates synthetic modalities, often to improve task performance.Source: §H.1 · p. 29

H.2 Time Series Explanation

  • Unlike saliency and surrogate explanations, case-based reasoning generates predictions and built-in explanations end-to-end from learned prototypes.The authors extend this paradigm to produce human-readable reasoning artifacts for temporal and contextual modalities.Source: §H.2 · p. 29

H.3 LLMs for Time Series Analysis

  • LLM time-series work includes fine-tuning, prompting, and knowledge-inference approaches; the authors distinguish their method through iterative, grounded, reflective interaction with an explainable model.Their approach provides case-based explanations from both modalities, highlighting time-series patterns and refined textual segments contributing to predictions.Source: §H.3 · p. 30

Related papers

Loading 2503.01013v4…