PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models Still Struggle to Zero-shot Reason about Time Series

Mike A. Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, Tim Althoff

arXiv:2404.11757v1cs.CL

TL;DR

It is unknown whether non-trivial zero-shot forecasting means language models can reason about time series, a modality important for decision-making across fields. The paper defines three reasoning tasks and builds a multidomain dataset of time series paired with text. Models show limited reasoning: they are near random on two tasks and make negligible gains from context in forecasting.

  • Problem

    Although language models have shown non-trivial forecasting, whether this implies time-series reasoning remains unknown, despite the modality’s importance across decision-making domains.

    Source: §1 · pp. 1–2

  • Method

    The paper defines etiological reasoning, question answering, and context-aided forecasting, and evaluates them using a synthetic multidomain dataset of time series paired with relevant text.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Models perform marginally above random on etiological reasoning and question answering, with humans up to thirty percentage points better, while GPT-4 gains negligibly from context in forecasting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results identify time-series reasoning as an impactful but underdeveloped direction for language-model research.

    Source: §8 · p. 10

    Because GPT-4 generated the data, its performance warrants caution, and its two-series MCQ evaluation may overestimate generalization.

    Source: §4 · p. 5, §5.2 · p. 7

Abstract

from arXiv · show

Time series are critical for decision-making in fields like finance and healthcare. Their importance has driven a recent influx of works passing time series into language models, leading to non-trivial forecasting on some datasets. But it remains unknown whether non-trivial forecasting implies that language models can reason about time series. To address this gap, we generate a first-of-its-kind evaluation framework for time series reasoning, including formal tasks and a corresponding dataset of multi-scale time series paired with text captions across ten domains. Using these data, we probe whether language models achieve three forms of reasoning: (1) Etiological Reasoning - given an input time series, can the language model identify the scenario that most likely created it? (2) Question Answering - can a language model answer factual questions about time series? (3) Context-Aided Forecasting - does highly relevant textual context improve a language model's time series forecasts? We find that otherwise highly-capable language models demonstrate surprisingly limited time series reasoning: they score marginally above random on etiological and question answering tasks (up to 30 percentage points worse than humans) and show modest success in using context to improve forecasting. These weakness showcase that time series reasoning is an impactful, yet deeply underdeveloped direction for language model research. We also make our datasets and code public at to support further research in this direction at https://github.com/behavioral-data/TSandLanguage

1 Introduction

The paper tests whether zero-shot forecasting ability amounts to time-series reasoning, proposing three reasoning tasks and a dataset to evaluate them. Models perform near random on etiological reasoning and question answering and make only negligible gains from context in forecasting.

1 Introduction

  • Time series carry information uncommon in language and are important in decision-making across fields including healthcare, finance, and agriculture.The introduction also names economics, political science, astronomy, and signal processing.Source: §1 · p. 1
  • Prior work shows non-trivial zero-shot forecasting, but whether that ability implies reasoning about time series remains unknown.The paper distinguishes interpreting temporal information from generating patterns that match inputs.Source: §1 · pp. 1–2
  • The framework treats time-series reasoning as etiological reasoning, question answering, and context-aided forecasting.Examples include inferring causes, comparing series in relation to outcomes, and using news context to forecast a stock trend.Source: §1 · p. 2
  • The evaluation dataset contains 230k time-series multiple-choice questions and 8.7k synthetic series-caption pairs across diverse scenarios.The captions describe both the series and the context in which it was observed.Source: §1 · p. 2
  • Humans outperform language models by up to thirty percentage points on etiological reasoning, while GPT-4 barely exceeds random; models also show limited question answering and negligible context-aided forecasting gains.Overall, the authors conclude that current models fail to reason about time series despite modest forecasting ability.Source: §1 · p. 2

2 Forms of Time Series Reasoning

The paper defines three forms of time-series reasoning: inferring causes, answering time-series-dependent questions, and using relevant text to improve forecasts. Its definitions specify what models should infer or predict from the series and context.

Forms of Time Series Reasoning

  • The authors present these definitions as a rigorous but non-exhaustive account of time-series reasoning.Source: §2 · p. 3

Etiological Reasoning

  • Etiological reasoning is the ability to hypothesize a time series’ cause, operationalized by assigning higher probability to its correct generating description than an incorrect one.The definition frames the task given a series and textual instructions as context.Source: §2 · p. 3

Question Answering

  • Question answering uses information in a time series to interpret queries about the series or events surrounding its scenario.Questions should depend on interpreting the series; answers should not be attainable without it.Source: §2 · p. 3

Context-Aided Forecasting

  • Context-aided forecasting uses relevant outside information to guide predictions of future observations from the observed series.The text description must provide meaningful information about the series; the formal criterion compares forecasts with and without that description.Source: §2 · p. 3

3 Dataset

The dataset pairs synthetic time series with relevant captions to support evaluation of time-series reasoning. The authors generate signals using executable code, curate the scenarios, and assess whether the data support reasoning tasks.

Dataset construction

  • The dataset pairs synthetic multidomain time series with highly relevant captions, addressing the need for linked series and text to evaluate reasoning.GPT-4 generates descriptions of changing environments alongside executable Python functions that produce corresponding series.Source: §3 · pp. 3–4, Figure 2 · p. 4
  • The generation process yields scenario captions, series characteristics, executable functions, and metadata, then filters the collection to 8.7k scenarios across ten domains.The authors encourage diversity by prompting against recent generations and remove multivariate series and those with complex, missing, or infinite values.Source: §3 · p. 4

Dataset quality

  • Human subjects perform far above random and substantially outperform existing language models, which the authors take as evidence that the data and prompts contain enough information for stronger performance.Source: §3 · p. 4

4 Etiological Reasoning: Near Random Performance

The section tests whether language models can infer the causes of time series from the series itself. Models perform poorly against human annotators, and results from multimodal models are similarly poor.

4 Etiological Reasoning: Near Random Performance

  • Models choose among four captions using only time-series values, while human annotators label a sample of 500 examples using plots without axis labels.The human evaluation used ten annotators with data-science and time-series expertise; the authors caution that their results may not represent the upper bound of human performance.Source: §4 · p. 5
  • GPT-4-Vision reaches 34.7% accuracy, over 30 percentage points below humans at 66.1%; LLAMA is near random at 27.3%.Across the models, performance is far below the human baseline.Source: §4 · p. 5, Table 1 · p. 5
  • Training existing multimodal models on the data also produced similarly poor performance.Source: §4 · p. 6
  • Taken together, the results indicate that current zero-shot language models are poor judges of time series etiology.Source: §4 · p. 6

5 Question Answering: Trailing Behind Human-Level Proficiency

The section develops question-answering tasks intended to require interpreting time series, then tests whether language models can answer them. Models perform near randomly on meaningful tasks, while humans do substantially better on the same information.

5 Question Answering: Trailing Behind Human-Level Proficiency

  • The section frames QA as interpreting time-series behavior or scenario implications, with questions designed to be unanswerable without the series.The task is evaluated with four-option multiple-choice questions.Source: §5 · p. 6

5.1 Questions About One Time Series

  • Single-series questions are unsuitable for evaluating time-series reasoning because models achieve 78–92% accuracy without seeing the series.The authors attribute this performance to parametric knowledge and caution that GPT-4 generated the data and questions.Source: §5.1 · pp. 6–7

5.2 Questions About Two Time Series

  • The two-series procedure generates counterfactual series and questions about their similarities and differences, then filters out questions GPT-3.5 can answer without either series.Almost half of the questions are discarded during filtering.Source: §5.2 · p. 7
  • With two time series, most language models fall close to random performance; GPT-4 is the only model with meaningful performance, but adding the series does not meaningfully improve it.Humans score 67% with the same information, significantly higher than the models.Source: §5.2 · p. 7

5.3 Manually-Perturbed MCQs

  • After minor answer-option perturbations, GPT-4’s performance falls to near random, suggesting that its earlier above-random results may not generalize beyond MCQs it generated.On manually created real-world questions, GPT-4 scores 36.6%, but performs similarly with and without the time series.Source: §5.3 · p. 8
  • On meaningful QA tasks, language models perform at or near random, and none performs better with than without the time series; humans do significantly better with the same information.The authors conclude that current language models cannot perform time-series reasoning in these zero-shot evaluations.Source: §5.3 · p. 8

6 Context-Aided Forecasting

The section tests whether relevant textual context helps language models forecast time series. Adding context changes forecasts only marginally, and language-model performance remains comparable to or worse than the median baseline.

6 Context-Aided Forecasting

  • GPT-4 forecasts the final 20% of 2,000 series after receiving the first 80% and corresponding captions, descriptions, or metadata; the study measures MAE and MSE after normalization.The baseline repeats the median of the first 80% throughout the forecast window.Source: §6 · p. 8
  • Adding all textual context barely changes MAE: only 1,040 of 2,000 series improve, while overall performance remains below or comparable to the Predict Median baseline.The figure reports marginal improvement and performance that is the same as or worse than the baseline.Source: §6 · pp. 8–9, Figure 3 · p. 9
  • The authors conclude that current language models largely fail to use context to inform forecasting, although the median baseline is particularly weak for these series.The series were designed to include interruptions from external events.Source: §6 · p. 9

7 Related Work

Prior work spans time-series datasets and language-model approaches, but the paper emphasizes evaluating the relationship between time series and text beyond forecasting and classification. Its cross-domain dataset leads it to use a zero-shot forecasting baseline.

7.1 Datasets for Time Series and Language

  • Unlike datasets focused exclusively on time series, this work targets the relationship between time series and text and reasoning beyond forecasting and classification.Prior question-answering datasets cited here focus on ECG and activity-recognition features or on tweets and historical stock prices.Source: §7.1 · p. 9

7.2 Language Models and Time-Series

  • Prior language-model approaches either fine-tune models for specific tasks and datasets or input specially tokenized series for forecasting, imputation, and classification.Source: §7.2 · pp. 9–10
  • Because each series comes from a different setting, fitting domains individually or transferring between sufficiently similar sequences is unsuitable, so the authors use zero-shot LLM-TIME as their baseline.Source: §7.2 · p. 10

8 Conclusion

The conclusion identifies three forms of time-series reasoning and evaluates them with a first-of-its-kind dataset of time series and relevant text. The results point to substantial gaps between language models and humans on two tasks and mediocre forecasting performance.

8 Conclusion

  • The paper evaluates etiological reasoning, question answering, and context-aided forecasting using its first-of-its-kind dataset of time series and highly relevant text.The conclusion frames these as three forms of time-series reasoning.Source: §8 · p. 10
  • Language models show a substantial gap from human performance on etiological reasoning and question answering, alongside mediocre performance on context-aided forecasting.Source: §8 · p. 10
  • The authors identify opportunities for NLP research to develop models that can deeply reason about critical time-series data.Source: §8 · p. 10

A.1 Numerical Tokenization

LLM-TIME preprocesses numerical observations by z-score normalizing them and scaling them to a constant power of ten. The section also notes subtle tokenization differences between GPT-3 and Llama.2.

A.1 Numerical Tokenization

  • LLM-TIME first z-score normalizes the observations, then scales them to a constant power of ten, such as 1e3.Source: §A.1 · p. 14
  • The example transforms 20.88, 20.20, and 20.48 into 1052, 1017, and 1032 after normalization and scaling.Source: §A.1 · p. 14
  • The authors note subtle differences in tokenization between GPT-3 and Llama.2.Source: §A.1 · p. 14

B.1 Training Multimodal Models on Etiological Reasoning Task

This section examines multimodal approaches and question-generation and prompting procedures for etiological reasoning, including single-series questions and comparisons of original and counterfactual time series. It also describes scenario and question construction and reports that trained models remain near chance, often relying on answer similarity rather than time-series evidence.

B.1 Training Multimodal Models on Etiological Reasoning Task

  • All trained multimodal models perform within an epsilon of random (25%) on etiological reasoning, indicating limited capacity to learn relationships between time series and text.Training held out the Health and Medical Data, Agricultural and Food Production, and Educational and Public Services categories; approaches included spectrogram-based and plotted-series inputs and a TimesNet encoder.Source: §B.1 · pp. 14–15

B.2 MCQ Generation using other LMs

  • GPT-4 performed significantly on MCQs generated using LLAMA-13B and GEMMA-7B even without time-series information, suggesting these models’ questions relied on limited textual descriptions of series dynamics.The evaluation used 100 time-series examples and almost 1,000 MCQs per language model.Source: §B.2 · p. 15

B.3 Using Different Methods to Prompt Time-series

  • Comma-separated values and LLM-TIME tokenization produced insignificant differences in MCQ-answering ability, although LLM-TIME yielded slightly better performance.The comparison evaluates alternative ways to pass time series to a language model.Source: §B.3 · pp. 15, 17

B.4 Examples of Single Time Series MCQs

  • In manual perturbation tests, language models selected altered answer options as well as correct ones, suggesting they often rely on similarity to their originally generated options rather than time-series evidence.The authors use the perturbation procedure to test robustness to modified versions of GPT-4-generated MCQs.Source: §B.5 · p. 18

B.6 Handcrafted MCQs

  • The handcrafted MCQs for Massachusetts COVID-19 cases ask about the dates of minimum and maximum values, the series’ visual pattern, and the difference between two peaks.The examples are based on a real-world time series.Source: §B.6 · p. 19

C Prompt For Scenario Generation

  • The prompt requests an external-event time-series scenario specifying sample rate and sampling duration, plus up to five expected characteristics covering patterns and event-driven distribution shifts in the data-generating process.The description must be under 100 words; the series must be under 1,000 observations, single-variable, have no missing values, and contain no values greater than 1e6. Its short and tiny summaries are limited to 25 words and three words, respectively.Source: C Prompt For Scenario Generation · p. 20

D.1 Prompt for Single-Time-Series MCQs

  • The single-series MCQ prompt requests five counterfactual question-answer pairs that require understanding time-series dynamics and avoid direct references to the description or code.It supplies the model with the series description, characteristics, and generating code as inputs.Source: §D.1 · pp. 20–21

D.1 Prompt for Single Time-Series MCQs

  • For single-series MCQs, the prompt adds three incorrect options of similar length to each question’s correct answer and requires a consistent one-line format without contractions.It instructs the model to preserve the question and answer and provide no additional text.Source: §D.1 · p. 21

D.2 Prompt for Multiple Time-Series MCQs

  • The appendix introduces a staged procedure for generating MCQs at the intersection of multiple time series.The described stages cover counterfactual scenario creation, generation of an updated series, and questions comparing the original and updated series.Source: §D.2 · p. 21, §D.2.1 · p. 21, §D.2.2 · p. 21

D.2.1 Creating a list of ’what-if’ scenarios for a time series

  • The prompt requests five distinct, consequential “what-if” questions based on a series description and generating code, with scenarios that are readily implemented by modifying the code.Questions should omit values from the original series and code and avoid references to random noise or its generator.Source: §D.2.1 · p. 21

D.2.2 Creating a new time-series

  • GPT-4 generates code for a counterfactual series that preserves the original length while making the series significantly different and realistic.The prompt allows adjustments to predefined bounds and offsets when needed to implement the scenario while maintaining realistic behavior.Source: §D.2.2 · pp. 21–22

D.3 Creating MCQs

  • The comparison MCQs ask five difference and five similarity questions, with answers and three incorrect options for each.Questions target visible patterns, trends, and rates of change; they avoid series length and random noise, and answers should be short and approximate where numerical values are used.Source: §D.3 · p. 22

E Additional Results for Context-Aided Forecasting

Adding textual context produces only slight forecasting improvements, even when all captions are supplied. Examples show both successful use of context and failures to interpret captions correctly.

E Additional Results for Context-Aided Forecasting

Loading 2404.11757v1…