PaperlayerView exact revision on arXiv

Source-linked AI summary

TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models

Fangxu Yu, Xingang Guo, Lingzhi Yuan, Haoqiang Kang, Hongyu Zhao, Lianhui Qin, Furong Huang, Bin Hu, Tianyi Zhou

arXiv:2601.18744v2cs.AIcs.LG

TL;DR

Generalist-model benchmarks largely overlook comprehensive time series reasoning despite its importance for real-world problem-solving. TSRBENCH addresses this gap with a multimodal benchmark of 4125 problems from 14 domains across four dimensions and 15 tasks. Experiments show strong perception performance, scaling breakdowns for prediction, and persistent gaps across model families.

  • Problem

    Existing evaluations often treat time series as isolated numerical sequences or cover only surface patterns, narrow domains, or restricted task scopes.

  • Method

    TSRBENCH evaluates 4125 problems from 14 domains across four dimensions and 15 tasks using textual, visual, interleaved, and embedding-based time series representations.

  • Results

    Models perform strongly on perception but struggle with reasoning, prediction, and decision-making; scaling benefits most dimensions but breaks down for prediction, while GPT-5 (T+V) reaches 55.6% overall accuracy.

  • Takeaways & Limitations

    TSRBENCH provides a standardized platform for evaluating the full spectrum of time series reasoning in generalist models.

Abstract

from arXiv · show

Time series are ubiquitous in real-world scenarios and crucial for applications ranging from energy management to traffic control. Consequently, the ability to reason over time series is a fundamental skill for generalist models to solve complex problems. However, current benchmarks for generalist models largely overlook this dimension. To bridge this gap, we introduce TSRBench, a comprehensive multi-modal benchmark designed to stress-test the full spectrum of time series reasoning capabilities. TSRBench features: i) a diverse set of 4125 problems from 14 domains, and is categorized into 4 major dimensions: Perception, Reasoning, Prediction, and Decision-Making. ii) 15 tasks from the 4 dimensions evaluating essential reasoning capabilities (e.g., numerical reasoning). Through extensive experiments, we evaluate over 30 leading proprietary and open-source LLMs, VLMs, and TSLLMs within TSRBench. Our findings reveal that: i) scaling laws hold for perception and reasoning but break down for prediction; ii) strong reasoning does not guarantee accurate context-aware forecasting, indicating a decoupling between semantic understanding and numerical prediction; and iii) despite the complementary nature of textual and visual forms of time series as inputs, current multimodal models fail to effectively fuse them for reciprocal performance gains. TSRBench provides a standardized evaluation platform that not only highlights existing challenges but also offers valuable insights to advance generalist models. Our code and dataset are available at https://tsrbench.github.io/.

1. Introduction

Time series reasoning is important for generalist models, but existing evaluations often omit semantic context and comprehensive reasoning. TSRBENCH addresses this gap with a large, multimodal benchmark spanning domains, capabilities, and model input formats.

  • Time series reasoning supports practical problem-solving because real-world information and decisions are often inherently temporal.
  • Existing benchmarks often reduce time series to numerical sequences, target surface-level patterns, or cover narrow domains and task scopes.
  • TSRBENCH contains 4125 problems collected from 14 domains to assess generalist models’ time series problem-solving abilities.
  • The benchmark organizes abilities into Perception, Reasoning, Prediction, and Decision-Making across 15 tasks.
  • TSRBENCH supports textual, visual, interleaved text-image, and time series embedding modalities for evaluating generalist models.
  • The evaluation includes proprietary and open-source LLMs, VLMs, and Time Series LLMs using text, vision, and combined representations.

2. Related Work

Prior time series benchmarks primarily evaluate conventional analysis tasks, while general reasoning benchmarks cover broad subjects without systematically testing temporal dynamics. TSRBENCH is introduced to provide a dedicated, comprehensive framework for time series reasoning.

  • Traditional time series benchmarks mainly target forecasting, classification, imputation, and anomaly detection.
  • General reasoning benchmarks assess knowledge and problem-solving across broad subjects, including science, engineering, embodied reasoning, and video reasoning.
  • The related benchmark landscape includes tasks involving pattern recognition, noise understanding, temporal relations, forecasting, weather, clinical management, and strategy selection.
  • Existing general reasoning benchmarks may include isolated time-series tasks but lack a comprehensive framework dedicated to temporal dynamics.

3. TSRBENCH

TSRBENCH is a multi-domain benchmark that evaluates four dimensions of time series reasoning through diverse tasks and carefully constructed context-aligned data. Its task taxonomy spans perception, reasoning, prediction, and decision-making.

  • 3.1. Overview of TSRBENCH: TSRBENCH comprises 4125 instances and 15250 time series channels across 14 domains and four reasoning dimensions.
  • 3.2. Time Series Perception: Perception covers Pattern Recognition, Noise Understanding, Anomaly Detection, and Comparative Analysis.
  • 3.2. Time Series Perception: Perception tasks assess temporal structure, noise magnitude, anomaly types, and shared or differing properties across series.
  • 3.3. Time Series Reasoning: Reasoning tasks derive conclusions from temporal patterns and prior knowledge, including causal, temporal, numerical, deductive, and inductive reasoning.
  • 3.4. Time Series Prediction: Prediction includes forecasting future numerical values from historical observations and contextual events, plus predicting future discrete events.
  • 3.5. Time Series Decision-Making: Decision-making evaluates qualitative decisions from patterns and context and quantitative choices by comparing outcomes under rules and environmental constraints.
  • Dataset construction prioritizes text-time-series alignment, domain diversity, and generalizability.

4. Experiments

Experiments across TSRBench show broad variation in model performance and a persistent mismatch between scaling benefits for most capabilities and forecasting performance. The strongest proprietary and open-source models still differ substantially in overall accuracy.

  • Overall Performance: Current generalist models perform strongly on perception but struggle with reasoning, prediction, and decision-making tasks.This pattern is reported across the aggregate results in Table 2.
  • Overall Performance: GPT-5 (T+V) achieves the highest overall accuracy at 55.6%, while Qwen2.5-72B and Qwen3-VL-32B lead open-source LLMs and VLMs at 42.4% and 44.9%.The proprietary result uses both textual and visual time series.
  • Reasoning Effects: Reasoning effort improves overall accuracy, with o4-mini-high gaining 4.3% and GPT-5-mini-high gaining 7.2% over their respective baseline T+V models.A substantial gap nevertheless remains between GPT-5 (T+V) at 55.6% and Qwen3-VL-32B at 44.9%.
  • Scaling Effects: Model size positively correlates with performance on most perception, reasoning, and decision-making tasks for both LLMs and VLMs.The benchmark analyzes these relationships using Spearman’s rank correlation.
  • Scaling Effects: Prediction tasks diverge from the scaling trend for both LLMs and VLMs, indicating persistent difficulty with effective forecasting even when models are scaled and given context.The same exception is summarized as the breakdown of scaling laws for prediction.

Appendix F.2 on correlations for each task.

TSRBENCH analyses how generalist models perform across time-series reasoning dimensions, modalities, task difficulty, inference-time computation, and failure types. The findings expose weak prediction, limited cross-modal fusion, uneven task difficulty, and bottlenecks in temporal perception and rigorous reasoning.

  • Task correlations: Perception, Reasoning, and Decision are highly correlated, but each has weak correlation with Prediction.Models can understand and reason about time series while still struggling to forecast numerical series and events.
  • Modality complementarity: Textual and visual representations achieve comparable overall accuracy but succeed on different subsets of samples, while T+V adds no significant gain.The union of single-modality solutions is high, whereas their intersection is low; T+V answers largely overlap with answers from either modality alone.
  • Task difficulty: High-variance tasks such as Abductive Reasoning and Event Prediction show uneven model performance, whereas Quantitative Decision-Making and Time Series Forecasting show uniformly low accuracy and variance.The former pattern suggests distillation opportunities, while the latter indicates a shared weakness requiring richer quantitative and temporal supervision.
  • Inference-time scaling: Reducing inference-time computation sharply degrades Reasoning, Prediction, and Decision-Making while leaving Perception relatively robust.The results suggest that logical conclusions from perceived temporal patterns require more deliberative computation than intuitive temporal perception.
  • Error analysis: Error analysis identifies deficiencies in perceiving temporal patterns and performing rigorous reasoning based on those perceptions as critical bottlenecks.The analysis examined 150 failure instances from three models using a four-tier taxonomy covering Reasoning, Perception, Question Understanding, and Domain Knowledge.

A.2. Benchmarks for Generalist Models.

Generalist-model benchmarks expanded from small-scale, single-task natural-language evaluations toward benchmarks assessing broader reasoning and problem-solving capabilities.

  • Early generalist-model benchmarks such as GLUE, BERTScore, and SuperGLUE primarily evaluated natural-language understanding through small-scale, single-task tests.

B. Future Research Directions

TSRBENCH identifies several directions for improving time-series reasoning, including multimodal fusion, pretrained time-series models, multi-agent systems, and adaptive test-time reasoning.

  • Multi-view Time Series Understanding: Current models struggle to fuse textual and visual time-series representations despite their complementarity.Future work should develop alignment techniques combining high-resolution visual patterns with semantic textual context.
  • Large-scale Pretrained Time Series Models: Generalist models show collective blind spots in quantitative forecasting, motivating foundation models pretrained on massive, diverse time-series corpora.The proposed goal is to bridge semantic reasoning and precise numerical extrapolation.
  • Multi-agent Time Series Systems: Complex time-series problems require capabilities spanning pattern recognition, logical deduction, and domain-knowledge retrieval.A multi-agent framework could assign specialized agents to collaborate on and verify predictions.
  • Test-time Scaling Approaches: Reasoning-intensive tasks suffer without sufficient inference-time computation.The paper reports that increased reasoning efforts benefit o4-mini and GPT-5-mini and calls for adaptive reasoning strategies.

D.4. Data Contamination & Quality Control

TSRBENCH applies leakage detection and structured analysis to support data quality and model evaluation, including checks for contamination and statistical properties.

  • Data Contamination Detection: The benchmark detects possible contamination by comparing model-predicted 5-grams with dataset 5-grams at randomly selected positions.A sample is considered contaminated when the predicted and actual 5-grams match.
  • Data Contamination Detection: Most evaluated models exhibit low n-gram accuracy, indicating low data leakage.The leakage analysis is performed on o4-mini, GPT-5-mini, and GPT-5.
  • Deterministic Analysis: A deterministic analysis module injects structured statistical summaries alongside raw time-series inputs.Its components include statistical profiling, trend detection, extremum identification, change-point detection, and series comparison.
  • Deterministic Analysis: Statistical profiling provides mean, standard deviation, range, median, variance, skewness, and kurtosis to characterize scale, shape, asymmetry, and tail behavior.These metrics give the model a global view of the series distribution.

E.3. Trend Analysis

The benchmark’s trend analysis quantifies trajectories with OLS, identifies meaningful extrema and change points, and compares multiple series using correlation and statistical tests.

  • Trend Analysis: Ordinary Least Squares models the series as x_t = β_1t + β_0 + ϵ to quantify its trajectory.The analysis reports trend-related statistics derived from this regression.
  • Trend Analysis: The slope β_1 indicates trend direction and magnitude, while R^2 measures the proportion of variance predictable from the time index.Trend strength is categorized using Pearson correlation thresholds.
  • Extrema and Change Points: Local peaks and valleys are detected with a prominence threshold so that only significant structural extrema are reported.The threshold uses the series standard deviation to reduce context noise.
  • Extrema and Change Points: Change points are flagged when first-order differences exceed a statistical threshold, capturing sudden shocks or regime changes.This targets shifts that may be difficult to infer from raw tokens alone.
  • Comparative Analysis: For multiple series, the analysis uses Pearson correlation, cross-correlation with lag, and Welch’s t-test to compare relationships and means.These tests assess linear association, temporal alignment, and statistical differences.
  • Benchmark Coverage: 58.1% of TSRBENCH samples are multivariate, covering reasoning over multi-channel signals in addition to univariate series.Multivariate denotes instances with more than one time-series channel presented jointly.

Perception Error

The perception error arose from misreading the time-series regime and selecting MACD instead of the ground-truth Bollinger Bands strategy for minimizing maximum drawdown.

  • Perception Error: The model chose MACD despite the ground truth identifying Bollinger Bands as the best strategy for maximum drawdown.Its rationale characterized the series as relatively steady and not highly volatile.
  • Perception Error: The error analysis attributes the mistake to misreading an oscillatory or mean-reverting regime.Bollinger entries near troughs and exits near peaks can limit time in market and reduce peak-to-trough equity declines.
  • Perception Error: MACD can remain exposed during adverse swings until a crossover, allowing larger equity drawdowns.The failure to ground the conclusion in the series’ features led to choosing MACD over the mean-reversion strategy.

Reasoning Error

The Lorenz-system answer is unsupported because the reasoning stops after only two Euler steps, despite the task requiring a complete 120-step, precision-sensitive comparison. Full high-precision integration identifies option C, not the model’s option A.

  • Task requirement: The task requires selecting the x(t) sequence that exactly represents the Lorenz system for all 120 steps.The system is specified by deterministic coupled equations, parameters, and initial conditions.
  • Result: Full high-precision integration and comparison show that option C is correct, whereas option A does not match.The candidate answer passage records the ground truth as C, while the model selected A.
  • Reasoning error: Early agreement across candidate sequences cannot establish the correct answer because discriminating differences emerge at later steps in chaotic dynamics.The model did not show mid- or late-step calculations or element-wise candidate comparisons.
  • Reasoning error: The model failed to simulate all 120 steps and address rounding or precision alignment, leaving its conclusion unsupported.The prompt explicitly emphasizes extreme precision for this task.

Domain Knowledge Error

The examples show errors caused by applying domain formulas incorrectly and by misunderstanding task-specific matrix semantics. These mistakes produce quantitatively wrong answers even when the model’s qualitative estimates or heuristics appear plausible.

  • Evapotranspiration: The FAO-56 task asks for cumulative reference evapotranspiration from meteorological time-series variables over the observed period.Inputs include temperature, relative humidity, net radiation, and wind speed.
  • Evapotranspiration: The model selected 216.3 mm, but the ground-truth answer is 166.4 mm.The model’s estimate was based on daily values inferred from the graph and an incorrect formula application.
  • Evapotranspiration: Omitting the FAO-56 wind-speed correction term caused approximately 27% inflation of the cumulative ET0 estimate.The incorrect denominator omitted the factor (1 + 0.34u2), underestimating aerodynamic resistance.
  • Causal matrix: The river-sensor task defines adjacency-matrix columns as causes and rows as resulting nodes, with 1 indicating a causal effect.Correctly applying this orientation is necessary to interpret each matrix entry.
  • Causal matrix: The model reversed the matrix orientation and used magnitude spikes as a weak causality heuristic, selecting D instead of the ground-truth A.Under the stated convention, D[2,3]=1 means node 3 causes node 2, not the reverse.
Loading 2601.18744v2…