TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models

• Existing benchmarks often reduce time series to isolated numerical sequences, target surface-level patterns, or remain limited to narrow domains and task scopes.The introduction argues that these limits motivate a comprehensive, multidimensional evaluation framework.Source: §1 · p. 1
• TSRBENCH contains 4,125 problems from 14 domains, organized into four dimensions and 15 tasks, and supports text, image, interleaved text-image, and time-series embedding inputs.Source: §1 · pp. 1–2
• Models perform relatively well on perception but struggle with complex reasoning, forecasting, and decision-making; scaling generally helps reasoning but not prediction.The reported scaling pattern holds for most reasoning tasks on LLMs and VLMs, with time-series prediction as the exception.Source: §1 · p. 2
• Time-series prediction tasks have weak relationships with other tasks, distinguishing forecasting performance from the broader task set.Source: §1 · p. 2
• Textual and visual time-series representations are complementary, but current models struggle to use both simultaneously for better performance.The representations often solve different sets of problems.Source: §1 · p. 2
• A unified evaluation setup maps time series to number sequences for LLMs, plots for VLMs, and embeddings for TSLLMs, while testing proprietary models with text, vision, and combined inputs.The evaluation covers 6 proprietary models, 13 open-source LLMs, 13 open-source VLMs, and 4 open-source TSLLMs.Source: §1 · p. 2

Time Series Benchmarks

• Existing time-series benchmarks center on forecasting, classification, imputation, and anomaly detection, while newer LLM evaluations remain narrow or task-specific.TimeSeriesExam focuses on holistic perception, MTBench on finance and weather, and TSR-SUITE and EngineMT-QA on narrow reasoning tasks; TimeMQA emphasizes traditional analysis.Source: §2 · p. 2

General Reasoning Benchmarks

• General reasoning benchmarks span broad subject, science, engineering, social, and multimodal domains, but lack systematic evaluation dedicated to temporal dynamics.The related work identifies TSRBENCH as filling this gap.Source: §2 · pp. 2–3
• TSRBENCH evaluates generalist models across Perception, Reasoning, Prediction, and Decision-Making.These four capabilities define the benchmark overview.Source: Figure 1 · p. 3, §2 · p. 3

3.1. Overview of TSRBENCH

• TSRBENCH comprises 4,125 instances and 15,250 time-series channels across 14 domains, covering four dimensions of time-series reasoning.The dimensions are Perception, Reasoning, Prediction, and Decision-Making.Source: §3.1 · p. 3

3.2. Time Series Perception

• Perception evaluates pattern recognition, noise characterization, anomaly identification and classification, and comparison of multiple series.These tasks cover temporal properties and statistics, anomaly types and underlying patterns, and shared patterns, statistical properties, distributions, or trend direction.Source: §3.2 · pp. 3–4

3.3. Time Series Reasoning

• Seven reasoning tasks assess causal, abductive, temporal, numerical, deductive, and inductive inference, as well as inference about generative sources.Inductive reasoning requires inferring rules from historical observations and domain knowledge, then applying them to predict specific future events rather than relying on curve-fitting.Source: §3.3 · p. 4

3.4. Time Series Prediction

• Prediction covers numerical forecasting conditioned on historical observations and contextual events, plus anticipation of future discrete events.Forecasting is posed as multiple-choice questions; event prediction combines pattern analysis with commonsense or domain-specific reasoning.Source: §3.4 · p. 4

3.5. Time Series Decision-Making

• Decision-making tests context-informed qualitative choices and quantitative selection of the best operational procedure under rules and environmental constraints.Quantitative decisions require comparing resulting performance metrics across procedures.Source: §3.5 · p. 4

3.6. Data Collection Principles for TSRBENCH

• Data collection prioritizes aligned text and time series, domain diversity, verifiable ground truth, and synthetic data for quantitative reasoning.Simulations provide unambiguous, noise-free data-generating processes for assessing precise numerical reasoning and deductive logic where real-world data may lack high-precision ground truth.Source: §3.6 · pp. 4–5

4.1. Experimental Setups.

• The evaluation covered proprietary and open-source LLMs, VLMs, and time series LLMs, using accuracy as its primary metric.Time series were provided as numeric sequences to LLMs, plots to VLMs, and projector embeddings to TSLLMs; charts were rendered at 100 PPI.Source: §4.1 · p. 5

4.2. Main Results

• Models performed strongly on perception but struggled with reasoning, prediction, and decision-making; high reasoning improved o4-mini-high by +4.3% and GPT-5-mini-high by +7.2% versus their respective baseline (T+V) models.Time series LLMs performed competitively with similarly sized LLMs and VLMs, but remained behind advanced models.Source: §4.2 · p. 5, Table 2 · p. 6
• 55.6% overall accuracy was highest for GPT-5 (T+V).The strongest open-source LLM and VLM scored 42.4% for Qwen2.5-72B and 44.9% for Qwen3-VL-32B, respectively; a substantial gap remained between GPT-5 and the best open-source model.Source: §4.2 · p. 5, Table 2 · p. 6

4.3. Further Findings

• Scaling correlated positively with performance on Perception, Reasoning, and Decision-Making, but Prediction diverged from this trend for both LLMs and VLMs.The findings indicate that generalist models continue to struggle with forecasting even when scaled and given context.Source: §4.3 · pp. 5–6, Table 3 · p. 6

Appendix F.2 on correlations for each task.

• Perception, Reasoning, and Decision-Making are strongly correlated, but each shows weak correlation with Prediction across LLMs and VLMs.The authors interpret this gap as evidence that understanding and reasoning about time series do not ensure accurate forecasting of numerical series or events.Source: Appendix F.2 · p. 6
• Textual and visual inputs achieve comparable overall accuracy but solve different samples; combining them yields no significant gain as current models largely reuse single-modality solutions.Visual representations outperform text in Perception, but that advantage diminishes in Reasoning, Prediction, and Decision-Making; the low intersection and high union of successes indicate complementary strengths.Source: Appendix F.2 · p. 7, Figure 5 · p. 7
• High-variance tasks such as Abductive Reasoning and Event Prediction may benefit from distillation, whereas uniformly poor low-accuracy tasks point to a need for richer quantitative and temporal pretraining data.Quantitative Decision-Making and Time Series Forecasting are identified as low-accuracy, low-variance tasks.Source: Appendix F.2 · p. 7, Figure 6 · p. 7

4.4. Further Investigation

• Tool-augmented statistical context yields modest overall improvement, while 100 PPI visualizations outperform the tested lower and higher resolutions.Explicit time-series features partially compensate for weak temporal understanding but do not resolve it in all settings; textual input mitigates the degradation at low visual resolutions.Source: Table 4 · p. 7, §4.4 · p. 8, Figure 8 · p. 8
• Reducing inference-time computation leaves Perception comparatively robust but sharply degrades Reasoning, Prediction, and Decision-Making.The results suggest that drawing logical conclusions from temporal patterns requires more deliberative computation than perceiving them.Source: §4.4 · p. 8

4.5. Error Analysis

• Across three evaluated models, Reasoning and Perception errors predominate, while Question Understanding and Domain Knowledge errors are marginal.The analysis attributes the main bottleneck to perceiving temporal patterns and reasoning rigorously from them, rather than to knowledge or linguistic comprehension.Source: §4.5 · pp. 8–9

5. Conclusion; Impact Statement

• TSRBENCH provides a standardized platform for evaluating time-series understanding and reasoning, with the broader aim of identifying blind spots relevant to high-stakes domains.The paper highlights healthcare, finance, and industrial systems as areas where stronger temporal reasoning could have profound implications.Source: §5 · p. 9, Impact Statement · p. 9

A.1. MLLM/LLM Reasoning.

• A.1 traces LLM/MLLM reasoning from prompt-engineered steps through test-time computation to internalized reasoning, while adaptation of these approaches to time series remains underexplored.Source: §A.1 · p. 15

A.2. Benchmarks for Generalist Models.

• Early benchmarks GLUE (Wang et al., 2018), BERTScore (Zhang et al., 2019), and SuperGLUE (Wang et al., 2019) focused on small-scale, single-task natural language understanding.Source: §A.2 · p. 15
• As generalist models scaled, MMMU (Yue et al., 2024), BIG-bench (Srivastava et al., 2023), Q-Bench (Wu et al., 2023), and Seed-Bench (Li et al., 2023) broadened evaluation to reasoning, factual knowledge, and visual recognition.Source: §A.2 · p. 15
• Specialized benchmarks assess science reasoning (Cobbe et al., 2021; Xu et al., 2025a; Lu et al., 2023; Zhong et al., 2025), social reasoning (Guha et al., 2023; Zhang et al., 2024a; Kim et al., 2023), and engineering reasoning (Chen, 2021; Jimenez et al., 2023; Yu et al., 2018; Guo et al., 2025b; Chen et al., 2025a; Guo et al., 2025c).Source: §A.2 · p. 15
• Together, these benchmarks reflect a shift from coarse-grained language understanding toward fine-grained, domain-specific, comprehensive assessment of complex reasoning abilities.Source: §A.2 · p. 15

B. Future Research Directions

• Future models should align high-resolution visual patterns with semantic textual context to improve holistic time-series understanding.Source: B. Future Research Directions · p. 15
• Massive, diverse time-series pretraining is proposed to address forecasting blind spots and bridge semantic reasoning with precise numerical extrapolation.Source: B. Future Research Directions · p. 15
• The authors propose specialized collaborating agents to decompose and verify complex tasks, alongside adaptive test-time reasoning for reasoning-intensive tasks.They report that increased reasoning efforts substantially benefit reasoning performance on o4-mini and GPT-5-mini.Source: B. Future Research Directions · pp. 15–16

C. Model Versions

• Proprietary models were accessed through APIs, while open-source models were locally deployed using VLLM (Kwon et al., 2023).Source: C. Model Versions · p. 16

D. Data Collection

• TSRBENCH’s data pipeline addresses sparse, noisy, temporally misaligned text through raw-data collection, human-designed questions, ground-truth generation, and verification.Text sources require extensive gathering and filtering, and temporal alignment must match reports to numerical time steps and relevant durations.Source: D. Data Collection · p. 17

D.1. Data Acquisition

• A dual-stream acquisition strategy combines synthetic series for sparse or difficult domains with public web data verified for textual alignment and anonymized to reduce leakage.Synthetic generation functions provide controlled, known parameters; web data is checked for strict temporal alignment by human annotators.Source: §D.1 · p. 17

D.2. Question & Answer Generation

• Questions use manually designed templates; synthetic answers come from code-based calculations, while web answers use rule-based extraction from sources or associated records.Source: §D.2 · p. 17

D.3. Answer Verification

• A two-stage verification mechanism re-executes synthetic generation logic or cross-checks web answers against ground truth, and validates multiple-choice distractors algorithmically or with LLMs.Source: §D.3 · p. 18

D.4. Data Contamination & Quality Control

• The contamination check combines each problem, textual time series, and solution, then uses n-gram accuracy to detect potential leakage.The procedure follows Xu et al. (2024).Source: §D.4 · p. 18
• Among tested best-performing models o4-mini, GPT-5-mini, and GPT-5, most exhibit low N-gram accuracy, indicating low data leakage, using 5-grams extracted from 20 positions.Table 7 reports data-leakage detection proportions for TSRBENCH.Source: Table 7 · p. 18, §D.4 · p. 19

E. Time Series Analysis Tools

• A deterministic analysis module injects structured statistical summaries into the model context alongside raw time-series input to support numerical reasoning.It processes raw series X.Source: E. Time Series Analysis Tools · p. 19
• The analysis pipeline has five components: statistical profiling, trend detection, extremum identification, change point detection, and series comparison.Source: E. Time Series Analysis Tools · p. 19

E.1. Data Preprocessing

• Before analysis, input series are sanitized by removing non-numeric artifacts such as formatting commas and converting values to a floating-point array.Source: §E.1 · p. 19

E.2. Descriptive Statistics

• Descriptive statistics include mean, standard deviation, range, median, variance, skewness, and kurtosis to summarize scale, shape, asymmetry, and tailedness.Skewness and kurtosis are computed using scipy.stats.Source: §E.2 · p. 19

E.3. Trend Analysis; E.4. Peak and Valley Detection

• OLS trend analysis uses slope, R2, and Pearson-correlation thresholds to characterize trend direction, variance explained, and strength, while peak and valley detection filters local extrema by prominence.Trend strength is strong for |r| > 0.7, moderate for 0.4 < |r| ≤ 0.7, and weak otherwise; the prominence threshold is dynamic and based on series variability.Source: §E.3 · p. 19, §E.4 · p. 19

E.5. Change Point Detection; E.6. Multivariate Comparison

• Change-point detection flags large first differences as potential shocks or regime changes, while multivariate comparison uses correlation, cross-correlation lag, and Welch’s t-test.The comparison tests linear association, identifies the lag of maximum similarity, and tests whether series means differ significantly.Source: §E.5 · p. 19, §E.6 · p. 20

F.1. Multivariate Coverage across Tasks

• 58.1% of TSRBench samples are multivariate, covering reasoning over multi-channel as well as univariate signals.Here, “multivariate” means more than one time-series channel is presented jointly to the model.Source: Table 9 · p. 20, §F.1 · p. 20, §F.3 · p. 20

F.2. Fine-grained Correlation Results

• Model size positively correlates with LLM and VLM performance on most perception, reasoning, and decision-making tasks, but not prediction tasks.The passage reports this task-level pattern without providing correlation values.Source: Table 10 · p. 20, §F.2 · p. 20

F.3. MCQ Format Faithfully Reflects Numerical Forecasting Ability

• Open-ended numerical forecasting and multiple-choice forecasting show highly consistent model-size correlation trends, supporting MCQ accuracy as a reflection of open-ended forecasting quality.The open-ended evaluation asks models to generate predicted values directly and uses nMAE.Source: Table 8 · p. 20, §F.3 · p. 20

G.1. Question Cases; Temporal Relation Reasoning; — TASK —

• Question cases include anomaly reconstruction and accelerometer-based activity identification, chronological ordering of events, and choosing a missing basketball event from surrounding game context.The examples test reconstruction, classification, temporal ordering, and inference about an omitted event.Source: §G.1 · pp. 21–22, Temporal Relation Reasoning · pp. 22–24, — TASK — · p. 24

Deductive Reasoning

• Deductive Reasoning examples span Lorenz-system trajectory reconstruction, sunspot and DJIA forecasting, sensor-causality inference, rain classification, and ECG management.The examples pose tasks over a chaotic physical system, historical or event-conditioned series, sensor networks, weather indicators, and normalized ECG signals.Source: Deductive Reasoning · pp. 24–28

Perception Error

• The correct answer was Bollinger Bands, but the model selected MACD as the strategy with the best Maximum Drawdown.The model’s rationale favored MACD for its ability to identify potential trend changes.Source: Perception Error · p. 29
• The model characterized the series as relatively steady and not highly volatile, then generalized that momentum would best control drawdowns.The error analysis says this misread the regime implied by the ground truth.Source: Perception Error · p. 29
• Bollinger Bands’ best-MDD result is consistent with an oscillatory, mean-reverting series in which prices often reach extremes around the 20-day mean.The error analysis links that regime to Bollinger Bands’ mean-reversion strategy.Source: Perception Error · p. 29
• In that regime, Bollinger entries near local troughs and exits near local peaks limit time in the market and cap peak-to-trough equity declines.The error analysis connects this mechanism to a lower MDD.Source: Perception Error · p. 29
• The model’s concern about false signals addressed return stability rather than the mechanics of Maximum Drawdown.That reasoning did not explain why MACD would minimize peak-to-trough losses.Source: Perception Error · p. 29
• MACD can remain exposed during adverse swings until a crossover, allowing larger equity drawdowns than the mean-reversion strategy.This exposure helps explain why MACD was not the ground-truth best-MDD choice.Source: Perception Error · p. 29
• The failure to ground its conclusion in the series’ features led the model to choose MACD over the mean-reversion strategy that better minimizes MDD.Source: Perception Error · p. 29

Reasoning Error

• The task required a precise match across all 120 steps of the Lorenz system, where early values can look similar across candidates.The prompt provides system parameters and initial conditions and emphasizes extreme precision.Source: Reasoning Error · pp. 29–31
• The model showed no mid- or late-step calculations, element-wise candidate comparison, or treatment of rounding and precision alignment.The failure was prematurely concluding from partial computation and neglecting the rigorous comparison needed to distinguish the trajectories.Source: Reasoning Error · pp. 30–31
• Ground truth is option C, not the model’s option A; its claim that A matched the full trajectory was unsupported.The model calculated only the first two Euler steps before asserting that it had simulated the entire sequence.Source: Reasoning Error · pp. 30–31

Domain Knowledge Error

• Using average conditions, the model computed 7.66 mm/day and 191.5 mm over 25 days, then estimated 215–220 mm and chose 216.3 mm.It estimated daily values from the graph and argued that temperatures often exceeded 28°C with high radiation.Source: Domain Knowledge Error · p. 31
• The model systematically overestimated cumulative reference evapotranspiration, giving 216.3 mm versus the 166.4 mm ground truth (about 30% higher); the correct answer was C and the model chose D.Source: Domain Knowledge Error · p. 31
• The model omitted the standard FAO-56 wind-speed correction term, using an incorrect denominator that inflated daily ET0 estimates.The stated standard denominator includes ∆ + γ(1 + 0.34u2); the model instead used ∆ + γ.Source: Domain Knowledge Error · p. 31

Question Understanding Error

• The model selected adjacency matrix D instead of the correct answer A for the causal relationships among five river sensors.The question specified that columns are cause nodes and rows are result nodes.Source: Question Understanding Error · p. 32
• The decisive error was reversing matrix orientation: D[2,3]=1 means node 3 causes node 2, not that row 2 causes row 3.The model treated rows as causes despite the prompt defining columns as causes and rows as results.Source: Question Understanding Error · p. 32
• The model also inferred causality from larger magnitudes and spikes, a weak heuristic that reinforced its selection of D.It reasoned that larger rivers might influence lower, more consistent series.Source: Question Understanding Error · p. 32