PaperlayerView exact revision on arXiv

Source-linked AI summary

TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks

Muyan Weng, Defu Cao, Wei Yang, Yashaswi Sharma, Yan Liu

arXiv:2602.13272v1cs.AIcs.LG

TL;DR

It is unclear whether forecasting strength reflects genuine temporal understanding under context and events. TemporalBench tests this through a four-tier, multi-domain benchmark with controlled information access, finding that numerical forecasting accuracy does not reliably translate into contextual or event-aware reasoning.

  • Problem

    Existing evaluations provide limited evidence about whether strong forecasting performance reflects temporal understanding under contextual and event-driven conditions.

  • Method

    TemporalBench evaluates four temporal competencies across four domains using T1–T4 tasks and controlled access to future targets, context, and events.

  • Results

    Strong numerical forecasting performance does not reliably translate into correct contextual or event-conditioned decisions, and agent strengths remain fragmented across settings.

  • Takeaways & Limitations

    TemporalBench supports evaluating temporal competence as multiple complementary capabilities rather than as a single forecasting score.

  • Takeaways & Limitations

    Labels are computed solely from historical and future time-series segments, while contextual descriptions and event narratives condition prompts but do not affect labeling rules.

Abstract

from arXiv · show

It is unclear whether strong forecasting performance reflects genuine temporal understanding or the ability to reason under contextual and event-driven conditions. We introduce TemporalBench, a multi-domain benchmark designed to evaluate temporal reasoning behavior under progressively richer informational settings. TemporalBench adopts a four-tier task taxonomy that examines historical structure interpretation, context-free forecasting, contextual temporal reasoning, and event-conditioned prediction across four real-world domains: retail, healthcare, energy, and physical systems. By controlling access to future targets and contextual information, the benchmark enables a diagnostic analysis of whether models can correctly interpret temporal patterns, align them with external context, and adapt predictions when conditions change. Extensive baseline experiments show that strong numerical forecasting accuracy does not reliably translate into robust contextual or event-aware temporal reasoning; instead, existing agent frameworks exhibit fragmented strengths and systematic failure modes that remain largely hidden under forecasting-only benchmarks. The TemporalBench dataset is publicly available at https://huggingface.co/datasets/Melady/TemporalBench, and we additionally provide a public leaderboard at https://huggingface.co/spaces/Melady/TemporalBench_Leaderboard.

1 Introduction

Real-world time series reflect changing contexts and events, yet common evaluations emphasize numerical forecasting under stable assumptions. TemporalBench addresses this gap by testing whether agents interpret temporal structure, use context, and adapt to events.

  • Real-world time series are shaped by evolving environments, external conditions, interventions, and unexpected events.
  • Existing benchmarks often abstract away contextual variation and event-driven regime changes while emphasizing numerical prediction accuracy.
  • Forecasting benchmarks rarely test whether models align contextual information with temporal segments or adapt predictions when conditions change.
  • Context determines which temporal segments are comparable and which historical changes should generalize to the future.
  • TemporalBench evaluates historical understanding, context-free prediction, contextual reasoning, and event-informed forecasting across four real-world domains.

2 TemporalBench

TemporalBench combines real numerical time series, contextual information, and reasoning-oriented tasks in a four-tier diagnostic benchmark. Its pipeline constructs T1–T4 tasks across four domains to separate temporal competencies under progressively richer information.

  • Benchmark overview: TemporalBench compares benchmarks using whether they include real numerical time series and explicit external context.
  • Benchmark overview: The benchmark spans retail, healthcare, energy, and physical systems, transforming heterogeneous datasets into structured evaluation tasks.
  • Benchmark overview: TemporalBench contains 2,775 evaluation tasks across 191 time-series instances probing structure, prediction, context, and event-informed decisions.
  • Task families: T1 and T2 use numerical signals for historical interpretation and context-free prediction, while T3 and T4 add textual context and event-conditioned reasoning.
  • Task families: T3 localizes contextual reasoning through alignment, slicing, difference judgment, lag, structure, and interaction understanding.
  • Diagnostic role: The benchmark’s diagnostic decomposition separates numerical prediction from temporal interpretation and tests alignment with context under changing conditions.
  • Construction pipeline: Events define historical–future boundaries through event injection or detection, with controlled regime boundaries supporting event-aware evaluation.

3 Experiments

Experiments evaluate agents across T1–T4 to separate numerical forecasting from temporal interpretation, contextual reasoning, and event-aware prediction. Results reveal sharp task-dependent degradation, inconsistent orchestration benefits, and fragmented agent strengths.

  • Experimental design: The experiments analyze agent behavior across T1–T4 rather than collapsing performance into a single aggregate score.
  • Global performance patterns: Performance degrades sharply from structural understanding to contextual reasoning, with T3 accuracy often reaching single-digit or low double-digit percentages in retail and clinical domains.
  • Global performance patterns: Numerical forecasting improvements do not reliably translate into accurate qualitative judgments about future behavior.
  • Agent frameworks versus direct prompting: Agent orchestration does not consistently improve temporal reasoning and can introduce noise when structure must be inferred from numerical sequences.
  • Time-series-specific versus general-purpose agents: Specialized temporal modeling stabilizes numerical prediction but does not resolve abstraction, comparison, or conditional reasoning under context.
  • Overall findings: Across T1–T4, agent frameworks show fragmented strengths, leaving gaps between numerical accuracy, temporal interpretation, and event-aware reasoning.
  • Event-aware reasoning: Event descriptions are not consistently converted into actionable temporal constraints, exposing a gap between contextual awareness and event-conditioned reasoning.

4 Analysis

TemporalBench reveals uneven temporal reasoning across agents and capabilities: structural and interaction tasks are easier, while comparative and alignment-sensitive tasks remain difficult. Explicit temporal representations help selectively, especially in clearer domains, but do not resolve broader reasoning challenges.

  • Temporal reasoning profiles: T3 performance varies systematically across task families, domains, and agent designs rather than exhibiting a uniform floor effect.
  • Capability decomposition: C5 structural pattern recognition and C6 interaction understanding consistently outperform C2 slicing-based comparison and C3 difference judgment across backbone models.C5 and C6 benefit from co-occurring conditions, peak patterns, and regime-level changes, whereas C2 and C3 require reliable segment comparison and relative-change quantification.
  • Agent comparisons: General-purpose agents modestly improve higher-level C5–C6 reasoning but often underperform on C1 alignment and C4 lag detection.Language-driven orchestration helps abstract interpretation without reliably improving fine-grained temporal analysis.
  • Agent comparisons: The time-series-specific agent is more balanced on C1 and C4 but does not substantially improve higher-level comparative or interaction-based reasoning.
  • Overall pattern: Current agents exhibit fragmented reasoning capabilities, with strengths in structural and interaction reasoning coexisting with weaknesses in comparative reasoning.The capability pattern suggests that T3 failures reflect specific reasoning gaps rather than a uniform limitation.
  • Temporal representations: Enhanced feature and visualization representations improve performance heterogeneously, with clearer gains in structured domains and mixed results in noisy real-world domains.The effects occur without additional training, task-specific tuning, or workflow changes, indicating that representation quality is helpful but insufficient.

5 Conclusion

TemporalBench evaluates temporal competence beyond forecasting by combining real-world series, event-centric tasks, and a four-tier taxonomy. Results show that forecasting accuracy does not reliably imply contextual or event-aware reasoning, motivating disentangled evaluation.

  • Benchmark scope: TemporalBench combines real-world time-series data, event-centric task construction, and T1–T4 to assess temporal competence across informational settings.
  • Empirical findings: Strong numerical forecasting performance does not reliably translate into correct decisions under contextual grounding or event-conditioned reasoning.
  • Empirical findings: Performance varies substantially across task families and domains, indicating that temporal competence is not a single unified capability.
  • Empirical findings: Current agentic approaches show fragmented strengths, with no system consistently integrating temporal structure, context, and decision requirements across settings.
  • Implication: The benchmark supports more principled analysis by disentangling complementary temporal competencies across task types, domains, and informational regimes.

B.1 Time-Series Forecasting Benchmarks

Existing time-series forecasting benchmarks primarily emphasize numerical prediction or analysis, supporting standardized comparison but offering limited evaluation of reasoning-oriented temporal competence.

  • Forecasting benchmarks: Large-scale benchmarks such as TFB, GIFT-Eval, and Chronos primarily evaluate forecasting or numerical analysis across diverse domains.
  • Forecasting benchmarks: These benchmarks generally frame evaluation as single-task numerical analysis without explicit reasoning-oriented objectives.
  • LLM–time-series benchmarks: LLM–time-series benchmarks add question answering but typically omit external context and a unified taxonomy for complementary temporal competencies.

B.2 Temporal Reasoning Benchmarks for LLMs

Text-only temporal reasoning benchmarks assess event ordering, duration, and causal-temporal relations, but omit numerical time-series signals. Multimodal efforts begin to connect numerical series with language supervision and downstream reasoning.

  • Text-only benchmarks: Textual temporal reasoning benchmarks probe event ordering, duration, and causal-temporal relations using language-model inputs.
  • Text-only benchmarks: These benchmarks omit real numerical time-series signals, limiting their coverage of temporal reasoning over quantitative data.
  • Multimodal benchmarks: TelecomTS combines real-world telecommunications time-series with language supervision for anomaly detection, root-cause analysis, and question answering.

B.3 Agentic and LLM-Based Methods for Time-Series

Prior work applies LLMs and agents to time-series analysis through staged workflows, direct reasoning, and tool-augmented architectures. General-purpose frameworks add structured coordination and multi-agent collaboration to these approaches.

  • TimeSeriesScientist decomposes univariate forecasting into data inspection, model selection, and result interpretation stages.
  • Direct LLM approaches study numerical forecasting, trend interpretation, and zero-shot reasoning over serialized temporal inputs.
  • TimeCAP contextualizes and semantically augments time-series data, while TS-Agent uses iterative statistical reasoning through tool invocation and evidence accumulation.
  • TimeCopilot presents an open-source agentic framework that automates parts of forecasting workflows.
  • MetaGPT and AgentScope support structured multi-agent reasoning through role-specialized collaboration, orchestration, tool use, and evaluation.

C.1 Error Analysis

The error analysis finds that many agent failures arise from execution and controllability problems rather than temporal reasoning alone. Horizon violations, formatting failures, and numerical instability differ substantially across agent designs.

  • Overall trends: More than 40% of AgentScope’s errors and nearly two-thirds of MetaGPT’s errors involve incorrect forecast horizon lengths.
  • Overall trends: Numerical instability produces formally valid sequences with extremely large MAE or RMSE values through scale drift, unbounded growth, or unsupported regime changes.
  • Comparison across agent types: Despite sharing the same base model, agents exhibit markedly different error profiles, with general-purpose systems often struggling most with low-level execution constraints.
  • Comparison across agent types: TimeSeriesScientist reduces invalid-option errors but shifts failures toward formatting violations and numerical anomalies.
  • Comparison across agent types: The single-LLM baseline has a majority of failures from formatting or horizon mismatches, while scaffolding filters some basic execution errors without resolving deeper issues.
  • Implications for future agent design: Robust agents need explicit horizon guarantees, schema enforcement, and numerical-plausibility checks because systematic failures persist without such safeguards.
  • Overall trends: Output-control failures account for a large fraction of errors, especially across T2 and T4, making validation a central design objective.

C.2 Length Sensitivity Study

The length sensitivity study shows that additional history does not consistently improve agent performance. Effects vary by dataset, task tier, and agent, with intermediate-context sweet spots, saturation, and degradation at longer inputs.

  • Overall trends: History length is not a universally monotonic driver of performance; its effects vary across domains and task tiers.
  • Overall trends: Intermediate input lengths can match or outperform the longest inputs, indicating that effective use of additional tokens is non-trivial.
  • Dataset-dependent sensitivity: PSML is relatively stable and often improves with more context, whereas FreshRetailNet shows weaker and more irregular sensitivity.
  • Dataset-dependent sensitivity: MIMIC shows the clearest length-induced instability, with moderate extensions sometimes helping but very long histories sharply degrading difficult-tier performance.
  • Task-tier effects: T1 benefits most consistently from longer history, while T2 and T4 show less stable gains, diminishing returns, and occasional reversals.
  • Differences across agent types: AgentScope is relatively robust but history-insensitive, the single-LLM baseline often has a moderate-length sweet spot, and TimeSeriesScientist can improve or degrade sharply depending on the setting.
  • Implications for agent design and benchmark use: The findings motivate length-aware agents that select or summarize salient history rather than relying on truncation alone.

E Illustrative Task Examples Across Domains

Illustrative examples apply the same T1–T4 task structure across retail, healthcare, energy, and physical-system data while preserving domain-specific contexts and events. The examples show a unified framework spanning understanding, forecasting, contextual reasoning, and event-conditioned prediction.

  • Unified task structure: Each domain example transforms one time-series sample into four task tiers under a unified abstraction while preserving domain-specific characteristics.
  • PSML: The PSML example uses power-load and meteorological covariates across tasks ranging from time-series understanding to event-conditioned forecasting and decision-making.
  • MIMIC: The MIMIC example uses multivariate physiological signals and conditions T3 and T4 on the clinical event fever.
  • FreshRetailNet: The FreshRetailNet example represents sparse, promotion-driven sales dynamics in which discount events shape T3 and T4.
  • Causal Chambers: The Causal Chambers example uses system measurements and operational variables to study load-regime interventions and causal temporal comparisons.
  • Cross-domain comparison: Across all examples, the same task tiers evaluate time-series understanding, forecasting, and event-driven reasoning within one benchmark framework.

F Performance with different base LLMs

Across backbone models, agent frameworks retain advantages over single LLMs, especially on reasoning-intensive tasks, while forecasting performance depends more strongly on the underlying model. Base-model choice mainly changes the performance ceiling rather than eliminating the benefits of agent-based designs.

  • Multi-choice tasks: Agent frameworks substantially outperform single-LLM baselines on multi-choice tasks T1 and T3 across backbone models.This pattern is consistent across claude-3.7-sonnet, gemini-2.5-flash, deepseek-chat, and qwen-plus.
  • Reasoning-heavy tasks: The accuracy gap between single LLMs and agent frameworks widens as tasks become more reasoning-heavy, with T3 showing the largest performance variance.Single LLMs often collapse to low accuracy on T3 despite high success rates, whereas agent frameworks retain meaningful gains.
  • Forecasting tasks: Forecasting tasks T2 and T4 depend more strongly on the backbone model, with alternative LLMs generally producing lower success rates and higher error metrics than gpt-4o.The largest degradation appears on PSML and Causal Chambers, although agent frameworks remain more robust than single LLMs.
  • Backbone effects: Differences among base LLMs primarily affect absolute performance rather than the relative ranking of agent frameworks.Stronger models more closely resemble gpt-4o, while lighter or compressed models show larger forecasting drops; no base model removes the agent advantage.
  • Overall conclusion: The gpt-4o conclusions generalize across diverse backbone LLMs: agent-based temporal reasoning improves robustness and accuracy, while the backbone mainly sets the performance ceiling.The existence of agent-based gains persists across the tested models even when absolute accuracy is constrained by model capacity.
Loading 2602.13272v1…