Source-linked AI summary
TimeSage-EV: A Live Benchmark for Agentic Time Series Analysis in Evolving Environments
Qingren Yao, Yaxuan Kong, Yuqi Nie, Yichen Li, Stefan Zohren, Anna Vettoruzzo, Qingsong Wen, Ming Jin, Joaquin Vanschoren
TL;DR
Existing time series QA benchmarks largely use fixed snapshots, leaving recurring, cutoff-aware analysis in evolving environments insufficiently evaluated. TimeSage-EV addresses this gap with a live benchmark, finding persistent gaps in agents’ temporal validity, memory management, and adaptation.
Problem
Existing benchmarks largely use fixed time series QA pairs and are not designed to evaluate recurring, cutoff-aware analytical workflows.
Method
TimeSage-EV evaluates agents on 1,485 scenario-period questions using pre-cutoff data and documents while withholding target releases for verification.
Results
Agents show persistent gaps in memory management, long-horizon reasoning, and temporal validity, with GPT-5.4 achieving the strongest overall benchmark score of 86.0.
Takeaways & Limitations
TimeSage-EV provides a diagnostic benchmark for developing agents that reliably update time series analyses as evidence evolves.
Takeaways & Limitations
The benchmark excludes enterprise data, poorly archived releases, non-English-dominant sources, and visual interpretation of charts and graphs.
Abstract
from arXiv · showhide
Time series analysis in high-stakes domains relies on recurring data releases, where new observations can alter the evidence base and the validity of later conclusions. Existing time series QA benchmarks mostly rely on fixed snapshots, leaving temporal validity and cutoff-aware evidence use unevaluated. We introduce TimeSage-EV, a live benchmark for agentic time series analysis in evolving environments. It tracks 60 real institutional scenarios across 6 domains, comprising 1,485 scenario-period QA pairs from Feb 2023 to May 2026 and spanning monthly, weekly, daily, and irregular release cadences. At each period, large language model (LLM) agents receive time series data and source reports, while the withheld target release provides ground truth. TimeSage-EV evaluates state identification, data summarization, and outlook reasoning. Experiments with frontier LLM agents and TimeSage-1.0, a novel self-evolving agent with a reusable analytical skill library, reveal significant performance gaps across model tiers and recurring failures in temporal validity, exogenous context use, and adaptation. We release TimeSage-EV as a research resource with monthly updates, code, a leaderboard, and failure-mode analyses.
1 Introduction
TimeSage-EV frames time series analysis as a recurring, cutoff-aware workflow over evolving data and reports, addressing limitations of fixed-snapshot benchmarks. It provides a live benchmark spanning institutional scenarios, structured questions, multi-axis evaluation, and agent failure analysis.
- Motivation: Time series analysis in high-stakes domains requires recurring data ingestion, historical updating, multimodal evidence synthesis, and timestamp-aware information cutoffs.Each new release shifts the available evidence distribution, making temporal validity central to the workflow.
- Benchmark gap: Existing time series QA benchmarks usually evaluate fixed QA pairs over predefined tasks rather than evolving analytical workflows.This leaves dynamic evidence use and recurring updates only partially covered.
- Evaluation setting: Agents receive pre-cutoff data and documents while the target release is withheld, and questions use a 27-tag taxonomy grouped into six families.The families are state identification, change detection, composition analysis, event detection, attribution, and outlook.
- TimeSage-EV: 60 scenarios across 6 domains run from February 2023 to May 2026, with 1,485 scenario-period QA pairs covering recurring institutional releases.Domains include agriculture, climate, energy, finance, healthcare, and transportation, organized into three difficulty tiers.
- Evaluation and findings: Four evaluation axes—keypoint accuracy, keypoint coverage, report faithfulness, and report quality—expose gaps in cutoff validity, memory management, temporal reasoning, and report synthesis.Experiments cover frontier LLM agents and TimeSage-1.0, a self-evolving agent with a reusable skill library.
2 Related Work
Prior work connects numerical time series with language instructions and domain context, while live benchmarks reduce contamination through refreshed tasks or evaluate forecasts after outcomes resolve. TimeSage-EV addresses the remaining gap: continuous agent evaluation for recurring time series analysis with cumulative numerical-documentary evidence.
- Language-grounded time series analysis: Language-grounded time series research reframes analysis as connecting numerical patterns with instructions and domain context for language models.Representative systems include LLMTime, Time-LLM, ChatTime, ChatTS, Time-MQA, TSRBench, and TimeSeriesExam.
- Live benchmarks: LiveBench and LiveCodeBench reduce contamination through refreshed task pools, while ForecastBench evaluates predictions after outcomes resolve.These benchmarks provide live or outcome-aware evaluation, but not recurring evolving scenarios with cumulative numerical-documentary evidence.
- Positioning TimeSage-EV: Existing settings remain refreshed snapshots rather than evolving scenarios requiring recurring releases and cumulative numerical-documentary evidence.The paper positions TimeSage-EV at the intersection of language-grounded analysis and live, continuous agent evaluation.
3 Methodology
TimeSage-EV models agentic time series analysis as recurring, cutoff-aware evaluation over expanding evidence states. Its benchmark pipeline constructs and audits reproducible period instances, then scores structured answers and reports under independent and sequential protocols.
- Evolving task formulation: Each period exposes cumulative time series data and strictly prior documents, with the target-period release withheld for ground-truth construction and evidence verification.External web evidence is empty by default and enabled only for scenarios supporting cutoff-valid retrieval.
- Evolving task formulation: Across successive periods, the task specification remains fixed while evidence expands, requiring agents to revise judgments without violating the temporal cutoff.Previous target releases become prior evidence, while the next release remains withheld, creating continuously updated evaluation periods.
- Benchmark construction: The benchmark pipeline collects recurring public institutional releases, generates scenario-level period instances, and applies quality control enforcing temporal isolation.Every evaluation period is reproducible from the underlying releases.
- Benchmark construction: Evidence packages combine accumulated time series and documents, using deterministic rules for time-series fields and transcribed textual evidence for document-dependent fields.Rules include threshold classification, multivariable comparison, and cross-period comparison; withheld target-period documents are used only for answer construction and verification.
- Evaluation protocol: Agents produce a structured answer and free-form report scored for keypoint accuracy, keypoint coverage, report faithfulness, and report quality.The protocol includes rule-based scoring and evaluates both independent fresh-start runs and sequential runs carrying state with only prior-period evidence.
4 Experiments
Experiments show substantial performance differences across LLM agents, with accuracy declining on analytically demanding, attribution, and forecast tasks. Persistent memory and reusable skills can improve adaptation and efficiency, while reasoning and temporal-misuse errors remain major failure modes.
- Overall performance: GPT-5.4 leads overall performance at 86.0, followed by Qwen-3.5-397B at 84.2 and Sonnet-4.6 at 82.4.All models exceed 92 on easy tasks, but the best model reaches only 72.7 on hard tasks; Kimi-K2.6 and Gemma-4-31B score 59.3 and 53.1.
- Overall performance: High-cost models do not necessarily lead in accuracy, suggesting that reasoning efficiency drives long-horizon time series performance.
- Live leaderboard: GPT-5.4 maintains the highest and most stable live score after the initial three-period ramp-up, while Gemma-4-31B and Kimi-K2.6 decline steadily.The leaderboard tracks each model as in-context information accumulates from t −23 to the cutoff time.
- Per-question-type analysis: Attribution and forecast remain the hardest question families, with best scores of 46.7 and 43.0, respectively.State identification, change detection, and composition are comparatively tractable, with strongest models reaching above 85–95.
- Failure-mode analysis: Reasoning overrun, instruction drift, and temporal errors are prominent failures caused by inefficient reasoning, long-context rule forgetting, and weak alignment between task semantics, data, and evidence.Temporal misuse includes misreading rows or timestamps, miscalculating variables or windows, and relying on misaligned evidence.
- Memory and self-evolving skills: Sequential memory benefits Qwen-3.5-397B, while TimeSage-1.0’s reusable skills improve performance in medium- and hard-difficulty scenarios and reduce token cost to 0.82×.For Gemma-4-31B, sequential evaluation helps early performance but its advantage narrows; skill invocation peaks in hard scenarios.
5 Conclusion
TimeSage-EV is a live benchmark for agentic time series analysis in evolving environments, formalizing recurring, cutoff-valid analytical updates across institutional scenarios. Experiments reveal persistent gaps in agents’ memory management, long-horizon reasoning, and temporal validity, positioning the benchmark as a diagnostic resource for reliable updating.
- Benchmark contribution: TimeSage-EV formalizes recurring, cutoff-valid analytical updates for agentic time series analysis in evolving environments.It evaluates agents across institutional scenarios and scenario-period QA pairs.
- Benchmark contribution: 60 institutional scenarios, 1,485 scenario-period QA pairs, and 4 complementary scoring axes define the benchmark’s evaluation scope.
- Findings: Experiments with frontier LLM agents and TimeSage-1.0 reveal persistent gaps in memory management, long-horizon reasoning, and temporal validity.
- Implication: TimeSage-EV serves as a diagnostic benchmark for developing robust analytical agents that reliably update time series analyses as evidence continually evolves.
Limitations
TimeSage-EV is limited by its reliance on public, predominantly English-language institutional sources and by incomplete coverage of data formats, conventions, and workflows. Its evaluation may retain judge noise and harness-specific effects, while experiments use a May 2026 snapshot alongside a planned frozen version for comparability.
- Data coverage: Public institutional sources exclude enterprise data, dashboards, and settings where historical releases are unavailable or poorly archived.The benchmark also cannot exhaust the diversity of document formats, institutional conventions, or analytical workflows.
- Data coverage: Predominantly English-language reports from established global institutions constrain the dataset’s source coverage.
- Evaluation protocol: Rule-based structured-field scoring and LLM-based judging may retain residual judge noise and bias despite structured rubrics and evidence grounding.The experimental baseline’s reliance on smolagent also means instruction drift or reasoning overrun may be partially entangled with the harness.
- Versioning: May 2026 is the experimental cutoff, with a planned frozen version containing all 1,485 evaluated scenario-period instances for direct baseline comparability.The live branch is intended to support testing on newly published institutional releases and measurement of out-of-distribution generalization.
Intended Use
TimeSage-EV is an academic research benchmark for diagnosing LLM agents on recurring time series analysis. Its scores and outputs are diagnostic decision-support signals requiring human verification, not guarantees of reliability or substitutes for professional judgment.
- TimeSage-EV is intended to diagnose LLM agents on recurring time series analysis in academic research.
- Scores indicate analytical behavior in covered scenarios, not agent reliability on unseen sources, domains, or release schedules.
- Agent outputs should support decisions rather than replace professional judgment, because fluent claims may lack support or violate temporal cutoffs.Human verification is required before downstream use, especially in high-stakes domains.
Ethical Considerations
TimeSage-EV follows ethical research standards by using publicly available institutional releases under their usage terms and excluding personally identifiable information. Its agent analyses are intended to support, not replace, human experts.
- The benchmark uses publicly available institutional releases under each source’s usage terms and contains only aggregate time series and reports without PII.Human reviewers checked ground-truth correctness using public, non-sensitive content.
- Agent analyses are designed as decision-support tools for human experts rather than replacements for professional judgment.
A Appendix · A.1 Task Rationale
TimeSage-EV evaluates recurring institutional analysis as an evolving, cutoff-aware task rather than standard forecasting, static QA, or generic agent evaluation. It tests whether agents can interpret new releases, use valid evidence, preserve analytical state, and produce supported reports as information changes.
- A.1 Task Rationale: Recurring institutional analysis keeps the task specification fixed while the evidence available to the analyst changes after each release.This distinguishes the benchmark from standard time series forecasting, static time series QA, and generic agent benchmarks.
- A.1 Task Rationale: Agents must interpret the latest observation, compare it with historical context, use cutoff-valid documents, and revise reports as the environment evolves.
- A.1 Task Rationale: Temporal isolation tests whether agents reason within the information limit applicable at the target period.Without temporal isolation, agents could use future reports, post-hoc explanations, or documents published after the target release.
- A.1 Task Rationale: TimeSage-EV exposes cumulative time series data and prior-period source documents while withholding the target-period release for ground truth construction.
- A.1 Task Rationale: Real analytical workflows require agents to inspect data files, use tools, consult documents, preserve state across periods, and produce structured answers and natural-language reports.
- A.1 Task Rationale: The integrated workflow evaluation exposes unsupported causal claims, poor evidence use, and incomplete reporting.
A.2 Core Concepts
TimeSage-EV is a live benchmark for agentic time series analysis in which recurring tasks are evaluated as institutional releases extend the time series and alter available evidence. Within each scenario, the task contract remains fixed while agents must update judgments and respect each period’s evidence cutoff.
- Live benchmark: TimeSage-EV’s evaluation set continuously expands as institutional releases are published, automatically extending each scenario’s time series and withholding the target-period release.Each scenario includes a self-contained, rerunnable pipeline that re-pulls the source and updates cumulative prior-period evidence.
- Evolving environment: An evolving environment keeps the domain background, question template, and answer schema fixed while the time series and document set change across periods.Agents must update their judgments while respecting the cutoff point for each period.
- Agentic time series analysis: Agentic time series analysis uses tools to inspect data, consult external evidence, answer analytical questions, and produce task-specific outputs under recurring period-specific evidence cutoffs.In TimeSage-EV, the time series extends and the document set changes as the analytical task recurs across an evolving environment.
A.3 Benchmark Construction
The appendix details the benchmark construction protocol, expanding the overview into a sequential pipeline from data collection and scenario generation through quality control and human annotation. It also documents data sources, volatility statistics, and illustrative scenario examples.
- Construction protocol: The construction protocol proceeds from data collection and scenario generation to quality control and human annotation.Sections A.3.1–A.3.4 present these stages sequentially.
- Data sources and volatility: The appendix documents the benchmark’s data sources and reports volatility statistics.These materials appear in Section A.3.5.
- Scenario examples: Three illustrative scenario examples are presented in Section A.3.6.
A.3.1 Stage 1: Data Collection … A.8 AI Usage
TimeSage-EV is constructed through staged source validation, cutoff-aware scenario generation, automated and human quality control, and multi-axis evaluation. The released corpus spans 60 scenarios with deliberately varying temporal volatility, while experiments show increasing difficulty from Easy to Hard and a diagnostic self-evolving harness for reusable skills.
- A.3.1 Stage 1: Data Collection: The pipeline filters inaccessible or unsuitable sources, assigns Easy, Medium, or Hard tiers, and rejects candidates that fail execution-level checks.The tiers target categorical state identification, temporal-pattern summarization, and report-grounded reasoning about causes or outlooks.
- A.3.2 Stage 2: Scenario Generation: Each evaluation state gives agents the cumulative table and prior-period reports while withholding the target-period release, and ground truth is generated reproducibly without exposing that release.Evidence must satisfy completeness and column-hygiene checks, and target-period material is used only for ground-truth construction and verification.
- A.3.3 Stage 3: Quality Control: Scenarios pass eight automatic grounding gates and human review, with leakage, column-hygiene, or instance-count defects causing blocking or quarantine.Three PhD student reviewers assess clarity, cutoff validity, schema conformance, correctness, grounding, and variation; disagreements are resolved through discussion.
- A.3.5 Released Sources: 60 scenarios span six domains and three tiers, with 20 scenarios per tier, after metadata filtering, execution validation, ground-truth extraction, automatic audits, and human review.The domains are agriculture, climate, energy, finance, healthcare, and transportation.
- A.3.5 Released Sources: Answer transition rates span 0.13 to 0.91 across all 60 scenarios, showing that temporal volatility varies deliberately at the scenario level.The rate measures how often structured answer fields change between consecutive periods.
- A.4 Evaluation Methodology: The atomic evaluation unit is a (scenario, period) instance producing a structured answer and free-form report, scored on Accuracy, Coverage, Faithfulness, and Quality.Keypoint accuracy compares structured values with ground truth; coverage measures required-content completeness, faithfulness measures source-supported claims, and quality measures writing alone.
- A.6.2 Self-Evolving Harness: TimeSage-1.0: TimeSage-1.0 holds the baseline interface, model, task exposure, time budget, and evaluation protocol constant while adding a lightweight self-evolving skill-library manager.Before later-period use, skills must be callable, self-contained, usable, faithfully described, and neither period-specific, duplicative, nor literal answers.
- A.7.1 Breakdown of Main Results: Hard-tier performance declines across models and dimensions, with Overall scores ranging from 72.70 for GPT-5.4 to 53.02 for Gemma-4-31B, driven mainly by Accuracy and Coverage.Most models exceed 90 Overall on Easy; GPT-5.4 and Qwen-3.5-397B are strongest overall, while model gaps widen on Hard.