PaperlayerView exact revision on arXiv

Source-linked AI summary

TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution

Tommaso Bendinelli, Artur Dox, Christian Holz

arXiv:2608.27182v1cs.LG

TL;DR

TraceBench addresses the lack of controlled evidence about LLM agents performing time-series root-cause attribution with system knowledge. It generates tasks from interpretable mechanical simulators and evaluates agents across controlled conditions, finding benefits from domain context, reliance on console output, and weaker performance for programmatic submissions. The benchmark’s conclusions are limited by its closed-ended, low-dimensional setting, small repetition count, and scaffold-specific model evaluations.

  • Problem

    Existing evaluations do not systematically test whether LLM agents combine temporal and physical reasoning over multivariate time series with process-specific knowledge.

  • Method

    TraceBench generates controlled attribution tasks from three interpretable mechanical systems and evaluates four agents across domain context, noise, labeled examples, and submission mode.

  • Results

    Agents primarily inspect numerical console output; performance generally decreases with higher noise and reusable Python submissions, while domain context helps and labeled support sets show no reliable improvement.

  • Takeaways & Limitations

    TraceBench provides controlled evidence about how agents use mechanistic context and tools when attributing causes in dynamical-system time series.

  • Takeaways & Limitations

    TraceBench uses closed-ended labels, low-dimensional regularly sampled fully observed mechanical trajectories, few repetitions, and different scaffolds across models.

Abstract

from arXiv · show

LLM agents are increasingly applied to anomaly detection and root-cause analysis in time-series observations collected from real-world systems; however, their performance on these tasks has not been systematically evaluated under controlled conditions. We introduce TraceBench, a simulation-based framework for generating controlled root-cause attribution tasks. In each generated task, an agent receives time-series observations produced by simulating a physical dynamical system and must determine whether a system parameter was altered during the simulation and, if so, which one. Using TraceBench, we generate tasks from three interpretable mechanical systems and systematically evaluate four LLM agents across controlled experimental conditions, yielding new insights into how these agents analyze time-series observations from dynamical systems. Our results show that agents benefit substantially from domain context and explore data primarily through numerical console output rather than visualizations. We also find that agents generally perform worse when required to produce a Python script that maps each time-series sample to a predicted root-cause label than when they submit predictions directly. We release our datasets, agent trajectories, experimental results, and a leaderboard on our website, tracebench.github.io.

1 Introduction

TraceBench addresses the lack of controlled evaluation for agents combining multivariate temporal and physical reasoning with system-specific knowledge. It introduces a simulation-based benchmark that varies task conditions and reveals how agents analyze and submit root-cause predictions.

  • Existing root-cause benchmarks do not systematically isolate temporal and physical reasoning over multivariate time series with process-specific knowledge.
  • Existing language–time-series datasets usually pair observations with questions, aligned text, visualizations, or visible-pattern descriptions rather than mechanistic system descriptions.
  • TraceBench evaluates four LLM agents across controlled changes in noise, domain context, labeled examples, and submission mode using three interpretable mechanical systems.
  • The benchmark tests whether agents can infer parameter changes from time-series observations by combining system descriptions, temporal evidence, and physical dynamics.
  • Agents rely primarily on console output rather than visualizations and perform worse with Python-script submissions than with direct predictions.

2 Related Work

TraceBench extends prior agentic and language–time-series benchmarks by isolating root-cause attribution in controlled physical simulations. Its sample-independent mechanistic descriptions test whether agents can formulate and test hypotheses about parameter changes.

  • Agentic benchmarks for time-series analysis and diagnosis: Prior agentic benchmarks evaluate multi-step time-series analysis, diagnosis, executable artifacts, and operational workflows across diverse applications.
  • Agentic benchmarks for time-series analysis and diagnosis: TraceBench differs by isolating root-cause attribution from multivariate observations with intervention-defined ground truth in physical simulations.
  • Agentic benchmarks for time-series analysis and diagnosis: TraceBench independently varies domain context, observation noise, labeled examples, and submission mode to analyze agent behavior systematically.
  • LLM benchmarks and datasets for time-series reasoning with textual information: Most language–time-series resources use sample-tied questions, annotations, or visible-pattern descriptions rather than sample-independent mechanistic system information.
  • LLM benchmarks and datasets for time-series reasoning with textual information: The benchmark tests whether agents use prior system descriptions to formulate hypotheses about candidate parameter effects and test them against observed trajectories.

3 TraceBench Framework

TraceBench constructs closed-set root-cause tasks from simulated dynamical systems, varying context, noise, examples, and submission mode. Its simulator interface models clean trajectories, noisy observations, interventions, no-intervention samples, and ambiguity filtering.

  • Controllable task axes: Each task selects one setting along four axes: domain context, observation noise, labeled examples, and submission mode.
  • Root-cause task definition: A task contains instructions, an unlabeled test batch, and optionally a labeled support batch of multivariate time-series samples with ground-truth classes.
  • Root-cause task definition: Direct-answer mode returns one prediction per test sample, whereas programmatic mode submits one reusable Python script applied independently across the batch.
  • Physical simulator interface: The simulator generates a noise-free trajectory from parameters and an initial state, then samples noisy observations from an observation model controlled by κ.
  • Physical simulator interface: Intervention samples use a single instantaneous step change in one parameter at an intervention time, while no-intervention samples keep all parameters fixed.
  • Physical simulator interface: A practical filtering procedure rejects intervention trajectories insufficiently distinguishable from no-intervention or static-change probes using an effect-to-noise threshold ρ = 2.

4 Experimental Setup

TraceBench evaluates agents on controlled tasks from three physical simulators, varying noise, context, examples, and submission mode while measuring accuracy and exploration behavior.

  • Physical simulators: The benchmark uses BallDrop, BounceBall, and MassSlide, each modeling an interpretable mechanical system with simulator-specific observations and interventions.BallDrop models vertical bouncing, BounceBall a damped mass on a tilted track, and MassSlide a mass on a periodically forced inclined plane.
  • Physical simulators: Two noise regimes scale simulator-specific noise components: κ_low = κ_ref and κ_high = 4κ_ref.The observation model combines heteroscedastic Gaussian noise, temporally correlated drift, and transition-localized noise.
  • Metrics: Accuracy is measured on visible episode batches for direct submissions, while programmatic submissions are additionally evaluated on held-out samples to measure generalization.The evaluation also records token usage, cost, time, tool calls, Python activity, plots, numeric console output, and console-token share.
  • Evaluated agents: Each evaluated configuration combines a language model, terminal-based scaffold, and inference settings, with all agents sharing submission modes, file layout, and scoring procedure.The evaluated configurations pair gpt-5.5, gemini-3.1-pro, claude-opus-4.6, and minimax-m2.7 with their respective agent environments.
  • Experimental conditions: The main experiment crosses low or high noise with direct-answer or programmatic submission, using three labeled support samples per class and detailed system descriptions.Matched conditions use the same test samples, enabling controlled comparisons across task-axis settings.
  • Ablations: The experiments remove domain context or labeled examples in separate ablations, with those ablations evaluated only using gpt-5.5 and gemini-3.1-pro.Removing domain context also removes process descriptions, column names, and class names.

5 Results

Across TraceBench’s controlled evaluations, performance varies with agent, noise, submission mode, resources, and available domain context. Agents primarily use numerical console output, while programmatic submission and missing domain context reduce accuracy.

  • Main results: gpt-5.5 achieves the highest mean accuracy in all six reported accuracy columns, while minimax-m2.7 averages approximately 46.0 percentage points lower.
  • Main results: 11.0, 17.7, and 19.3 percentage points: increasing noise reduces observed mean episode accuracy for gpt-5.5, claude-opus-4.6, and gemini-3.1-pro, respectively.The decrease is averaged across the two submission modes for each closed-weight configuration.
  • Main results: Held-out accuracy is lower than episode accuracy for all three closed-weight agents under programmatic submission, indicating limited stability of submitted physical signatures.Programmatic submission requires a reusable per-sample function and removes opportunities for sample-specific judgment.
  • Exploration and interaction metrics: Fewer than one plot per episode: gpt-5.5 inspects plots at most this often in direct-answer conditions and none in programmatic conditions.Plot inspection is rare among configurations supporting image inspection; minimax-m2.7 does not support image inspection in the evaluated configuration.
  • Exploration and interaction metrics: 77–80%: console output accounts for this share of gpt-5.5’s cumulative input context, compared with approximately 29–55% for other configurations.Across the closed-weight configurations and main conditions, more than 4,600 numeric values are exposed through console output per episode on average.
  • Ablations: 20.0–32.0 percentage points: removing domain context reduces average accuracy, whereas removing labeled examples increases accuracy by 4.2 points for gpt-5.5 and 6.6 points for gemini-3.1-pro.Removing domain context increases total token usage in three of four cases, while removing examples consistently reduces token usage and substantially lowers cost.

6 Conclusion

TraceBench provides a controlled framework for evaluating LLM agents on time-series root-cause attribution across interpretable mechanical systems. The experiments show how agents explore data, how performance changes with noise and submission mode, and where the benchmark’s scope limits interpretation.

  • Agents primarily inspect numerical console outputs to guide analysis and final root-cause predictions.
  • Agents generally perform worse as observation noise increases and when required to translate analyses into reusable programs.Held-out results also indicate that submitted scripts have limited out-of-sample generalization.
  • TraceBench uses a closed-ended label set, limiting how fully it captures open-ended root-cause analysis.The benchmark’s scope also includes regularly sampled, fully observed trajectories from low-dimensional mechanical systems with at most one instantaneous parameter change.
  • TraceBench is a controlled diagnostic testbed rather than a realistic replica of complex industrial root-cause analysis.Its results should therefore be interpreted as evidence of controlled temporal and mechanistic reasoning, not readiness for complex industrial deployment.

C.1 Data Generation and Sample Selection

TraceBench constructs candidate intervention tasks from simulator-specific configurations and filters them to reduce ambiguity and leakage. Test samples and support examples are separated across initial configurations, and datasets are frozen before evaluation.

  • Candidates vary intervention class, intervention time, and post-intervention parameter value before passing through a documented rejection pipeline.
  • Initial configurations are retained only when at least two distinct root-cause interventions pass filtering, with no-intervention samples generated from them.
  • Samples within an episode use distinct initial configurations and baseline parameter values to reduce bias from shared starting conditions.
  • The held-out test set contains 142 samples per class, and all datasets and sample assignments were frozen before agent evaluation.

D.2 Accuracy versus Evaluation Cost

Figures 8–10 compare episode top-1 accuracy with mean evaluation cost for each simulator under the four main-experiment conditions. The comparisons summarize accuracy–resource trade-offs across model configurations.

  • Figures 8–10 plot episode top-1 accuracy against mean evaluation cost for BallDrop, BounceBall, and MassSlide.
  • Each plotted point represents one model’s mean episode accuracy and mean evaluation cost across five repetition seeds.Horizontal and vertical error bars show the corresponding sample standard deviations.

E Classical Time-Series Baselines

The benchmark compares agents with six learned classical time-series baselines and uniform random guessing on matched direct-answer episodes. The baselines include ordinary and privileged representations using nearest-neighbor, dynamic-time-warping, and MiniRocket methods.

  • Six learned classical baselines and uniform random guessing are evaluated on matched visible-test direct-answer episodes under low and high observation noise.
  • Uniform-random-guessing accuracy is 0.200 for BallDrop, 0.167 for BounceBall, 0.200 for MassSlide, and 0.189 in aggregate.
  • Each learned baseline fits three labeled examples per class and predicts the same ten test trajectories for each simulator and seed.
  • The baselines pair ordinary and privileged Euclidean 1-NN, multivariate DTW 1-NN, and MiniRocket with ridge classification.Privileged inputs subtract the clean paired no-intervention trajectory from the observed intervention trajectory.

G Metric definitions and computation

This section defines trajectory, tool-use, visualization, console-output, token, cost, and timing metrics used to characterize agent episodes.

  • Agent iterations count LLM calls in one trajectory, including text, reasoning, and zero or more tool calls.
  • Tool calls count all well-formed tool calls, while Python calls count parsed interpreter invocations in tool commands.Displaying Python source without executing it does not count as a Python call.
  • Python statements count abstract-syntax-tree statement nodes across distinct Python source versions first displayed in the trajectory.
  • Plots generated count distinct image-like files created, whereas plots inspected count generated images subsequently read or viewed successfully.Each generated file contributes once, regardless of how often it is later opened.
  • Numeric values printed count numeric literals exposed through tool output, excluding source code, metadata, identifiers, warnings, tracebacks, and embedded image payloads.
  • Console output token share estimates the percentage of cumulative input-context tokens attributable to console output.
  • Total tokens combine prompt and completion tokens, while total time measures active agent-system wall-clock time including inference and tool execution.
  • Evaluation cost is reported in U.S. dollars from prompt, cached-prompt, and completion-token totals using configured per-million-token prices.The per-agent input, cached-input, and output prices are reported in Table 21.

H Paired Statistical Analysis and Uncertainty

The paired analysis compares agents and conditions within matched simulator–seed settings, while the task design controls labels, observations, and submission requirements.

  • Table 23 reports paired episode-level differences in percentage points with confidence intervals obtained by exhaustive sign-flip inversion.
  • Agent-comparison differences are calculated within each simulator–seed pair and averaged across low- and high-noise conditions.
  • Condition-comparison differences are averaged across three closed-weight agents, excluding minimax-m2.7, with W/T/L determined from paired differences.
  • Each sample has either no intervention or one allowed parameter change, and any change is a single instantaneous step at an unknown time.
  • Direct-answer tasks return ranked labels in results.json, whereas programmatic tasks require rule.py to define predict(df) for held-out samples.
  • The primary evaluation metric is top-1 accuracy, while shortlist score assigns 1/m when the correct label appears in a returned list of length m.Unnecessary extra labels reduce the secondary shortlist score.

J Per-simulator results tables

The per-simulator results are organized into four tables crossing noise level with direct-answer and programmatic submission modes.

  • Table 24 reports per-simulator results under low noise with direct-answer submission.
  • Table 25 reports per-simulator results under low noise with programmatic submission.
  • Table 26 reports per-simulator results under high noise with direct-answer submission.
  • Table 27 reports per-simulator results under high noise with programmatic submission.

J.1 Ablation Results

The ablation results compare performance without labeled examples and without domain context, alongside a representative low-noise direct-answer trajectory.

  • Ablation Results: Table 28 reports per-simulator results without labeled examples under high noise with direct-answer submission.
  • Ablation Results: Table 29 reports per-simulator results without labeled examples under high noise with programmatic submission.
  • Ablation Results: Table 30 reports per-simulator results without domain context under high noise with direct-answer submission.
  • Ablation Results: Table 31 reports per-simulator results without domain context under high noise with programmatic submission.
  • Representative Trajectory: A representative agent trajectory is summarized for the main low-noise direct-answer condition, with the complete trajectory provided in an ancillary PDF.
Loading 2608.27182v1…