PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows

Aayam Bansal, Keertan Balaji

arXiv:2609.09203v1cs.AI

TL;DR

Existing AI-scientist benchmarks discard reasoning traces, limiting process-level auditing and failure analysis. OpenDiscoveryTrace addresses this with a public 558-trajectory dataset containing structured nine-field steps across scientific tasks and models. Its pilot shows comparable frontier success rates but markedly different error frequencies and types.

  • Problem

    Existing benchmarks evaluate only final outputs and discard the reasoning process needed to distinguish outcome luck, audit decisions, and diagnose trajectory-level failures.

  • Method

    OpenDiscoveryTrace captures 558 scientific-agent trajectories using nine fields per step and defines five process-level benchmark tasks across seven models.

  • Results

    Comparable frontier success rates coexist with Claude producing 30× more errors than GPT-5.4, with tool misuse dominating Claude’s errors and reasoning errors dominating GPT-5.4’s.

  • Takeaways & Limitations

    The traces support process-aware evaluation, early failure intervention, error-type-specific improvement, reward modeling, governance auditing, and reproducible open-weight benchmarking.

  • Takeaways & Limitations

    Where provider terms restrict verbatim reasoning, the release uses abstracted summaries rather than full traces.

Abstract

from arXiv · show

Existing benchmarks for autonomous AI scientists evaluate only final outputs---generated code, hypotheses, or papers---yet discard the reasoning process by which those outputs were obtained. This makes it impossible to audit scientific methodology, diagnose failure modes, or distinguish systematic reasoning from fortunate guessing. We present \textbf{OpenDiscoveryTrace}, a public dataset of 558 complete AI scientific agent trajectories that captures how models reason, not just what they produce. Each trajectory records a structured 9-field-per-step trace---including thoughts, tool calls, observations, errors, revision triggers, and self-reported confidence---as models execute 124 scientific tasks spanning drug discovery, materials science, genomics, and scientific literature analysis. The dataset covers seven models: three frontier models (GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro; 124 trajectories each, fully balanced across domains and difficulty levels) and four open-weight models (Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B; 30 each), plus 60 live-retrieval variant trajectories. Pilot analysis on 363 LLM-judged trajectories reveals that process traces expose behavioral differences invisible to output-only evaluation: all three frontier models achieve comparable success rates (84--89%), yet Claude Opus 4.6 produces 30$\times$ more errors than GPT-5.4 (2.5 vs. 0.08 per trajectory, $p < 0.0001$, Cliff's $δ= 0.613$), with qualitatively different error profiles---66.7% tool misuse for Claude versus 83.6% reasoning errors for GPT-5.4. We define five benchmark tasks with baselines from logistic regression, random forests, LSTMs, and Transformer models. The dataset, trace schema, agent harness, and benchmark definitions are publicly available under CC BY 4.0 to support research on process-level evaluation, scientific agent auditing, and AI governance.

1. Scientific Bottleneck

Existing AI-scientist benchmarks judge only final outputs, discarding the reasoning process. OpenDiscoveryTrace addresses this gap with structured process traces for auditing and failure analysis.

  • Existing benchmarks evaluate generated code, hypotheses, or rubric scores while discarding the reasoning process.
  • Output-only evaluation conflates reasoning quality with outcome luck and cannot expose trajectory-level failure modes.
  • OpenDiscoveryTrace provides the first public dataset of structured AI scientific process traces.

2. Proposed Dataset

OpenDiscoveryTrace is a 558-trajectory corpus capturing structured, stepwise scientific-agent behavior across models, domains, and task difficulties. Its process-level analyses reveal error differences that comparable success rates conceal.

  • 558 trajectories cover 7 models, 4 scientific domains, 3 difficulty levels, and 60 live-retrieval variants.The collection includes frontier, open-weight, tool-scaffolded, and live-retrieval trajectories.
  • Each step records 9 fields, including thoughts, tool interactions, observations, errors, revision triggers, confidence, identifiers, and timestamps.The dataset defines five benchmark tasks: outcome prediction, error localization, claim verification, autonomy classification, and process quality scoring.
  • 88.6%, 89.3%, and 83.9% are the success rates for GPT-5.4, Gemini, and Claude, respectively, despite large error-rate differences.The comparison uses LLM-as-judge evaluation on 363 frontier trajectories.
  • 30× more errors occur for Claude Opus 4.6 than GPT-5.4: 2.5 versus 0.08 per trajectory.The difference is reported with H=122, p<10^-4, and Cliff’s δ=0.613.
  • 66.7% of Claude’s errors are tool misuse, whereas 83.6% of GPT-5.4’s are reasoning errors.The difference remains robust across all four domains, while output-only benchmarks miss these process-level distinctions.

3. Acquisition Roadmap

The acquisition roadmap expands the operational dataset, adds expert validation and live retrieval, and supports sustainable community updates. Its phases target broader coverage, measured annotation agreement, and hybrid computational-physical traces.

  • Phase 1 (current): Phase 1 operationalizes 558 trajectories across 7 models, 124 tasks, 4 domains, 60 live-retrieval variants, and 6 tool-scaffolded trajectories.The reported total cost is approximately $340.
  • Phase 2 (months 1–2): Phase 2 targets 1,800+ trajectories across 9 models by completing 200 designed tasks and adding Llama 3.1-8B and Gemma-2-9B.The target enables domain-stratified and difficulty-stratified cross-architecture analyses.
  • Phase 3 (months 2–3): Phase 3 will have three domain experts score 120 trajectories across correctness, reasoning, tool efficiency, autonomy, and error localization.The target is Krippendorff’s α ≥0.67; the LLM pilot reports α=0.63–0.82 across axes.
  • Phase 4 (months 3–6): Phase 4 integrates PubMed and PubChem retrieval and pilots electronic-lab-notebook capture for hybrid computational-physical settings.A 30-task live-retrieval pilot is already included.
  • Sustainability: Semver versioning, open hosting, community contributions, and annual refreshes support dataset sustainability.The current schema is version 1.0, with the dataset on Hugging Face and code on GitHub.

4. Metadata, Governance & Safety

The release specifies governance, privacy, licensing, formats, and leakage-controlled evaluation procedures. It provides full traces where permitted and abstracted summaries when provider terms restrict verbatim reasoning.

  • The dataset is CC-BY-4.0 licensed and distributed as self-contained JSON with per-step and aggregate metadata.A datasheet accompanies the release.
  • Release tiers provide full traces where provider terms permit and abstracted summaries preserving >90% analytical utility otherwise.Abstracted summaries retain phase transitions, tool calls, and errors without verbatim reasoning.
  • The release contains no personally identifiable information and uses public scientific knowledge only.
  • An 80/20 domain-stratified train/test split is designed without leakage, with evaluation following HELM.

5. Acceleration Potential

OPENDISCOVERYTRACE supports process-aware evaluation and scientific-agent auditing by pairing structured workflow traces with final-output assessment. Its benchmark framing exposes differences in model behavior that output-only metrics can miss.

  • Acceleration Potential: Six capabilities connect process traces to evaluation, intervention, improvement, reward modeling, governance, and open-weight benchmarking.The dataset is presented as supporting process-aware evaluation, early failure intervention, error-type-specific improvement, process-aware reward modeling, trustworthy auditing, and reproducible benchmarking.
  • Trace Schema: Each trajectory stores structured step-level fields covering reasoning, actions, observations, errors, revisions, confidence, identifiers, and timing.The schema extends ReAct with nine fields, while the complete JSON schema also records tool inputs and outputs, verification, recovery, and aggregate trajectory metadata.
  • Benchmark Comparison: OPENDISCOVERYTRACE is distinguished from prior resources by combining structured scientific process traces, error logging, revision tracking, and multi-phase workflow annotation.Existing benchmarks mainly evaluate final artifacts, while trajectory datasets such as ToolBench and MINT focus on web or software domains.
  • Inter-Model Comparison: Output metrics show no significant differences, whereas process metrics reveal highly significant differences with large effect sizes.The process comparison includes errors, revisions, and wall time, showing why final-output evaluation alone can rate models as interchangeable.

E. Failure Taxonomy

The failure taxonomy separates explicit tool failures from broader error events and distinguishes tool misuse, reasoning errors, and hallucinations. These categories expose qualitatively different model behaviors, while automated classification remains a limitation.

  • Metric Definitions: Two error metrics differ: trajectory metadata counts explicit harness-detected failures, whereas the taxonomy also includes keyword-detected reasoning errors.Consequently, taxonomy totals are higher than per-step failure counts.
  • Error Types: Tool misuse includes incorrect API calls, wrong parameters, malformed tool inputs, and misinterpreted tool outputs.Examples include passing a gene name to PubChem or calling python_exec with syntactically invalid code.
  • Error Types: Reasoning errors include logical fallacies, incorrect inferences, flawed calculations, and unsupported conclusions detected through revision triggers.The triggers include terms such as “mistake,” “reconsider,” and “let me correct.”
  • Error Types: Hallucination covers fabricated data, citations, or tool outputs and occurred once across 688 total errors.The paper characterizes hallucination as rare in the observed data.
  • Model Profiles: Models differ in both error frequency and error type: Claude predominantly exhibits tool misuse, whereas GPT-5.4 more often exhibits reasoning errors.The distinction is presented as relevant to model improvement because the two profiles suggest different intervention targets.
  • Limitation: Automated keyword heuristics may undercount implicit reasoning errors that do not trigger explicit revision keywords.The authors propose validation against human-labeled error types on an annotated subset.

F. Matched-Task Within-Domain Comparisons

Matched-task comparisons show that process-level behavior differs significantly across frontier models even when task coverage controls for domain and difficulty. Errors diverge across domains, concentrate on harder tasks, and remain difficult to use for predicting final success.

  • Matched-task controls: p<0.0001 error-rate differences persist in every domain, while step counts remain non-significant in every domain.The comparisons use 31 shared tasks per domain, with all three frontier models executing every task.
  • Difficulty effects: 4.14 errors on medium tasks versus 0.27 on easy tasks for Claude, showing that process divergence concentrates on challenging problems.Easy tasks produce uniformly short, low-error trajectories across models.
  • Outcome prediction: Outcome prediction remains difficult: no method exceeds the majority baseline accuracy of 0.693, and sequential models do not outperform aggregate-feature methods.The evaluated trajectory features include steps, tool calls, errors, revisions, tools, phases, response length, and error-type count.
  • Error localization: 68.1% of first errors occur at step 0, compared with a ∼7% random baseline.The step-0 heuristic matches 68.1% accuracy, whereas always predicting the last step achieves 0%.
  • Caveat: The step-0 concentration is partly attributable to harness design because the first substantive action is typically a tool call.Future work should vary tool-initialization ordering to separate harness effects from generalizable behavior.

H.3. Task 3: Claim Verification

The claim-verification analysis evaluates frontier-model process quality using matched comparisons and a composite score. Process quality differs significantly across models despite comparable success rates, while the expanded evaluation broadens task coverage but still requires human anchoring.

  • Evaluation: n=363 frontier trajectories are evaluated with Wilson score 95% confidence intervals using cross-model LLM-as-judge assessment.The expanded evaluation includes medium and hard tasks and tightens confidence intervals relative to the n=127 ground-truth-only evaluation.
  • Evaluation: 3.6% of trajectories were not judged because malformed claims could not be parsed, and human expert evaluation remains a Phase 3 priority.The limitation affects the expanded LLM-judged evaluation rather than the dataset’s trajectory collection.
  • Process quality scoring: The composite score averages four normalized components: efficiency, tool diversity, low errors, and conclusion reached.The components are equally weighted as a deliberate baseline.
  • Process quality scoring: H=149.9, p<0.0001: composite process quality scores differ significantly across models, with Claude scoring lower because of its higher error rate.This difference is reported despite Claude matching the other models on success rates.

I. LLM-Based Inter-Annotator Agreement

The annotation pilot evaluates trajectories on four axes and finds initial evidence that the autonomy taxonomy is reliably distinguishable, while weighted agreement exceeds 0.85 across all axes. The pilot also uses LLM annotators as a proxy pending definitive human-expert agreement measurements.

  • 60 trajectories were annotated in a stratified sample with 5 trajectories per model per domain.
  • Weighted agreement exceeds 0.85 across all axes, measured as agreement within one point on the 5-point scales.
  • The L1–L4 autonomy taxonomy exceeds the α≥0.67 target, providing initial evidence that its levels are reliably distinguishable.
  • LLM-based annotation remains a proxy for human annotation, with a planned three-annotator study of 120 trajectories for definitive IAA measurements.

K. Open-Weight Model Details

The open-weight component provides reproducible trajectories from four models and a smaller tool-scaffolded Qwen2.5-7B comparison under the multi-step harness. These materials test schema compatibility across model interfaces while exposing process differences under matched conditions, with harness choices recognized as a source of trajectory sensitivity.

  • Open-weight coverage: 120 single-response trajectories cover four open-weight models, each running the same 30 tasks from the task bank.
  • Open-weight coverage: Six tool-scaffolded Qwen2.5-7B trajectories average 10.3 steps, 1.7 tool calls, 1.7 errors per trajectory, and a 100% conclusion rate.
  • Open-weight coverage: Under identical conditions, Qwen2.5-7B has step counts comparable to frontier models but higher error rates, enabling process-level open-versus-frontier comparison.
  • Task bank: The task bank contains 200 tasks spanning four domains, with examples covering drug discovery, materials science, genomics, and literature analysis.
  • Harness design: The harness standardizes a six-phase workflow, four tools, a 30-step limit, deterministic settings, checkpointing, concurrency controls, and logged error recovery.
  • Harness design: The six-phase prompt, 30-step cap, and tool suite affect trajectory characteristics, motivating systematic variation of harness parameters.

N. Code and Data Availability

The dataset, code, task bank, harness, and analysis pipelines are publicly available under CC-BY-4.0. The release supports reproducible regeneration of reported figures, statistics, and benchmark baselines, with open-weight replication requiring no proprietary API keys.

  • Code, data, and the complete task bank are publicly available under CC-BY-4.0.
  • The Hugging Face Hub hosts trajectory JSON files, the task bank, and analysis results, while GitHub hosts the harness, analysis pipelines, benchmark baselines, and paper source.
  • The repository’s analysis pipelines reproduce all figures and statistics from the released trajectory data.
  • Frontier-trajectory reproduction requires API keys for the three frontier model providers.
  • Open-weight reproduction requires only a V100 16GB-or-better GPU and model weights, enabling independent replication without proprietary API keys.
Loading 2609.09203v1…