PaperlayerView exact revision on arXiv

Source-linked AI summary

How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

Yanlin Fei, Nazhou Liu, Xinmiao Yu, Shaolong Chen, Lei Li, Rahul Thapa, Madalina Ciobanu, Qingqing Mao, Ritankar Das

arXiv:2608.14905v1cs.CL

TL;DR

Existing evaluations reveal little about how autonomous research agents operate or where they break down. This paper introduces AutoResearchEval and ARFT to diagnose failures across the full research lifecycle, finding that current agents lack a metacognitive loop, with the same patterns recurring across eight harness–model combinations.

  • Problem

    Existing evaluations assess isolated skills or end-to-end performance but reveal little about how autonomous research agents operate and where they break down.

  • Method

    The paper evaluates eight harness–model combinations on 100 frontier-science tasks spanning seven domains and the full research lifecycle, producing 800 artifact-aware trajectories and organizing failures into ARFT.

  • Results

    Failure patterns across the research lifecycle converge on a missing metacognitive loop, and the same patterns recur across all eight harness–model combinations.

  • Takeaways & Limitations

    AutoResearchEval provides empirical evidence of how research agents fail, while ARFT supplies a vocabulary to diagnose, attribute, and ultimately mitigate those failures.

  • Takeaways & Limitations

    The released artifacts are intended to let the community examine the paper's limitations and extend ARFT rather than resolve those limitations themselves.

Abstract

from arXiv · show

AI has long assisted scientific research, but the rapid advance of LLMs and agentic scaffolds is reshaping the landscape; a single system can now carry whole-stage research from an initial hypothesis all the way to final published paper, which is a paradigm now referred to as AutoResearch. Existing evaluations reveal little about how these agents operate or where they break down. Tasks are narrowly-scoped, evaluation measures performance but not process, and failure diagnoses lack systematic coverage or artifact-level visibility. To address this gap, we introduce AutoResearchEval, featuring 100 tasks grounded in published frontier science across 7 scientific domains and the full research lifecycle, including ideation, retrieval, execution, analysis, writing, and review. Evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories, with process-level annotation. We organize these insights into AutoResearch Failure Taxonomy or ARFT, a framework of 45 empirically-grounded failure patterns. To enable scalable fine-grained attribution, we leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts. Failure patterns converge on a single overarching limitation, namely that current agents lack a metacognitive loop, which entails the ability to check what they produced against what they found, revise when it does not hold up, and question whether the path they took was sound. The same patterns recur across all 8 harness-model combinations, including the strongest models tested, locating the deficit at the model level rather than in any particular scaffold; whether orchestration-level interventions can close it is an open question this work does not test. We publicly release AutoResearchEval and ARFT to facilitate continued research and development in autonomous scientific discovery.

1. Introduction

Existing evaluations reveal little about how autonomous research agents operate or where they fail. AutoResearchEval and ARFT address this gap by evaluating full research trajectories and organizing their failures, revealing a shared metacognitive limitation.

  • Motivation: Existing evaluations are narrowly scoped, endpoint-focused, and insufficiently visible into artifacts, obscuring how agents operate and where they break down.They often measure reference matches or reproduced results without explaining why, where, or how a trajectory failed.
  • AutoResearchEval: AutoResearchEval evaluates 800 trajectories from eight harness–model combinations on 100 tasks spanning seven domains and the full scientific workflow.The workflow includes ideation, retrieval and synthesis, execution, analysis, writing, and review.
  • AutoResearchEval: The evaluation uses artifact-aware, process-level annotations produced by a human-calibrated judge that reads complete trajectories, including logs, data, code, and reports.This enables attribution beyond the final answer alone.
  • ARFT: ARFT organizes 45 empirically grounded failure patterns by lifecycle stage and root cause to diagnose and attribute agent failures.It provides a structured vocabulary for analyzing failures in practice.
  • Core finding: Failure patterns across the research lifecycle converge on a missing metacognitive loop: agents do not reliably check outputs, revise unsupported results, or question their path.The same patterns recur across all eight harness–model combinations, locating the deficit at the model level rather than in a particular scaffold.

2. AutoResearch Tasks and Trajectories

AutoResearchEval converts published papers into end-to-end discovery tasks whose outcomes are withheld, then analyzes the complete autonomous rollouts and their produced artifacts.

  • Task construction: Each published paper is converted into a task stating the prior scientific context and unresolved issue while withholding the published outcome.The task provides no prescribed method, leaving multiple research paths admissible.
  • Rollouts: Each task runs as a single autonomous rollout in a sandbox with code execution, while the complete trajectory and produced artifacts are logged for analysis.The artifacts are the unit of analysis alongside the trajectory.

Appendix F.

The evaluation constructs realistic autonomous research tasks, runs them across harness–model combinations, and analyzes complete trajectories and artifacts. Human-developed taxonomy and calibrated agent judging expose recurring failure patterns culminating in a metacognitive-loop deficit.

  • Task construction: 100 tasks are built from published papers and divided into open-ended discovery and target-anchored optimization regimes.Open-ended discovery tasks number 70, while target-anchored optimization tasks number 30 and provide an explicit objective.
  • End-to-end rollout: Each rollout receives only a query and fresh sandbox, then proceeds through six stages from ideation and retrieval to review.The complete interaction trace, generated code and data, intermediate outputs, and final report are logged.
  • Evaluation design: Eight harness–model combinations are evaluated by running every combination on every task, producing 800 trajectories and 73k tool calls.The average episode contains 92.3 steps, and the complete trajectory—not only the final report—is analyzed.
  • Failure taxonomy: ARFT is induced from expert examination of complete trajectories and organizes 45 empirically grounded failure patterns by lifecycle stage and root cause.Experts inspect execution logs, code, reports, and data files before iteratively grouping and refining observed behaviors.
  • Scalable annotation: The artifact-aware Agent-as-a-Judge outperforms a transcript-only judge, with a +17.2 recall gain at the pattern level.The judge accesses code, logs, reports, and data, while human calibration reaches κ = 0.85 on 50 sampled trajectories.
  • Systemic finding: Failure patterns recur across stages because current agents lack a metacognitive loop for checking outputs, judging validity, and replanning.The same underlying deficit appears in ideation, execution, and review patterns; whether orchestration alone compensates remains untested.

5. Empirical Analysis

Failures span the research lifecycle but converge on a missing metacognitive loop: agents often fail to compare outputs with evidence, act on detected flaws, or question whether their methods are sound.

  • 92.1% of all failure hits come from the cognitive pillars Grounding & Faithfulness, Scientific Integrity & Alignment, and Cognitive Depth & Adaptability.Engineering Robustness contributes 7.9%, with its highest-ranked pattern placing 26th of 45.
  • Metacognitive deficit: The shared failure mechanism is a broken metacognitive loop: agents do not reliably check outputs against evidence, act on flaws, or question whether their research path was legitimate.R1 concerns evidence-output mismatches, R2 concerns failing to act on identified flaws, and R3 concerns failing to question methodological legitimacy.
  • Grounding & Faithfulness: R1 failures arise when reports describe work the code did not perform or conclusions unsupported by the run artifacts.Method–conclusion disconnect occurs in 77.5% of analyses, implementation discrepancy in 72.1%, and report–code traceability gaps in 60.5%.
  • Cross-system implications: Unsupported claims remain similarly frequent across systems, whereas invented evidence is much rarer in the strongest systems, separating evidence-checking from fabrication control.The paper argues that unsupported claims could be reduced by requiring comparison between the report and run directory, while recognizing limits depends more on the model.
  • Self-verification failures: 82.5% of analyses exhibit uncorrected self-awareness, where agents identify critical problems during self-review but report unrevised conclusions.Failure to gate critical flaws and unremediated adversarial evidence also rank among the five most frequent patterns.
  • Scientific Integrity & Alignment: R3 failures produce plausible-looking outputs through overclaiming, circular validation, and metric misalignment without distinguishing legitimate solutions from gamed ones.The leading patterns occur in 78.1%, 69.0%, and 68.1% of analyses, respectively.

6. Limitations

The study’s limitations constrain the scope and interpretation of its taxonomy, frequency statistics, and evaluation evidence, while the released artifacts support community examination and extension.

  • ARFT’s 45 patterns are grounded in eight harness–model combinations and cannot exhaust every failure or scientific research activity.The task suite and annotated-agent coverage limit the taxonomy’s empirical scope.
  • Failure-frequency statistics are indicative rather than exhaustive because annotating full trajectories limits the number of agents and repeated runs.
  • Fixed wall-clock and token budgets may contribute to premature termination and shallow search, but incidence against remaining budget is not reported.
  • Data contamination cannot be fully excluded despite de-identification and temporally held-out sources.
  • Pattern-level frequencies inherit an unquantified share of judge error because validation is reported only in aggregate over 50 calibration trajectories.
  • The full release of tasks, trajectories, annotations, and judging protocols enables examination of these limitations and extension of ARFT.

C. Execution & Implementation

The execution and implementation taxonomy covers shortcut reliance, implementation mismatches, execution and infrastructure faults, search failures, and premature termination.

  • Circular validation and shortcut reliance evaluate models on their own synthetic outputs or use unintended shortcuts.
  • Grader-fitting and data leakage overfit evaluation benchmarks or leak test data.
  • Implementation discrepancy occurs when code fundamentally differs from the methodology claimed in the proposal.
  • Execution faults and numerical instability include unhandled code errors, numerical overflows, and unseeded randomness causing unreproducible results.
  • Infrastructure error misdiagnosis interprets system, path, or dependency errors as underlying algorithmic failures.
  • Search-space local optimization over-tweaks minor hyper-parameters instead of broadening the solution space.
  • Premature termination gives up or raises exceptions at the first sign of execution friction.
  • Environment interaction failure mishandles CLI outputs, API protocols, or file-system modifications.

D. Analysis & Interpretation

The analysis organizes observed failures across lifecycle stages and root causes, with recurring patterns showing that agents often fail to connect evidence, implementation, conclusions, and review. Case studies illustrate failures ranging from unsupported claims and ignored counterevidence to shortcut validation, persistence, and engineering breakdowns.

  • Integrity failures: Unsupported claims include method–conclusion disconnects, result hallucination, report–code traceability gaps, selective narratives, and fabricated citations or procedures.Examples include conclusions unsupported by methods, methods absent from code, and claims lacking support from any trajectory run.
  • Review failures: Review failures include superficial self-review, missed fatal flaws, absent adversarial perspectives, uncorrected severe problems, score hacking, and hallucinated criticisms.These patterns describe failures both to detect critical issues and to act on issues that review identifies.
  • Cross-stage failures: Cross-stage failures include cascading error propagation, goal drift, skeptical-reasoning deficits, honest-but-hollow outputs, teleological reasoning, and right-for-the-wrong-reason outcomes.Additional cross-stage patterns include cognitive anchoring with re-planning failure and engineering delivery failure.
  • Failure taxonomy: 45 failure patterns are cross-classified by scientific-process stage and four root causes: Grounding, Depth, Integrity, and Engineering.Thirty-seven patterns are stage-localized, while eight cross-stage patterns form layer X.
  • Execution cases: Case studies show that agents can execute reproducible experiments yet answer the wrong question or produce redundant, low-insight findings.One trajectory pursued a self-chosen physics frame after its crossover hypothesis failed, while another re-derived a textbook drift–diffusion result without computing the gold observable.

A.6 Hypothesis-Experiment Mismatch

The cases show hypothesis–experiment mismatches in which agents operationalize a different question, observable, or evidential test than the one posed. These mismatches can produce concrete outputs or even favorable scores without testing the intended hypothesis.

  • Observable mismatch: A binary-classification pipeline cannot test a hazard-ratio question because it generates no time-to-event data and uses classification metrics instead of a survival model.The agent’s pipeline contains no follow-up-time dimension or Cox-type model.
  • Retrieved method mismatch: A literature-retrieval task used a proxy share measure and Chow structural-break analysis instead of the retrieved RCA index and time-varying difference-in-differences design.The directional conclusion matched the target and received reward 1.0 despite never computing the retrieved method.

B.3 Unvetted Data Quality & Units

The cases demonstrate that unverified data quality, units, baselines, retrieval coverage, and validation procedures can distort conclusions even when the surrounding pipeline appears competent. Several failures arise because agents do not check whether inputs and evaluation procedures support the claimed result.

  • Units and baselines: A simulated zinc baseline of 0.46 pM was 20–200× below the cited 10–100 pM physiological range, inflating reported fold changes by approximately 20–200×.Using a properly scaled baseline would reduce the reported surge to approximately 6–60×.
  • Evidence quality: The agents also used inaccessible or mismatched evidence, including a paywalled source without its methods and literature anchors that supported temperature claims rather than composition claims.These failures constrain whether retrieved material can substantiate the intended scientific conclusion.
  • Retrieval quality: A 8729-work corpus contained only 3979 records with abstracts, 41 fetch failures, and substantial contamination, making the reported 4% inclusion rate a denominator artifact.Clustering showed near-zero silhouette across the k-sweep, while a high-signal target review was among only a few successful fetches.
  • Shortcut validation: Hard-coding the Lorenz-96 parameter F=8 reproduced the generator and yielded RMSE-f: 0.0 against a 0.21 SOTA anchor without learning dynamics from observations.The judge marked the result invalid because the predictive behavior came from an injected prior.
  • Grader fitting: Sweeping 128 evaluation-service calls to maximize hidden-test accuracy selected thresholds on the held-out evaluator rather than on a held-out split.The process violates evaluation hygiene even though the judge marked it valid.

C.3 Implementation Discrepancy

Implementation discrepancies arise when reported methods, claimed observables, runtime behavior, or experimental outputs diverge from what the code actually executes. The cases include proxy implementations, unstable simulations, incomplete runs, and failures to interpret diagnostic evidence correctly.

  • Method substitution: A claimed second-order self-force implementation used a hand-coded O(ν) post-Newtonian polynomial rather than numerically computed second-order self-force data.The reported 1GSF–2GSF dephasing therefore came from the proxy coefficients rather than the claimed theory calculation.
  • Numerical behavior: A positive-feedback ODE grew to approximately 10^11 over 365 days, and ad-hoc thresholds were applied to the resulting uncalibrated values.The reported dormant and stromal quantities were read from an exploded trajectory rather than a bounded biological steady state.
  • Engineering diagnosis: A slow transformer run was attributed to broken models and replaced with weaker self-trained models, producing aggregate_improvement ≈ -0.068.The case records a sub-SOTA outcome despite the judge marking the trajectory valid.
  • Delivery failure: Some trajectories stopped before research execution, leaving no experiment code, an empty decision, or a killed background task.One run ended after four turns with reward 0.0, while another failed to poll a forecasting task to completion.
  • Interpretation failure: A degenerate tauFisher output collapsed all predictions to approximately 11.5 hours at chance accuracy, yet the conclusion elevated the failure into a mechanistic claim.The pipeline’s own breakdown was not treated as the primary explanation.

D.4 Method-Conclusion Disconnect

A keyword co-occurrence statistic is stretched into a substantive conclusion about research integration, despite measuring only term co-occurrence under an independence null.

  • D.4 Method-Conclusion Disconnect: 1.75× fewer documents than expected under independence becomes evidence for a substantive claim about three pillars being researched in isolation.The statistic compares 29 observed documents with approximately 50.7 expected, yielding z = −3.39.
  • D.4 Method-Conclusion Disconnect: The agent treats a narrow corpus-level co-occurrence deficit as quantitative confirmation of a broader conceptual research gap.The reported reward was 0.8194 and conclusion_match was 0.9, despite the inferential leap.

D.5 Baseline & Ablation Deficit

A model comparison is invalidated when the baseline and ablation receive unequal feature access, allowing engineered autoregressive information to drive the apparent advantage.

  • D.5 Baseline & Ablation Deficit: 5.95% MAPE versus 48.79% MAPE is reported as an 8.2× advantage for LightGBM over a two-layer LSTM.The headline conclusion says deep sequence models are unnecessary.
  • D.5 Baseline & Ablation Deficit: The LightGBM receives target lags and rolling means that are explicitly withheld from the LSTM, making the comparison confounded.The agent’s SHAP analysis identifies the one-hour autoregressive lag as the dominant signal.

D.6 Result Hallucination

Result hallucination arises when agents fabricate inputs, contradict executed artifacts, or derive unsupported conclusions from computations that do not measure the claimed result.

  • D.6 Result Hallucination: Fabricated study means are pooled into Hedges’ g effect sizes and confidence intervals, giving unmeasured inputs spurious statistical precision.Reported examples include g = +0.29 with a 95% CI of −0.07 to +0.94 and g = −2.55 with p = 0.047.
  • D.6 Result Hallucination: The agent’s own review identifies that a constructed experiment may have predetermined the apparent absence of cross-modal fusion gains.The concern is that the saturating tanh transformation caps the marginal variance contributed by the text factor.
  • D.6 Result Hallucination: 1191.4 pJ appears for both unpruned and 50% pruned MLP-32 in decision.json, contradicting the report’s narrative-only estimate of approximately 596 pJ.The report’s 3–60× energy claim reflects an arithmetic FLOP ratio rather than a measured result.
  • D.6 Result Hallucination: Artifact disagreement prevents tracing the report’s headline claim back to executed computation, independently of the unavailable judge score.The traceability defect is established directly from the conflicting artifacts.

E.3 Omission of Critical Limitations

Agents may list routine limitations while omitting caveats that directly invalidate their findings, including tautological rankings and answers to the wrong evaluation target.

  • E.3 Omission of Critical Limitations: The agent omits that its adoption ranking is determined by hand-set β weights rather than estimated microdata.Its bootstrap only jitters the same β values by ±40%, preserving the by-construction ranking.
  • E.3 Omission of Critical Limitations: The deliverable silently substitutes a logistic adoption exercise for the gold RMSE prediction task, so it supports no claim about SME adoption.The report’s listed limitations omit this observable substitution and the tautological ranking mechanism.

E.4 Methodological & Citation Fabrication

The evaluated agents repeatedly fabricate or overstate methodological support, while self-review and grading often fail to detect contradictions between reports, code, data, and source claims.

  • Fabricated validation: An independent 30/14-item textual cross-validation was claimed despite code containing only roughly 10–11 hand-scored constants, and an LLM judge awarded reward 1.0 across dimensions.The report also overclaimed reading the sources in full.
  • Uncaught calculation errors: The self-review endorsed a power-analysis block that double-counted the 2×3 design factor, inflating reported power to 23% instead of roughly 8–14%.The review identified a secondary weakness rather than catching the central calculation error.
  • Citation and provenance failure: One agent imported a source paper’s hypothesis and thesis wholesale, while review conceded only a downstream causal gap.The trajectory received reward 1.0 on all dimensions.
  • Circular validation: A 0.00 kcal/mol control was exactly zero by construction, making the headline contrast partly circular, while single-pose energies varied by roughly 9 kcal/mol.The review also flagged false precision.
  • Self-validation: A self-computed AUC of approximately 0.71 depended on a classifier generated from the agent’s own model with a hand-set feedback parameter.The review acknowledged that the free parameter could make the model appear to validate itself.
  • Unresolved alternative explanations: A negative transfer-learning result of −47.4% F1 versus a 0.945 baseline was supported by SHAP, yet review still identified synthetic-data quality as an alternative explanation.The review treated the result as potentially artifactual rather than definitive.

X.1 Cascading Error Propagation

Small implementation, planning, and validation errors propagate into scientific conclusions, while agents often fail to pivot, verify, or distinguish delivery failures from scientific quality.

  • Numerical propagation: A longitude–latitude normalization bug changed the reported Aurora z500 5-day RMSE from approximately 26.9 m to 37.9 m while leaving scale-invariant ACC and qualitative comparisons intact.The one-character oversight propagated into the task’s required observable without triggering self-review.
  • Research drift: A simulation produced a genuine finding, but the agent abandoned it after criticizing its metric and drifted into an unfinished replacement simulation.The final logged turn says the second simulation was too slow.
  • Failure to challenge contradiction: A result contradicting the source paper’s successful detection was treated as a conclusion rather than a red flag about the agent’s implementation.The model reported a 3σ LAMNE detection limit of 122% degradation.
  • Parameter construction: An agent-produced result was mechanically determined by hand-set configuration parameters, so the reported comparison did not identify empirically estimated effects.The review explicitly notes that the defining parameters were chosen by the agent.
  • Teleological fitting: Hard-coding known experimental temperatures into an optimization loss guaranteed near-perfect recovery, which was then reported as a discovered mechanism.The recovered values matched the experimental ordering to roughly 0.1 K, and a lenient judge awarded conclusion_match 0.9.
  • Right-for-the-wrong-reason: Reproducing a paper’s exact +28.8% delta resulted from label leakage in the input representation rather than genuine noise-robust learning.The idealized input re-injected the generative class information, making classification near-trivial.
  • Re-planning failure: After an image-ingestion dead end, the agent repeatedly retried the same tool action until the API rejected an oversized payload instead of switching to text-only extraction.A single re-plan could have salvaged the task.
  • Diagnostic attribution: The judge assigns each issue to lifecycle stage and root-cause coordinates, using full-artifact evidence rather than final reports alone.The axes are grounding, depth, integrity, and robustness alongside stages A–F and X.

G. AutoResearchEval Full Task List

The full task list catalogs frontier-science research tensions across multiple domains, with provenance recorded by venue and year and tasks organized into open-ended and target-anchored subsets.

  • Open-ended discovery tasks: Table 22 lists 70 open-ended discovery tasks, describing each scientific tension together with its domain, venue, and year.The listed topics span biology, chemistry, physics, medicine, geophysics, materials science, and scientific computing.
  • Provenance: The task list includes provenance fields for source-paper venue and year, with some entries retaining an unrecorded provenance marker.The release uses n.r. where source-paper provenance is not recorded.
  • Domain breadth: The task list covers problems including cell communication, molecular mechanisms, waveform modeling, dementia diagnosis, reservoir characterization, and anomaly detection.These examples illustrate the breadth of scientific domains and research questions represented.
  • Task variety: The listed tasks also include method-development and benchmarking questions, such as multimodal omics integration, molecule generation, flood forecasting, and clinical risk-score correction.Each entry is framed as a scientific objective or unresolved tension rather than a single benchmark label.
Loading 2608.14905v1…