Source-linked AI summary
Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers
Hanhua Hong, Yizhi Li, Luu Gia Huy, Jian Yang, Ming Zhou, Chenghua Lin
TL;DR
Reproducing scientific experiments at scale is difficult because publications are increasingly complex while existing evaluations often inspect only final repositories and mainly cover ML. AgentActionBench records agent actions with an MCP-based Action Recorder and evaluates them using paper-specific rubrics across ML and AI4Science. Current systems remain limited, especially in execution, while model-generated rubrics strongly agree with human annotations.
Problem
Manual reproduction verification is increasingly impractical, while existing benchmarks largely focus on final repositories and machine learning papers rather than the full reproduction process.
Method
AgentActionBench combines an MCP-based Action Recorder with fine-grained, paper-specific rubrics to evaluate the complete reproduction workflow across 150 ML and AI4Science papers.
Results
Current systems remain limited, with execution and result verification as primary bottlenecks; model-generated rubrics achieve Pearson 0.93 and Spearman 0.88 correlations with human annotations.
Takeaways & Limitations
Auditable action traces support process-oriented evaluation, while strong agreement with human annotations supports scaling rubric generation across the benchmark.
Takeaways & Limitations
Rubric-based evaluation cannot eliminate hallucinations entirely, the evaluator is limited to GPT-4o-mini, and AI4Science coverage remains preliminary.
Abstract
from arXiv · showhide
Reproducibility is essential to scientific progress, yet the growing volume and complexity of scientific publications make exhaustive manual verification increasingly impractical. Although recent advances in large language model (LLM) agents enable automated experiment reproduction, existing evaluations largely focus on final repositories and are typically limited to machine learning (ML). We introduce AgentActionBench, a process-oriented benchmark for evaluating agent-based experiment reproduction across ML and AI4Science domains. Our framework uses an MCP-based Action Recorder to capture agents' behaviour throughout the reproduction process and evaluates the resulting traces with paper-specific rubrics. AgentActionBench contains 150 papers, including 120 ML papers and 30 AI4Science papers. A human-annotated subset covering 10% of the benchmark provides validation data, while model-assisted augmentation expands the full benchmark to more than 10,000 rubric items. Experimental results show that current systems remain limited, with execution as the primary bottleneck. Meanwhile, the strong Pearson and Spearman correlations between model-generated and human-annotated rubrics validate the reliability of our scalable rubric-generation approach.
1 Introduction
AgentActionBench addresses limitations of output-oriented, ML-focused reproduction evaluation by recording auditable actions and extending rubric-based assessment across ML and AI4Science domains.
- 1 Introduction: Recent LLM advances make agents promising for automated reproduction because the workflow requires paper comprehension, planning, code generation, execution, and result verification.These applications span machine learning experimentation and AI4Science tasks in domains including biology and chemistry.
- 1 Introduction: Output-oriented benchmarks can accept plausible repositories or descriptions without establishing that implementations were grounded in the paper or actually executed.Prior evaluations may overlook paper content, execution, or fabricate execution records, making reproduction evidence ambiguous.
- 1 Introduction: The framework records reading, writing, and command-execution actions in a standardised environment, enabling direct auditing of each reproduction stage.The resulting action logs provide evidence for paper-specific rubric evaluation rather than relying only on final repositories.
- 1 Introduction: The benchmark extends reproduction evaluation beyond ML to AI4Science papers spanning astronomy, biology, chemistry, environmental science, materials science, and medicine.Its full version contains 120 ML and 30 AI4Science papers, with more than 10,000 rubric items generated through an agent-assisted pipeline.
- 1 Introduction: AgentActionBench evaluates the complete reproduction workflow across 150 ML and AI4Science papers using process-oriented, rubric-grounded assessment.The benchmark combines an MCP-based Action Recorder with paper-specific rubrics to assess paper observation, planning, implementation, execution, and result matching.
2 Related Work
Related work studies autonomous agents for scientific workflows, rubric-based evaluation for structured supervision, and LLM-based augmentation for scaling benchmark data.
- 2 Related Work: Autonomous scientific systems support different portions of reproduction, from end-to-end research workflows and closed-loop feedback to executable repository construction and specialised sub-agents.Examples include The AI Scientist, DOLPHIN, PaperCoder, and HiRAS.
- 2 Related Work: Rubrics decompose complex tasks into fine-grained criteria, but human-authored rubric construction is costly and difficult to scale.PaperBench and HealthBench use human-authored rubrics for stepwise evaluation, motivating model-generated alternatives.
- 2 Related Work: LLM-based data augmentation scales supervision through prompt-based, retrieval-based, and progressively diversified instruction-generation methods.Prior work ranges from bootstrapping instruction-response pairs from small seed sets to rewriting instructions into more complex variants.
3 Methodology
The methodology captures agent actions through an MCP-mediated environment and evaluates reproduction with paper-specific, stage-based rubrics across a 150-paper ML and AI4Science benchmark.
- 3.1 Action Recorder: The MCP-based Action Recorder mediates agent interactions and exports chronological logs containing inputs, results, and timestamps for auditable evaluation.Its Read, Write, and Execute tools capture file inspection, artifact creation, command execution, status, standard output, and standard error.
- 3.2 Benchmark Construction: Each paper receives a paper-specific rubric whose items encode a criterion, importance score, and reproduction stage across five rubric types.The categories are Paper Observation, Plan Writing, Code Implementation, Command Execution, and Result Matching.
- 3.2 Benchmark Construction: A consensus-annotated subset of 15 papers provides human validation, while LLM-assisted generation expands the benchmark beyond 10,000 rubric items.The human subset contains 12 ML and 3 AI4Science papers; the full benchmark averages 69.92 items per paper.
- 3.3 Evaluation: For each rubric item, an evaluator selects relevant action-log evidence and assigns Pass or Fail before computing an importance-weighted reproduction score.The evaluator filters logs by rubric type and uses the source paper, criterion, and extracted logs to make each binary judgment.
4 Experimental Setup
The experiments evaluate shared-task systems and a reproduction baseline using rubric-type performance and correlations between human-annotated and model-generated rubrics.
- 4 Experimental Setup: The main leaderboard excludes the human-annotated training subset, while comparing participating systems with the authors’ reproduction baseline.AGENTACTIONBENCH-HUMAN is released for participants to develop and refine their systems.
- 4 Experimental Setup: System performance is analyzed across Paper Observation, Plan Writing, Code Implementation, Command Execution, and Result Matching using percentages of maximum category scores.Table 4 abbreviates these categories as PO, PW, CI, CE, and RM.
- 4 Experimental Setup: Rubric-generation quality is compared through correlations between evaluation scores from human-annotated and model-generated rubrics on AGENTACTIONBENCH-HUMAN.The analysis includes 95% confidence intervals and summary statistics.
5 Result Analysis
Current systems achieve low reproduction scores, with execution and result matching as major weaknesses, while model-generated rubrics strongly agree with human annotations but differ in category emphasis.
- 5.1 Leaderboard: 49.64% is the best overall score, achieved by YNU-HPCC-Task11-AgentRep; no system exceeds 50%, and Codex-GPT-5.4 reaches 24.19%.All systems perform better on AI4Science papers than on ML papers, possibly because the selected papers use more standard methods or simpler configurations.
- 5.1 Leaderboard: Execution remains a primary bottleneck: systems perform relatively well on observation, planning, and implementation but worse on Command Execution and Result Matching.The reported difficulties include runtime failures, environment and dependency issues, and reproducing reported results.
- 5.2 Rubric Generation Analysis: 0.93 Pearson and 0.88 Spearman correlations indicate strong agreement between scores from model-generated and human-annotated rubrics.The correlations have narrow confidence intervals, supporting the rubric-generation pipeline for scaling benchmark construction.
- 5.2 Rubric Generation Analysis: Score distributions are comparable across most rubric types, but generated rubrics are more verbose and allocate more score to Code Implementation than to Plan Writing and Command Execution.The authors identify calibration of rubric granularity and category emphasis as an area for improvement.
6 Conclusion
AGENTACTIONBENCH evaluates agent-based experiment reproduction across ML and AI4Science using recorded actions and fine-grained rubrics.
- 6 Conclusion: The benchmark evaluates the full reproduction process with an MCP-based Action Recorder and paper-specific rubrics across 150 papers.Its construction combines a human-annotated subset with an LLM-assisted rubric-generation pipeline.
- 6 Conclusion: Current agents remain far from reliable reproduction, with execution and result verification emerging as the primary bottlenecks.Strong agreement between model-generated and human-annotated rubrics supports scaling the benchmark construction approach.
Limitations
The benchmark’s rubric-based evaluation reduces hallucinations but does not eliminate them. Its evaluation model and AI4Science coverage also remain limited.
- Rubric-based evaluation substantially reduces hallucinations but cannot eliminate them entirely.
- Using only GPT-4o-mini as evaluator leaves stronger-model reliability unassessed.
- AI4Science coverage remains preliminary and should be extended to broader scientific domains.
A Log Example
Figure 4 provides an example of the logs generated by the Action Recorder.
- Figure 4 shows an example of Action Recorder logs.