PaperlayerView exact revision on arXiv

Source-linked AI summary

SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers

Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, Yulan He

arXiv:2504.00255v2cs.CLcs.AIcs.MAcs.SE

TL;DR

Reproducing algorithms from research papers requires both synthesizing scholarly descriptions and implementing code within repositories, a gap existing benchmarks do not specifically evaluate. The paper introduces a benchmark and dual-agent framework for this task; performance remains challenging, with failures often linked to missing or inconsistent descriptions.

  • Problem

    Existing benchmarks do not specifically assess whether LLMs can reproduce real-world algorithms from peer-reviewed papers, a task requiring synthesis of dispersed algorithm details and repository dependencies.

    Source: §1 · pp. 1–2

  • Method

    The authors introduce SciReplicate-Bench and Sci-Reproducer, a benchmark and dual-agent framework, and evaluate comprehension with reasoning graph accuracy alongside implementation metrics.

    Source: §1 · p. 2

  • Results

    The task remains highly challenging: Claude-Sonnet-3.7 achieved only 0.390 execution accuracy with Sci-Reproducer.

    Source: §5.1 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    Algorithm reproduction failures often stem from incomplete paper descriptions, which Sci-Reproducer effectively addresses.

    Source: §1 · p. 3

    Paper descriptions may omit hyperparameters, numerical-stability techniques, implementation logic, and coding strategies needed for reproduction.

    Source: §5.3.2 · p. 10

Abstract

from arXiv · show

This study evaluates large language models (LLMs) in generating code from algorithm descriptions in recent NLP papers. The task requires two key competencies: (1) algorithm comprehension: synthesizing information from papers and academic literature to understand implementation logic, and (2) coding expertise: identifying dependencies and correctly implementing necessary APIs. To facilitate rigorous evaluation, we introduce SciReplicate-Bench, a benchmark of 100 tasks from 36 NLP papers published in 2024, featuring detailed annotations and comprehensive test cases. Building on SciReplicate-Bench, we propose Sci-Reproducer, a dual-agent framework consisting of a Paper Agent that interprets algorithmic concepts from literature and a Code Agent that retrieves dependencies from repositories and implements solutions. To assess algorithm understanding, we introduce reasoning graph accuracy, which quantifies similarity between generated and reference reasoning graphs derived from code comments and structure. For evaluating implementation quality, we employ execution accuracy, CodeBLEU, and repository dependency/API recall metrics. In our experiments, we evaluate various powerful non-reasoning and reasoning LLMs as foundational models. The best-performing LLM using \ModelName~achieves only 39% execution accuracy, highlighting the benchmark's difficulty. Our analysis identifies missing or inconsistent algorithm descriptions as key barriers to successful reproduction. We make available our benchmark and code at https://github.com/xyzCS/SciReplicate-Bench and project homepage at https://xyzcs.github.io/scireplicate.github.io/.

1 Introduction

The introduction frames reproducing algorithms from papers as a joint challenge in understanding dispersed research details and implementing code across repository dependencies. It presents a benchmark, a dual-agent framework, and measures for assessing both capabilities.

  • Reproducing paper algorithms requires synthesizing details dispersed across papers and references, then resolving interdependent repository files, dependencies, and external APIs.The introduction notes that researchers may also face barriers from limited coding expertise or inaccessible implementations.Source: §1 · pp. 1–2
  • SciReplicate-Bench contains 100 code-reproduction tasks derived from 36 papers published in leading conferences in 2024, and Sci-Reproducer’s Paper and Code Agents collaboratively perform algorithm understanding and code implementation.The recent publication window was deliberately chosen to minimize the risk of data leakage.Source: §1 · p. 2
  • Reasoning graph accuracy evaluates algorithm comprehension, while execution accuracy, CodeBLEU, and dependency/API recall assess implementation correctness.The reasoning graph represents reasoning steps as code-aligned comments linked by data-flow relationships.Source: §1 · p. 2
  • Execution accuracy remains below 40% for all evaluated models, which show stronger algorithmic comprehension than practical implementation and often fail amid incomplete descriptions.The authors also report reasoning models over-relying on internal reasoning instead of available information-extraction tools.Source: §1 · p. 3

2 Related Work

The related work places paper-to-code reproduction between scientific research automation and LLM code generation, distinguishing it from research ideation and conventional software benchmarks.

2.1 AI for Automating Scientific Discovery

  • Scientific-automation studies integrate idea generation, validation, and writing, while others investigate LLM-generated hypotheses.The section situates this work at the intersection of automating scientific discovery and LLM-based code generation.Source: §2.1 · p. 3, §2 · p. 3
  • PaperBench and Paper2CodeBench study codebase reconstruction but rely substantially on manual or LLM-based correctness judgments.SciReplicate-Bench prioritizes execution accuracy as an objective validation measure.Source: §2.1 · p. 3

2.2 LLMs for Code Generation

  • General code-generation benchmarks such as SWE-bench target software engineering, while ML benchmarks lack rigorous paper analysis.Sci-Reproducer addresses the distinct challenge of converting academic publications into functional implementations.Source: §2.2 · pp. 3–4

3 SciReplicate-Bench

SciReplicate-Bench evaluates repository-level reproduction of algorithms from academic papers, with task inputs, reference implementations, annotations, and tests supporting both comprehension and implementation assessment.

3 SciReplicate-Bench

  • The benchmark contains repository-level tasks from recent NLP papers, each focused on implementing a function or class method.Each task has nine components grouped around code generation, evaluation, and analysis.Source: §3 · p. 4
  • LLMs receive a function signature, algorithm description, literature context, and repository context, and must implement the target function.The benchmark also annotates missing or mismatched algorithm information, including details absent from or inconsistent with the paper description.Source: §3 · p. 4

3.1.1 Evaluating Algorithm Comprehension

  • Reasoning graph accuracy compares generated and reference DAGs whose nodes align reasoning comments with code and whose edges represent data-flow dependencies.Node and edge matches are weighted by significance derived from code complexity and connected-node significance.Source: §3.1.1 · p. 5

3.1.2 Evaluating Code Generation

  • Code generation is assessed with execution accuracy, CodeBLEU, and recall of intra-file dependencies, cross-file dependencies, and external APIs.Execution accuracy requires all test cases to match reference outputs; CodeBLEU also incorporates syntax and data-flow information.Source: §3.1.2 · p. 5

4 Sci-Reproducer

Sci-Reproducer combines a Paper Agent and a Code Agent to reproduce algorithms from papers through literature-supported understanding and repository-aware implementation.

4 Sci-Reproducer

  • Sci-Reproducer uses two collaborating agents with predefined actions to replicate algorithms described in research papers.The Paper Agent and Code Agent work on algorithm understanding and code implementation, respectively.Source: §4 · p. 6

4.1 Paper Agent

  • The Paper Agent selectively queries literature context using ReAct, then produces a report of findings that fill gaps in the target algorithm’s LaTeX description.The report becomes an input to the Code Agent.Source: §4.1 · pp. 6–7

4.2 Code Agent

  • The Code Agent uses the algorithm description, literature report, and code context to find repository dependencies, browse for information, and compile and iteratively debug its implementation.Source: §4.2 · p. 7

5 Experiments

Across seven LLMs, Sci-Reproducer improves code reproduction, but execution accuracy remains low. The analyses relate remaining challenges to tool-use patterns, syntax errors, and missing or mismatched implementation details.

5.1 Results on SciReplicate-Bench

  • 0.390 execution accuracy is the highest achieved, by Claude-Sonnet-3.7; Sci-Reproducer raises execution accuracy by 0.181 on average, yet the benchmark remains challenging.Without agent access to literature and repository context, most LLMs achieve less than 0.1 execution accuracy.Source: §5.1 · p. 8
  • Reasoning graph accuracy averages 0.716 without agents; adding the Paper Agent improves it by 0.009, the Code Agent by 0.036, and both agents by 0.037.The Paper Agent gathers literature context, while the Code Agent extracts repository code patterns and dependency structures.Source: §5.1 · p. 8
  • With Sci-Reproducer, average execution accuracy is 0.235 and CodeBLEU is 0.320; Claude-Sonnet-3.7 reaches 0.390 execution accuracy and dependency/API recalls of 0.776, 0.636, and 0.626.Integrating the Code Agent raises average recall by 0.441, 0.239, and 0.100, respectively, compared with cases without it.Source: §5.1 · p. 8
  • Reasoning models gain 0.13 execution accuracy with Sci-Reproducer, compared with 0.212 for non-reasoning models; the paper attributes the smaller gain to overthinking that impedes tool use.The reported recall improvements are also smaller for reasoning models: 0.243, 0.061, and 0.041 versus 0.560, 0.345, and 0.135 for non-reasoning models.Source: §5.1 · p. 8

5.2 Tool Usage Analysis

  • 25.0, 3.3, and 0.0 average code-search calls by reasoning models contrast with 210.4, 68.2, and 16.8 for non-reasoning models; reasoning models also use paper-search actions less.The chart reports tool-use frequency by action and model; reasoning models invoke Compiler more often, while models generally search the target paper more than external literature.Source: Figure 2 · p. 9, §5.2 · p. 9

5.3.1 Syntax Errors

  • Syntax errors fall from 80.3% without agents and 83.3% with only the Paper Agent to 29.4% with the Code Agent and 24.9% with Sci-Reproducer.The remaining syntax errors mainly arise from incorrect repository dependency use; the approach cannot access runtime information through a compiler during code generation.Source: §5.3.1 · p. 9

5.3.2 Logic Errors

  • Logic errors include valid implementation choices that produce different outputs and gaps between paper descriptions and code; missing details cover hyperparameters, numerical stability, implementation logic, and coding strategy.Mismatched information is less frequent than missing information, and its categories largely overlap.Source: §5.3.2 · pp. 9–10
  • The first category of missing information can generally be addressed through papers, related literature, or repository configurations; the other three require knowledge of machine-learning coding conventions and patterns.The authors suggest that future work could use insights from similar algorithms and real-time compiler feedback.Source: §5.3.2 · p. 10

6 Conclusion

The paper evaluates LLMs on reproducing algorithms from recent NLP papers using SciReplicate-Bench and Sci-Reproducer. Results show the task remains challenging, with failures largely attributed to missing or inconsistent algorithm descriptions.

6 Conclusion

  • The task remains highly challenging, and failures are largely caused by missing or inconsistent algorithm descriptions.The evaluation uses reasoning graph accuracy and standard implementation metrics.Source: §6 · p. 10
  • SciReplicate-Bench provides rich annotations for evaluating algorithm reproduction from recent NLP papers.The paper also presents Sci-Reproducer, a dual-agent framework for bridging algorithm understanding and code generation.Source: §6 · p. 10
  • The evaluation assesses performance using reasoning graph accuracy and standard implementation metrics.Source: §6 · p. 10

A Details of the Annotation Process

The benchmark was assembled by selecting and screening recent NLP papers with open-source implementations, then preparing aligned, annotated tasks and verification suites. Human review, environment setup, and reproducibility controls were used to support reliable evaluation.

A Details of the Annotation Process

  • Papers were collected from leading 2024 NLP conferences, and open-source implementations were identified through PapersWithCode and GitHub APIs.Source: A Details of the Annotation Process · p. 17
  • Selection excluded surveys and exploratory papers, repositories below five stars, and candidates failing manual quality and feasibility review.Reviewers discarded papers with excessive compute demands, poorly structured code, ambiguous documentation, missing preprocessing, or reported reproduction issues.Source: A Details of the Annotation Process · p. 17
  • Annotators set up candidate repositories from their READMEs, attempted to resolve common environment and code problems, and excluded irrecoverable repositories.Source: A Details of the Annotation Process · p. 17
  • Algorithm components were mapped to implementations and refactored into standalone functions or methods; overly fragmented implementations were excluded.Source: A Details of the Annotation Process · p. 17
  • For aligned functions, annotators recorded inputs, outputs, dependencies, and API use, then added comments linking code segments to algorithm components.LLMs assisted with alignment and descriptions, but human annotators reviewed and corrected their outputs.Source: A Details of the Annotation Process · p. 18
  • Verification suites generally contain 10 test cases per task, with fewer when source data or algorithm constraints limited case construction.Annotators fixed random seeds and nondeterministic operations; users ran reference and generated code locally to control hardware variation, with task-specific comparison scripts handling return-type differences.Source: A Details of the Annotation Process · p. 18

B Details of the Task Categories

SciReplicate-Bench spans five NLP task categories, whose distribution is illustrated in Figure A1.

B Details of the Task Categories

  • The five categories are representation and embedding methods, loss functions and optimization objectives, information extraction and aggregation, model architecture components, and inference and search algorithms.Figure A1 illustrates the distribution across these categories.Source: B Details of the Task Categories · p. 18, Figure A1 · p. 18

C Details of the Actions

Sci-Reproducer’s actions retrieve paper and repository information, then execute generated code in the task’s Python environment.

C Details of the Actions

  • SearchPaper parses a paper’s LaTeX and iteratively extracts subsection information relevant to the Paper Agent’s query; SearchSection retrieves a requested section.Source: C Details of the Actions · p. 19
  • SearchLiterature retrieves query-relevant paper content from arXiv when LaTeX is available, otherwise returning no information.Source: C Details of the Actions · p. 19
  • SearchCode uses Python AST parsing to return named code definitions, while SearchFile returns the full contents of a named file.Source: C Details of the Actions · p. 19
  • SearchWeb filters Google Search API results for query-relevant information, and Compiler inserts generated code into the original Python file and runs it.The compiler returns execution output as feedback.Source: C Details of the Actions · p. 19

D Human Evaluation of Reasoning Graph Accuracy

A human evaluation on 20 tasks aligned with the automated reasoning-graph assessments. The authors identify full-benchmark human evaluation as future work.

D Human Evaluation of Reasoning Graph Accuracy

E Figures and Tables

The appendix materials illustrate the reproduction task, index benchmark and evaluation tables, and provide prompts and output examples for the agents.

E Figures and Tables

Loading 2504.00255v2…