PaperlayerView exact revision on arXiv

Source-linked AI summary

ReproAgent: Contract-Guided Paper-to-Code Reproduction

Xue Hu, Zewei Pan, Zhongyuan Wang, Zhou Liu, Zeli Su, Wentao Zhang

arXiv:2608.24291v1cs.AIcs.SE

TL;DR

Paper-to-code reproduction must preserve paper-specific methods, protocols, and artifacts despite incomplete explicit specifications and implicit implementation details. ReproAgent uses a four-stage pipeline with persistent requirement and reference-evidence contract channels bound to work packages and file contracts. On PaperBench Code-Dev, it achieves the highest mean score among same-backbone scaffolds, while ablations support contributions from both channels.

  • Problem

    Paper-to-code agents can produce runnable repositories that remain unfaithful because explicit obligations drift and implicit implementation choices lack grounded evidence.

  • Method

    ReproAgent uses a Prepare–Plan–Generate–Repair pipeline with persistent implementation-requirement and reference-evidence channels projected into work-package and file-level contracts.

  • Results

    ReproAgent reaches the highest mean score among same-backbone scaffolds under Claude-Sonnet-4.5 and Gemini-3-Flash, with ablations supporting contributions from both channels.

  • Takeaways & Limitations

    Persisting paper obligations and reference-repository evidence in one contract anchors planning, exposes coverage failures, and scopes repair.

  • Takeaways & Limitations

    The evaluation targets repository-level reproduction on PaperBench Code-Dev, with extension to other settings, modalities, and protocols left for future work.

Abstract

from arXiv · show

Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as algorithms, metrics and artifacts is often lost across long agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare--Plan--Generate--Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev, ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.

1 Introduction

Paper-to-code reproduction must preserve explicit paper obligations while grounding implicit implementation choices. ReproAgent addresses this with a persistent two-channel contract carried through a four-stage pipeline and shows the highest mean score among same-backbone scaffolds.

  • Motivation: Paper-to-code systems can produce runnable but unfaithful repositories that omit or alter paper-specific algorithms, metrics, artifacts, or loss terms.Runtime success alone does not establish reproduction fidelity.
  • Motivation: Explicit paper requirements and implicit implementation details are split across paper text, framework defaults, cited implementations, and community conventions.Without a persistent contract, explicit obligations can drift across long trajectories and implicit choices can rely on model priors.
  • Method: ReproAgent uses an implementation-requirement channel for paper-derived code obligations and a reference-evidence channel for related-repository evidence.Both channels are bound to work packages and projected into file-level contracts that guide generation and audit.
  • Method: The Prepare–Plan–Generate–Repair pipeline extracts requirements, organizes work packages and file contracts, generates files, and repairs violations using the same contract.The pipeline preserves obligations across generation and repair.
  • Results: ReproAgent reaches the highest mean score among same-backbone scaffolds under both evaluated backbones, while ablations support contributions from both channels.The reported evidence combines same-backbone comparisons, channel ablations, and per-paper analysis.
  • Evaluation: The evaluation uses the full 20-paper PaperBench Code-Dev suite with same-backbone comparisons and channel ablations.The study runs ReproAgent with Claude-Sonnet-4.5 and Gemini-3-Flash.

2 Related Work

Related work spans research agents, repository-level paper reproduction, retrieval, contracts, and fidelity evaluation. ReproAgent targets a traceability layer that persistently binds paper obligations and repository evidence across the implementation pipeline.

  • Research and code agents: Research agents increasingly support research workflows, ML engineering, and repository-level coding, but paper reproduction requires fidelity to scientific methods, protocols, and artifacts.The target is more than generic execution success.
  • Paper-to-code reproduction: Prior paper-to-code systems use staged planning, multimodal parsing, hierarchical supervision, citation-graph retrieval, or agentic coding for repository reproduction.The cited systems include PaperCoder, Sci-Reproducer, AutoP2C, HiRAS, AutoReproduce, and DeepCode.
  • Paper-to-code reproduction: Prior systems do not make both paper-derived obligations and retrieved repository evidence a persistent, file-level audit object shared across planning, generation, and repair.ReproAgent positions this shared traceability layer as its target distinction.
  • Retrieval and contracts: Retrieval-augmented code generation retrieves documentation, repositories, or code graphs but does not tie each snippet to the scientific requirement it supports.ReproAgent connects retrieval with requirements traceability and design-by-contract ideas.
  • Retrieval and contracts: ReproAgent separates reference evidence into content evidence for implementation and structure evidence for architecture planning, binding both to work-package and file-level contracts.This makes evidence persistent across the pipeline.
  • Evaluation and fidelity: RePro applies paper-derived criteria after a candidate repository exists, whereas ReproAgent places those requirements in the forward generation contract before generation.The audit target is fixed in advance rather than reconstructed afterward.

3 Method

ReproAgent converts paper requirements and related-repository evidence into persistent work-package and file-level contracts. Generation, requirement review, runtime validation, and targeted repair then operate against these contracts.

  • Contract design: ReproAgent maintains implementation-requirement and reference-evidence channels that persist from paper analysis through planning, generation, validation, and repair.The requirement channel captures explicit obligations; the evidence channel supplies implicit implementation knowledge.
  • Implementation requirements: An implementation requirement is a paper snippet normalized into a code obligation with identifiers, source spans, citations, expected code surfaces, artifacts, and status.The unit representation is u = (id, src, stmt, cite, surf, art, status).
  • Implementation requirements: Extraction segments the paper into section-level chunks and retains snippets implying concrete implementation work, including algorithms, losses, training settings, preprocessing, metrics, and artifacts.Motivation, related-work narration, and discussion are skipped.
  • Preservation invariant: Each active requirement must receive ownership, file projection, provenance, and a review verdict; failed predicates create contract violations and repair obligations.Verdicts include pass, missing, wrong-surface, and inconsistent-with-source.
  • Reference evidence: Reference evidence addresses implicit details by collecting related-repository snippets and binding each item to the work package owning its supported requirements.Evidence records identify repositories, paths, regions, supported units, roles, and matching bases.
  • Reference evidence: Content evidence informs implementation choices, while structure evidence informs file trees, interfaces, entry points, dependencies, and architecture.Architecture planning combines structural priors with task and requirement information.
  • Work packages and generation: Work packages bind requirements to evidence through goals, dependencies, interfaces, artifacts, and attached evidence, after which a frozen global contract defines requirements, packages, evidence, architecture, and validation.File contracts project this global specification into per-file responsibilities, symbols, artifacts, evidence, and review checks.
  • Generation and repair: Generation proceeds file by file under package-local contracts, while requirement review and runtime validation distinguish paper inconsistency from broken execution.Repair issues targeted tickets and patches affected files while protecting units that already passed.

4 Experimental Setup

The evaluation measures repository-level paper reproduction on the 20-paper PaperBench Code-Dev benchmark through full-suite, same-backbone, and channel-ablation comparisons. The protocol fixes the judging metric, repair budgets, and comparison scaffolds across runs.

  • Benchmark: PaperBench Code-Dev contains 20 ICML 2024 papers with hierarchical rubrics grading paper-specific methods, protocols, and artifacts rather than execution alone.The benchmark evaluation has full-suite, same-backbone, and ablation layers.
  • Evaluation layers: The full-suite comparison uses Claude-Sonnet-4.5, while same-backbone and ablation comparisons use Gemini-3-Flash shared with BasicAgent, IterAgent, and AiScientist.The ablation layer removes each contract channel under the same Gemini backbone.
  • Baselines: The aggregate full-suite figure reports Code-Dev rows, while per-paper tables compare BasicAgent, IterAgent, AiScientist, DeepCode, and both ReproAgent backbones.External rows reuse source-reported model, scaffold, and runtime budgets.
  • Metric: Macro-averaged PaperBench score is the primary metric, computed by aggregating weighted leaf scores into per-paper scores and averaging across 20 papers.Token, time, cost, and judge-provenance diagnostics are reported separately.
  • Implementation and repair: REPROAGENT combines paper chunking, metadata- and search-based reference collection, frozen contract planning, file-level generation, requirement review, and sandboxed runtime validation.Repair is capped at three stage-review attempts and five repository-level rounds.
  • Ablations: The Gemini channel ablations hold all other components and repair budgets fixed while removing either requirement extraction and review or reference-repository surveying.This isolates the effect of each contract channel.

5 Results and Analysis

REPROAGENT achieves strong repository-level reproduction scores across Claude and Gemini comparisons, and ablations show that both contract channels contribute complementary capabilities. Case analyses connect requirement evidence to preserving paper-specific obligations and reference evidence to repository-level scaffolding and artifacts.

  • 5.1 Full-Suite Comparison: 73.7 mean PaperBench Code-Dev score for REPROAGENT with Claude-Sonnet-4.5 exceeds DeepCode’s 73.5 and earlier published systems.The comparison includes PaperCoder at 45.1, AutoP2C at 49.2, AutoReproduce at 49.6, and Deep-Reproducer at 63.2.
  • 5.1 Full-Suite Comparison: 39.7 mean PaperBench Code-Dev score for REPROAGENT with Gemini-3-Flash exceeds BasicAgent, IterAgent, and AiScientist.The corresponding scores are 19.3, 20.6, and 30.5, respectively.
  • 5.3 Ablation Study: 18.1-point and 14.1-point mean drops follow removal of reference evidence and implementation requirements, respectively, under Gemini-3-Flash.The full setting beats both ablations on all 20 targets, indicating that each channel contributes capability the other cannot recover.
  • 5.3 Ablation Study: 53.8 full-contract score for Sample-specific Masks falls to 21.3 without implementation requirements and 28.0 without reference evidence.The requirement channel preserves named mask-generation obligations, baselines, ablations, and mask-channel switches in the generated repository.
  • 5.3 Ablation Study: 50.6 full-contract score for Bridging Data Gaps falls to 13.9 without reference evidence, which supplies protocol, layout, wiring, configuration, and artifact conventions.Without that channel, the repository fails to produce expected checkpoint, training-trace, and resolved-configuration artifacts.
  • 5.4 Discussion: The requirement channel preserves explicit paper obligations, while the evidence channel grounds implicit repository knowledge.Both channels are persisted in a global contract that guides generation and provides an audit target during repair.

6 Conclusion

REPROAGENT is a contract-guided pipeline that binds paper obligations and reference-repository evidence to work packages, file contracts, and repair decisions. Across PaperBench Code-Dev’s 20 papers, it scores 73.7, while comparisons, ablations, and case traces support the role of both channels.

  • 6 Conclusion: REPROAGENT binds explicit paper obligations and implicit reference-repository evidence to work packages, file contracts, and repair decisions.The contract persists both evidence types across planning, generation, and repair.
  • 6 Conclusion: 73.7 is REPROAGENT’s PaperBench Code-Dev score across all 20 papers.Same-backbone comparisons support scaffold-level gains, while channel ablations and case traces support both contract channels.

7 Limitations

REPROAGENT targets repository-level reproduction where the paper and benchmark-allowed reference material jointly specify the implementation target. Extension to other reproduction settings, modalities, evaluation protocols, and deployment-cost optimizations remains future work.

  • 7 Limitations: The approach is scoped to repository-level reproduction using the paper and benchmark-allowed reference material as the joint implementation target.Other reproduction settings, modalities, and evaluation protocols are identified as directions for future work.
  • 7 Limitations: Token, time, and call diagnostics are reported for auditability rather than deployment-cost comparison, leaving engineering optimizations to future work.The contract design is described as compatible with such optimizations.

A.1 An Implementation-Requirement Unit

An implementation-requirement unit is a structured, addressable record that preserves one paper-derived coding duty across planning, generation, validation, and repair. Its schema connects evidence, verification targets, implementation surfaces, and concrete code obligations to work packages and file contracts.

  • Running example: The example contract records EWC as a method obligation with Fisher-matrix calculation and EWC-loss implementation as its verification artifact.Its implementation surface is model_or_method.
  • Unit schema: Each unit stores a stable identifier, type, normalized requirement statement, paper evidence, source paragraph anchors, verification targets, implementation surfaces, code obligations, and lifecycle status.These fields let review trace a requirement from its paper origin to its intended code location and verdict.
  • Auditability: The unit schema supports pointwise review by declaring artifact, surface, or metric checks alongside the code locations where obligations must appear.This turns a paper requirement into an auditable contract rather than a transient planning fragment.
  • Preservation invariant: Plan requires the EWC unit to have an owner work package, and file planning requires projection into a file contract covering model_or_method.In the example, the unit is owned by a method package and lands in a dedicated method file.
  • Preservation invariant: Generate cannot finalize the file until it emits provenance citing unit_004, keeping the requirement visible in the per-file provenance map.The same identifier therefore links the requirement to generated files and later repair actions.
  • Reference binding: Reference evidence is scoped to the work package owning the unit, so generation receives relevant repository structure without unrelated sibling-package code.The evidence item records a repository path, snippet region, match basis, confidence, and matched keywords for later review and repair.

C Gemini Ablation Case Studies

The Gemini ablation case studies use qualitative traces alongside aggregate scores to examine how the two contract channels affect paper-to-code fidelity. The examples focus on whether requirements and repository organization survive channel removal, rather than treating execution alone as sufficient.

  • Case-study design: The appendix keeps qualitative ablation cases separate from aggregate results, using them as mechanism-oriented evidence for the channel analysis.The cases illustrate channel roles behind the aggregate ablation pattern in Section 5.3.
  • Test-Time Model Adaptation: The FOA/CMA trace tests whether implementation requirements preserve named method duties while contrasting how reference evidence affects repository organization.The trace uses the corresponding Gemini full-run artifacts for test-time-model-adaptation.
  • Test-Time Model Adaptation: 54.9, 47.1, and 23.8 are the full Gemini, without implementation requirements, and without reference evidence scores for Test-Time Model Adaptation.The snapshot is identified as the consolidated 2026-05-24 PaperBench judging snapshot.
  • Context: Table 4 provides contextual external scores rather than a primary controlled comparison because its rows differ in scaffold, model, runtime budget, and reporting protocol.The table includes official PaperBench results, recent systems, and separately collected reference rows.
  • FOA/CMA trace: The FOA/CMA requirement package preserves prompt insertion, CMA-ES optimization, population size K ∈[2, 28], forward-only updates, source statistics, and the fitness function.Its contract also records output files, registries, parameter inventory, and validation checks.
  • Evidence binding: The FOA/CMA package is marked self_contained and has no repository evidence_links because its decisive obligations are present in paper and addendum evidence.This supports the interpretation that reference evidence can add organization even when requirements remain recoverable without it.

C.2 Tracing Reference Evidence in Bridging Data Gaps

Bridging Data Gaps isolates an asymmetric channel effect: implementation requirements preserve high-level duties and outputs, while reference evidence supplies concrete repository protocol and artifact organization.

  • Ablation result: 50.6, 38.4, and 13.9 are the full Gemini, without implementation requirements, and without reference evidence scores for Bridging Data Gaps.The no-reference ablation shows the larger drop in this case.
  • Requirement channel: The requirement channel keeps adaptation tasks, runnable experiment paths, checkpoint saving, and inspectable result-artifact writing visible in file contracts.The contracts expect separate training and artifact-writer routes rather than one monolithic script.
  • Reference-evidence channel: The reference channel contributes implementation texture through file layout, scripts, data-handling conventions, and experiment wiring that the paper does not specify line by line.In the full run, this includes checkpoint and reproducible-configuration logging patterns.
  • Generated artifacts: The full repository writes checkpoints/adaptor.pth, checkpoints/trained_model.pth, results/ant_training_trace.json, and results/config_resolved.json.These artifacts match the expected pattern of checkpointed training and reproducible configuration logging.

C.3 Additional Ablation Patterns

Additional cases show that both channels improve fidelity, but their relative value depends on how standardized, broad, or implementation-critical the target paper is. The traces connect score patterns to preserved obligations, repository structure, and expected artifacts.

  • Implementation traces: The code excerpts materialize contracts as file-owned method registries, training loops, checkpoints, and JSON result artifacts rather than leaving them only in planning prose.The table caption defines the ablation labels as removal of the implementation-requirement or reference-evidence channel.
  • Sample-specific Masks: The Sample-specific Masks trace preserves mask generation, low-resolution interpolation, joint optimization, named baselines, and shared-pattern and mask-channel ablations in the full run.The full contract binds these obligations to files and method registries checked by the rubric.
  • BAM: 48.5, 30.8, and 20.7 are the full, no-implementation, and no-reference scores for BAM.The case is presented as a similar large two-channel-gain pattern.
  • Channel roles: The requirement channel keeps paper-specific modeling and protocol obligations explicit, while reference evidence recovers practical implementation and evaluation structure.Removing reference evidence retains obligations but reduces concrete organization for baselines and ablation routes.
  • PINN: 61.3, 53.3, and 52.2 are the full, no-implementation, and no-reference scores for PINN, a smaller-margin case with strong public implementation priors.Common collocation, PDE-loss, optimizer, and diagnostic-plot patterns make reasonable recovery possible after removing one channel.
  • PINN: The full PINN contract binds common implementation priors to the paper-specific objective, PDE setup, loss-landscape diagnostics, and expected artifacts.The evidence supports the channel hypothesis without reducing the case to generic repository conventions.
  • What Will My Model Forget: 16.8 is the full Gemini score for What Will My Model Forget, with both ablations lower in this scale-sensitive case.Broad long-tail coverage and many benchmark-specific details make the case sensitive to backbone capacity as well as channel utility.

C.4 Why Ablation Gaps Vary

Ablation gaps vary because the two contract channels provide leverage in different circumstances: reference evidence transfers repository structure, while implementation requirements preserve paper-specific obligations. Small gaps are ambiguous, reflecting strong public priors, shared bottlenecks, or limited coverage rather than channel irrelevance.

  • Observed variation: +43.5 is the largest implementation-requirement gain, on Mechanistic Understanding, while +36.7 is the largest reference-evidence gain, on Bridging Data Gaps.Other large gains occur on Adaptive Pruning, Test-Time Adaptation, FTRL, All-in-One, BAM, Sample-Specific Masks, and LCA-on-the-Line.
  • Observed variation: The two channels help for related but distinguishable reasons, with representative cases exposing different concrete implementation surfaces.Sample-Specific Masks relies on preserving method-specific masks, interpolation, baselines, and ablations; Bridging Data Gaps relies more on protocol and training-artifact wiring.
  • Reference transferability: Reference-evidence gains are largest when external artifacts transfer concrete protocol and repository-structure decisions into executable data handling, experiment wiring, and comparison surfaces.Removing reference evidence can leave paper obligations visible while weakening executable repository structure.
  • Requirement hardness: Implementation requirements help most when compact, implementation-critical mechanisms, constraints, losses, ablations, metrics, or artifacts determine repository fidelity.Sample-Specific Masks depends on obligations including mask generation, interpolation, joint optimization, named baselines, and channel ablations.
  • Small-gap cases: Small ablation gaps can reflect strong public implementation priors, shared implementation bottlenecks, or scale and coverage stress cases rather than irrelevance.PINN has strong public priors, BBOX has compressed gaps across variants, and What Will My Model Forget contains many long-tail requirements despite the full contract leading.
Loading 2608.24291v1…