Source-linked AI summary
DeepRepro: State-Aware Subplanning for Paper-to-Code Reproduction in Evolving Repositories
Hongru Song, Ruqing Zhang, Jiafeng Guo, Xueqi Cheng, Maarten de Rijke
TL;DR
Paper-to-code reproduction requires maintaining a repository whose dependencies, interfaces, and execution feedback evolve beyond static upfront plans. DeepRepro uses state-aware iterative subplanning and outperforms strong scientific and commercial code-agent baselines on PaperBench Code-Dev, achieving an average score of 84.2 on five papers.
Problem
Paper-to-code reproduction lacks planning that continually aligns implementation with evolving repository state and runtime feedback during long-horizon construction.
Method
DeepRepro conditions iterative subplans on the current repository state, using repository-aware orchestration, execution diagnostics, memory, and bounded repair.
Results
84.2 average score on five PaperBench Code-Dev papers, improving over DeepCode, Cursor, and Codex by 1.3, 17.7, and 23.2 points, respectively.
Takeaways & Limitations
DeepRepro consistently outperforms strong scientific and commercial code-agent baselines across PaperBench Code-Dev evaluation settings.
Abstract
from arXiv · showhide
Recent advances in agentic large language models (LLMs) have enabled increasingly autonomous software engineering workflows, yet automatic machine learning (ML) paper-to-code reproduction remains a challenging long-horizon problem. Unlike conventional code generation, this task requires constructing and maintaining a fully functional repository whose state continuously evolves during execution. Existing systems typically rely on static upfront planning followed by sequential file-level generation, which often leads to inconsistencies as dependencies, interfaces, and execution feedback change over time. We propose DeepRepro, a state-aware framework for paper-to-code reproduction based on execution-state-aware subplanning. DeepRepro dynamically transforms evolving repository states and runtime feedback into fine-grained implementation subplans, keeping planning aligned with execution throughout repository construction. The framework further incorporates repository-aware orchestration and a lightweight process-aware interface for transparent monitoring of long-horizon reproduction. Experiments on PaperBench Code-Dev show that DeepRepro consistently outperforms strong scientific and commercial code-agent baselines.
1 Introduction
Paper-to-code reproduction requires autonomous agents to construct complete, executable repositories whose evolving state can invalidate static plans. DeepRepro addresses this challenge through state-conditioned iterative subplanning, repository-aware orchestration, and strong PaperBench performance.
- Motivation: Scientific paper-to-code reproduction requires reconstructing a complete, executable codebase from a research paper description.
- Limitations: Existing systems often depend on an initial global plan, while specialized frameworks remain limited in long-horizon reproduction settings.
- Limitations: Evolving repositories make static plans rapidly misaligned as interfaces, dependencies, and assumptions change, causing cascading inconsistencies and repeated repair cycles.
- DeepRepro: DeepRepro continuously conditions planning on the current implementation state through iterative subplanning over evolving repositories.
- DeepRepro: DeepRepro combines Blueprint planning, repository-aware orchestration, and dynamic selection of tightly coupled files or repair targets from repository state.Blueprint distills the paper into a repository-level skeleton covering modules, execution flow, dependencies, and evaluation protocols.
- Evaluation: 84.2 average score on PaperBench’s five-paper subset exceeded DeepCode, Cursor, and Codex by 1.3, 17.7, and 23.2 points, respectively.On the shared human-expert subset, DeepRepro exceeded reported human-expert results by 5.4 points.
2 The DeepRepro Framework
DeepRepro preprocesses paper inputs into structured representations and an initial blueprint specifying repository construction. It then repeatedly adapts file-level implementation and repair subplans to repository state, memory, and diagnostics, while exposing progress through a monitoring interface.
- Paper input and preprocessing: DeepRepro normalizes papers, figures, and user requirements into a clean workspace and preprocessed document representation for planning.It converts local files or URLs to Markdown, standardizes artifact paths, and selects full-document or segmented reading based on document size and preprocessing status.
- Blueprint planning: Planning extracts visual, conceptual, and algorithmic evidence, then synthesizes these views with user requirements into an initial blueprint.The extracted details cover problem settings, methods, datasets, metrics, results, algorithms, equations, model components, losses, training procedures, hyperparameters, and assumptions.
- Blueprint planning: The blueprint specifies repository structure, implementation order, modules, interfaces, data flow, reproduction settings, environment requirements, and validation procedures.It serves as the global specification for the subsequent generation loop, with associated repositories for important cited papers available as optional references.
- State-aware execution: During execution, DeepRepro selects related files or repair targets, updates repository memory, collects diagnostics, and uses these signals to compute the next subplan.Subplans may target new or repair files; fast mode specifies only the target set, whereas deepplan mode additionally includes grouping rationale and further planning fields.
- State-aware execution: Diagnostics include tool errors, invalid writes, syntax errors, dependency issues, and layout problems, while synchronized memory preserves file purposes, interfaces, dependencies, and implementation notes.The loop stops when the selected target set is empty and then finalizes the repository and reproduction report.
- Process-aware monitoring: A process-aware frontend exposes file status, subplans, diagnostics, memory updates, and execution status for monitoring and issue identification.It receives execution updates from the repository-aware orchestration loop during long-horizon reproduction.
3 Experimental Setup
DeepRepro is evaluated on PaperBench Code-Dev using hierarchical Code Development rubrics, with experiments covering a five-paper subset and comparisons against scientific code-agent baselines. The main LLM-judge experiments use GPT-4.1-mini, while human-expert comparisons additionally use o3-mini to align with PaperBench judging.
- Dataset and evaluation: Evaluation uses PaperBench Code-Dev, a code-development subset built from 20 ICML 2024 Spotlight and Oral papers.The setup follows the official Code-Dev evaluation protocol.
- Dataset and evaluation: Scores are reported as percentages from hierarchical Code Development rubrics, with reproduction execution skipped.The evaluation grades only Code Development requirements rather than executing reproductions.
- Dataset and evaluation: The five-paper subset comprises FRE, RICE, PINN, BaM, and Mech-U, with full-set scores and fixed subset averages reported when needed.These subset papers support the reported subset-level comparisons.
- Evaluation models: Main LLM-judge experiments use GPT-4.1-mini, while human-expert comparisons additionally use o3-mini for fairer alignment with the PaperBench judging setup.The model choice is motivated by cost control for the main experiments and comparability for human-expert evaluation.
- Baselines: DeepRepro is compared with scientific code agents including DeepCode, PaperCoder, and AutoRepro.DeepCode is identified as the main scientific baseline.
4 Experimental Results
DeepRepro’s deepplan mode improves average performance over fast mode and remains effective across execution models, with larger gains for weaker executors. On all 20 Code-Dev papers, it achieves an overall average score of 75.09, demonstrating robustness beyond the controlled five-paper setting.
- Overall performance: Deepplan raises the average score from 82.07 to 84.22, consistently outperforming fast across four execution settings.External code references have only marginal impact on this subset.
- Effect of subplanning and model choices: Deepplan improves performance across both strong and weaker execution models, with more pronounced gains for weaker executors.The result suggests subplanning compensates for execution-level limitations and improves global coordination over long-horizon tasks.
- Full-set evaluation: 75.09 overall average score is achieved on all 20 Code-Dev papers, demonstrating robustness across diverse tasks beyond the controlled five-paper setting.Table 3 reports the full-set evaluation results.
5 Related Work
Prior coding-agent research has advanced from code completion toward autonomous software engineering through collaboration, workflow decomposition, repository editing, testing, search, localization, task graphs, and iterative validation. Scientific reproduction is more demanding because agents must convert research contributions and experimental methodology into executable implementations.
- General coding agents: Coding agents have progressed from code completion to autonomous software engineering, with prior work studying role-based collaboration, workflow decomposition, repository editing, and testing.
- General coding agents: Search, localization, task graphs, and iterative validation support issue resolution in IDE- and terminal-based coding agents.
- Scientific coding agents: Scientific reproduction is more demanding than general coding because agents must translate research contributions and experimental methodology into executable implementations.
6 Conclusion
DeepRepro presents ML paper-to-code reproduction as a state-aware, iterative repository-construction process. It combines execution-state-aware planning, adaptive execution support, and process-aware monitoring for long-horizon reproduction.
- DeepRepro treats ML paper-to-code reproduction as a state-aware, iterative process over evolving repositories.
- The framework combines blueprint planning with execution-state-aware round-level subplanning.
- DeepRepro integrates asynchronous memory compression, bounded repair, and a process-aware interface for long-horizon execution and monitoring.