Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning

1 INTRODUCTION

• Only 19.5% of papers accepted to top-tier machine-learning conferences in 2024 provide code, leaving reproduction slow and labor-intensive.The paper identifies incomplete documentation, missing experimental details, and lack of data or proprietary tools as additional obstacles.Source: §1 · p. 1
• Prior LLM experimental-automation studies (Huang et al., 2024; Zhang et al., 2024; Trirat et al., 2024) typically rely on existing implementations, snippets, or well-defined APIs, leaving paper-only generation uncertain.Source: §1 · pp. 1–2
• PaperCoder decomposes paper-to-repository implementation into planning, file-and-function analysis, and dependency-aware code generation.Planning produces a roadmap, architecture diagrams, file dependencies and execution order, and configuration files; analysis specifies file and function behavior, inputs, outputs, interactions, and paper-derived constraints.Source: §1 · p. 2
• PaperCoder repositories were rated best over baselines in 88% of cases, and 92% of author-judges found them helpful.All three stages contributed to performance gains; when execution errors occurred, repositories averaged 0.81% of code lines needing minor modifications.Source: §1 · p. 2

Large Language Models for Code

• Code-specialized LLMs have demonstrated capabilities across software design, development, requirements elicitation, and formal specification, motivating their application to scientific code generation.The cited work situates this paper within research on specialized LLM capabilities.Source: §2 · p. 2

Repository-Level Coding

• Repository-level coding has expanded beyond single-file generation to multi-file systems, with ChatDev (Qian et al., 2024) and MetaGPT (Hong et al., 2024) using collaborative agent workflows.The paper targets the less-explored task of turning full, complex papers into repository-level code.Source: §2 · p. 3

LLM-Powered Scientific Research

• Many LLM-assisted scientific coding efforts build on original codebases, limiting applicability when implementations are unavailable; PaperBench (Starace et al., 2025) evaluates paper reproduction, while this work focuses on the transformation method.PaperBench emphasizes evaluation, whereas this work complements it by addressing how to create repository-level implementations from papers.Source: §2 · p. 3

3.1 REPOSITORY-LEVEL CODE GENERATION FROM MACHINE LEARNING PAPERS

• PaperCoder defines repository-level generation as mapping a paper to a cohesive multi-file repository, decomposing the task because single-pass generation struggles with paper complexity, context limits, and cross-file consistency.The repository’s files implement different components of the paper’s methods and experiments while forming a unified pipeline.Source: §3.1 · p. 3

3.2 PAPERCODER: LLM-POWERED MULTI-AGENT FRAMEWORK FOR PAPER-TO-CODE

• PaperCoder coordinates specialized agents across Planning, Analysis, and Coding to transform a paper into an implementation plan, detailed analysis, and final repository.The planning outputs guide file-level analysis, which in turn informs code generation; the workflow assumes no pre-existing artifacts or implementations.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.2.1 PLANNING

• Planning converts a paper’s human-oriented narrative into implementation abstractions through four sequential components: an overall plan, architecture design, logic design, and configuration generation.The overall plan identifies methods and experiments; architecture specifies files, modules, relationships, and diagrams; logic orders files and elaborates their contents; configuration records hyperparameters, model settings, and runtime options in config.yaml.Source: §3.2.1 · pp. 4–5

3.2.2 ANALYSIS

• Analysis processes each planned file to specify its functional goals, input-output behavior, dependencies, and algorithmic details derived from the paper.It distinguishes how components should be implemented from planning’s description of which components to build and how they relate.Source: §3.2.2 · p. 5

3.2.3 CODING

• Coding generates files sequentially in the planned execution order, using the paper, planning outputs, file-specific analyses, and previously generated code as context.This gives each file awareness of its dependencies and the repository’s evolving state.Source: §3.2.3 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation covers 90 papers in Paper2CodeBench, 21 additional papers for human evaluation, and 20 PaperBench Code-Dev papers, using reference-based, reference-free, and author-led human assessments.Related repository-generation approaches and PaperCoder variants serve as baselines; reference-based assessment uses author code when available, while reference-free assessment judges generated code from the paper alone. Human participants rank implementations of papers they first-authored.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTAL RESULTS AND ANALYSIS

• PaperCoder consistently outperforms all baselines on Paper2CodeBench and performs on par with author-released repositories in the Oracle setting, without statistically significant differences.The authors attribute the performance gap to its top-down sequence of planning, analysis, and coding for long-form scientific papers.Source: Table 1 · p. 7, §4.2 · p. 7
• Author evaluations rank PaperCoder best, model-based evaluations correlate strongly with human judgments, and inter-annotator agreement is high (Cohen’s kappa 0.79).Reference-based and reference-free evaluations also have a strong positive correlation (Pearson r = 0.79); the authors use o3-mini-high as the default evaluator.Source: Table 2 · p. 7, §4.2 · pp. 7–8, Table 5 · p. 8
• PaperCoder achieves the highest replication scores across o3-mini-high and Claude 3.5 Sonnet on PaperBench Code-Dev, outperforming that benchmark’s baselines.This evaluation uses human-annotated, paper-specific rubrics and supports the method’s generalizability across benchmarks and models.Source: Table 3 · p. 8, §4.2 · p. 8
• On Paper2CodeBench, o3-mini-high outperforms the tested open-source backbones across evaluation settings, with DS-Distill-Qwen performing best among those models.Qwen-Coder follows DS-Distill-Qwen, then DS-Coder; the authors consequently use o3-mini-high primarily.Source: Table 4 · p. 8, §4.2 · p. 8
• Adding PaperCoder components generally improves performance, although architecture design alone causes a drop; Self-Refine improves planning and analysis outputs and yields downstream gains.The architecture-design issue is addressed when logic design specifies file dependencies and generation order; integrating all modules yields the highest performance.Source: Table 6 · p. 8, §4.2 · pp. 8–9, Table 7 · p. 8
• Human analysis finds that 92% say PaperCoder’s top-ranked repository would ease reproduction, while Method and Evaluation coverage reaches 80% and 79%, respectively.Data Processing accounts for many observed errors because papers may underspecify formats, preprocessing, or loading; 22 of 25 annotators preferred PaperCoder repositories, citing completeness, clean structure, and faithfulness. Oral/spotlight papers also score slightly higher than posters on GPT-4o and o3-mini model-based evaluations.Source: Figure 4 · p. 8, Figure 5 · p. 9, §4.2 · p. 9

4.3 ADDITIONAL ANALYSIS ON REPRODUCTION FROM IMPLEMENTED CODE REPOSITORY

• Across five manually tested repositories, an average of 0.81% of code lines needed minor changes for successful execution; evaluations also find stronger reproduction results than baselines.PaperCoder scores highest on a 10-paper PaperBench sample and outperforms baselines on author rubrics with or without LLM-assisted debugging; four of five case-study repositories at least partially reproduce results, while one fails due to loss-function design.Source: §4.3 · pp. 9–10, Table 8 · p. 9, Figure 6 · p. 9

5 CONCLUSION

• PaperCoder consistently outperforms existing baselines on Paper2CodeBench and PaperBench Code-Dev in model-based and human evaluations.Paper2CodeBench comprises recent papers from top-tier machine-learning venues; PaperBench Code-Dev provides fine-grained evaluation protocols.Source: §5 · p. 10
• Its three stages plan the roadmap, architecture, execution logic, and configuration; analyze files; then generate code sequentially using prior-stage artifacts.Source: §5 · p. 10
• PaperCoder remains effective across LLM backbones, needs minor fixes for only 0.81% of lines, and benefits from every pipeline stage.Source: §5 · p. 10
• The authors envision PaperCoder as a step toward accelerating scientific progress by aiding research-paper reproduction.Source: §5 · p. 10

ETHICS STATEMENT

• The authors believe generating faithful code repositories can contribute to open science and facilitate rapid experimentation.Source: ETHICS STATEMENT · p. 10
• The framework could reproduce sensitive implementations that some papers withhold for security reasons, including jailbreaking or exploitation techniques.Source: ETHICS STATEMENT · p. 10
• Responsible production use would require safeguards such as harmful-content filters, protective prompting, and secure execution environments.Source: ETHICS STATEMENT · p. 10

ACKNOLEDGEMENTS

• The work was supported by IITP and NRF grants funded by the Korean government, a Global AI Frontier Lab collaborative research grant, and the InnoCORE program.The cited support includes grant numbers RS-2019-II190075, RS-2023-00256259, RS-2024-00469482, RS-2024-00509279, and N10250156.Source: ACKNOLEDGEMENTS · p. 10

A.1 IMPLEMENTATION DETAILS

• Generating repositories for 90 papers with o3-mini-high cost $76.65 total, or approximately $0.90 per paper.The pipeline used OpenReview and Semantic Scholar APIs for paper data and s2orc-doc2json to convert documents into structured JSON.Source: §A.1 · p. 18

A.2 HUMAN EVALUATION PROCESS

• Human annotators with peer-reviewed publications and computer-science degrees evaluated anonymized repositories spanning model variants, naive baselines, and related frameworks.They reviewed 10 repositories per paper, taking about 45 minutes on average; participants were paid $15 per hour.Source: §A.2 · p. 18

A.3 REFERENCE-BASED EVALUATION

• When a reference repository exceeded the model’s context length, the evaluator selected relevant files for scoring and used gpt-4o-2024-11-20.The file-selection procedure followed Starace et al. (2025).Source: §A.3 · p. 18

A.4 PAPERBENCH CODE-DEV EVALUATION

• For PaperBench Code-Dev evaluation, the coding agent was additionally prompted to generate the required reproduce.sh script.Source: §A.4 · p. 18

A.5 ADDITIONAL DETAILS ON EXECUTION AND REPRODUCIBILITY EXPERIMENTS

• Execution experiments used LLM-assisted debugging, while Table 10 reports average PaperBench Code-Dev replication scores and the reporting scheme.o4-mini was the primary debugger, with GPT-5 as fallback for persistent identical errors; runs used Docker with an RTX 2080 or, for larger-memory needs, an RTX A6000. Hardware constraints sometimes required hyperparameter adjustments or rare training-data subsampling. PaperCoder results average three runs except for o1 and o3 due to costs.Source: §A.5 · p. 18, Table 10 · p. 19

B.1 CODE AVAILABILITY

• 19.5% of accepted ICLR, ICML, and NeurIPS papers in 2024 provided official code, based on GitHub URLs in their ArXiv abstracts.The estimate uses accepted-paper lists from OpenReview and corresponding ArXiv abstracts retrieved through Semantic Scholar.Source: §B.1 · p. 19, Table 9 · p. 19

• Human execution fixes are overwhelmingly minor syntax, import, or variable-name adjustments rather than logic- or architecture-level revisions, and require little developer time.Figures 8–12 show the fix statistics reported in Table 15; the developer-time estimate is reported in Table 11.Source: §B.2 · p. 19, Table 11 · p. 19
• Developer time is estimated by multiplying modified lines by a 1–3 difficulty factor: 1 for simple edits, 2 for local reasoning, and 3 for deeper debugging.The categories range from syntax, typo, renaming, and comment fixes to condition or API adjustments and asynchronous or memory-related debugging.Source: §B.2 · p. 19
• LLM-assisted repair resolves all issues mostly within a small number of iterations, with errors primarily syntactic or import-related rather than logic-level.The repair stage uses o4-mini-high and GPT-5, recording iteration counts and error categories identified using GPT-5.1.Source: §B.2 · p. 20

B.3 PAPERBENCH CODE-DEV RESULTS

• 60.86% was PaperCoder’s highest replication score, achieved with o3 across the reasoning models tested.Source: §B.3 · p. 20
• The authors report that newer, larger models with stronger reasoning and coding capabilities tend to yield better performance.Source: §B.3 · p. 20

B.4 IMPACT OF PAPER CONTENT ON CODE GENERATION

• 4.26 to 3.75: the average score fell when the Methodology section was removed from 30 papers, indicating lower generated-code quality without detailed specifications.Source: §B.4 · p. 20, Table 12 · p. 20

B.5 MOST COMMON TYPES OF ERRORS AND FAILURE MODES

• MissingDependency, ImportError, and ModuleNotFoundError were the most frequent error categories, suggesting environment and packaging issues outweigh algorithmic or logic errors in practice.Source: §B.5 · p. 20, Table 13 · p. 20

B.6 ADDRESSING ENVIRONMENT SETUP

• PaperCoder prioritizes faithfully reconstructing methodological pipelines over automating paper-specific environment configuration, which users can often adjust manually.Source: §B.6 · p. 20
• In an experiment on 30 papers, prompts to infer and repair environment requirements eliminated dependency-related failures, suggesting these failures are not intrinsic framework weaknesses.Source: §B.6 · p. 20
• The authors identify a dedicated environment-construction or DevOps agent as a possible future extension.Source: §B.6 · p. 20

B.7 ANALYSIS OF PERFORMANCE ACROSS PAPER CATEGORIES

• Across 90 papers, reference-based category scores ranged from 3.38 to 4.21, with a maximum gap of about 0.83.Figure 13 reports average reference-based scores per category.Source: §B.7 · p. 21
• Theory scored 4.21 and interpretability/explainability 3.97, while reinforcement learning/control and dataset-focused papers each scored 3.38.The results indicate measurable performance variation across paper categories.Source: §B.7 · p. 21

B.8 DISCUSSION AGAINST DATA CONTAMINATION IN PAPER2CODEBENCH

• The evaluation targets ICLR/ICML/NeurIPS 2024 papers, which postdate the October 2023 knowledge cutoff of OpenAI o3-mini and GPT-4o.Although indirect exposure cannot be fully ruled out, this temporal gap substantially reduces contamination likelihood, which the authors say is unlikely to meaningfully affect results.Source: §B.8 · p. 21

B.9 CROSS-FAMILY VALIDATION FOR PAPER2CODEBENCH MODEL EVALUATION

• On 30 randomly selected papers, Gemini-Flash 2.5 and o3-mini-high had a Spearman correlation of 0.73, indicating consistent cross-family evaluation.This provides evidence that the results are not an artifact of model-family alignment.Source: §B.9 · p. 21
• o3-mini-high correlated with human judgments at 0.78 in the reference-based setting and 0.73 in the reference-free setting.The authors interpret these correlations as suggesting LLM-based evaluation could be a reliable proxy for human assessments.Source: §B.9 · p. 21
• Because full-scale human evaluation across all baselines and papers would be prohibitively expensive (Zheng et al., 2023), the study uses LLM-as-a-judge for primary assessment.The authors also investigate possible circularity or bias from an evaluator favoring code generated by its own model family.Source: §B.9 · p. 21

C LIMITATIONS AND FUTURE WORK

• PaperCoder’s current scope is limited to machine learning papers, where code implementations are especially helpful and usually necessary for validating research ideas.The authors identify reproduction in domains where code is not the primary validation medium, such as theoretical mathematics, as a future direction.Source: C LIMITATIONS AND FUTURE WORK · p. 21
• PaperCoder processes only textual inputs; extending it to figures and tables using OCR models is identified as an avenue for future work.The passage cites recent OCR work by Wang et al. (2024a), He et al. (2024), Wei et al. (2025), Li et al. (2025), Zhang et al. (2025), and Niu et al. (2025).Source: C LIMITATIONS AND FUTURE WORK · p. 21
• Improving executability remains an important but challenging future direction for PaperCoder and other repository-level code-generation approaches.Source: C LIMITATIONS AND FUTURE WORK · p. 21

D PROMPTS

• Planning prompts generate an overall plan, then architecture and logic designs, followed by a configuration file.The architecture prompt follows the overall-plan prompt, and the logic-design prompt follows the architecture prompt.Source: Figure 14 · p. 28, Figure 15 · p. 29, Figure 16 · p. 30
• Analysis and coding prompts are applied to each file in logic-design order, with previously generated code supplied during coding.The coding prompt accumulates earlier generated code files as input.Source: Figure 18 · p. 32, Figure 19 · p. 33
• Reference-based evaluation scores target-repository correctness from 1 to 5 against a gold repository and requires critiques of implementation mismatches.The review focuses on core logic and methodology rather than minor logging, organization, or documentation details.Source: D PROMPTS · p. 34
• Reference-based critique severity distinguishes high-impact missing core components, medium-impact functionality problems, and lower-impact deviations that can be worked around.Evaluation-process errors are low severity unless they affect core concepts.Source: D PROMPTS · p. 34
• Reference-free evaluation also scores repository correctness from 1 to 5, assessing completeness and accuracy against the paper and requiring critiques tied to affected files and functions.The rubric considers critical components such as preprocessing, core algorithms, and evaluation steps.Source: D PROMPTS · p. 36
• Architecture verification rates fidelity to the paper across the implementation approach, file list, class-diagram interfaces, and sequence-diagram call flow.Its indicators include faithful algorithmic and reproducibility details, modular file responsibilities, paper-consistent interfaces, and required control-flow and edge handling.Source: D PROMPTS · p. 41
• Logic-design verification checks whether the ordered file and module generation sequence, dependencies, and build or run scripts match what the paper requires.Source: D PROMPTS · p. 44
• The logic-design review checks pipeline coverage, dependency order, module boundaries, reproducibility hooks, and file-level data flow, then assigns a 1–5 score.Its rubric ranges from an impossible build or missing essentials to coherent generation with reproducible builds and experiment hooks.Source: D PROMPTS · p. 44
• The logic-design refinement prompt uses critique to fix dependencies, missing files, and API gaps while aligning the task list with architecture and executability.It asks for a summary of changes and treats logic analysis as both a dependency map and implementation guidance.Source: D PROMPTS · p. 46
• The configuration review checks paper and plan alignment, required settings, and naming consistency, then scores fidelity and critiques omissions by severity.Completeness checks include dataset paths and preprocessing, model hyperparameters, training and evaluation settings, ablation toggles, and random seeds.Source: D PROMPTS · p. 48
• The analysis-file review assesses whether responsibilities and workflows align with the paper and planning artifacts, cover implementation needs, and are clear enough to follow.It directs reviewers to use only explicitly supplied config values and check orchestration, data, model, training, evaluation, configuration, and utility workflows.Source: D PROMPTS · p. 51

E EXAMPLES OUTPUT OF THE PLANNING PHASE

• Transformer planning examples show outputs for the overall plan and architecture design, including class-diagram and program-call-flow visualizations.Source: Figure 34 · p. 55, Figure 35 · p. 56, Figure 36 · p. 57
• The Transformer examples also include planning outputs for logic design and the configuration file.Source: Figure 39 · p. 59, Figure 40 · p. 59
• Figure 43 is labeled as the third part of the Human Evaluation Guideline.It appears under “E EXAMPLES OUTPUT OF THE PLANNING PHASE.”Source: Figure 43 · p. 66