PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepCode: Open Agentic Coding

Zongwei Li, Zhonghang Li, Zirui Guo, Xubin Ren, Chao Huang

arXiv:2512.07921v1cs.SEcs.AI

TL;DR

High-fidelity document-to-code synthesis remains difficult because information overload conflicts with LLM context limits. DeepCode manages information flow through four operations, and its PaperBench evaluation reports performance beyond commercial agents and human experts. The discussion also identifies constraints on training and evolving autonomous agents.

  • Problem

    High-fidelity document-to-repository synthesis remains challenging because detailed specifications compete with finite LLM context, while existing methods fall short of reliable end-to-end replication.

    Source: §1 · pp. 2–3

  • Method

    DeepCode manages repository synthesis through blueprint distillation, stateful code memory, retrieval-augmented knowledge injection, and closed-loop verification.

    Source: §1 · p. 3

  • Results

    Across PaperBench, DeepCode establishes state-of-the-art performance and surpasses human experts; on a 5-paper subset, its average score of 0.8482 exceeded Codex (0.3997), Cursor (0.5841), and Claude Code (0.5871).

    Source: §4.2 · p. 12, §7 · p. 15

  • Takeaways & Limitations

    The paper presents information-flow management as an approach to maximizing task-relevant signals under finite context budgets for repository generation.

    Source: §1 · p. 3

    Supervised fine-tuning of small models is constrained by the scarcity and curation cost of high-quality agentic trajectories.

    Source: §6 · p. 15

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have given rise to powerful coding agents, making it possible for code assistants to evolve into code engineers. However, existing methods still face significant challenges in achieving high-fidelity document-to-codebase synthesis--such as scientific papers to code--primarily due to a fundamental conflict between information overload and the context bottlenecks of LLMs. In this work, we introduce DeepCode, a fully autonomous framework that fundamentally addresses this challenge through principled information-flow management. By treating repository synthesis as a channel optimization problem, DeepCode seamlessly orchestrates four information operations to maximize task-relevant signals under finite context budgets: source compression via blueprint distillation, structured indexing using stateful code memory, conditional knowledge injection via retrieval-augmented generation, and closed-loop error correction. Extensive evaluations on the PaperBench benchmark demonstrate that DeepCode achieves state-of-the-art performance, decisively outperforming leading commercial agents such as Cursor and Claude Code, and crucially, surpassing PhD-level human experts from top institutes on key reproduction metrics. By systematically transforming paper specifications into production-grade implementations comparable to human expert quality, this work establishes new foundations for autonomous scientific reproduction that can accelerate research evaluation and discovery.

1 Introduction

High-fidelity paper-to-repository synthesis asks coding agents to translate complex scientific specifications into complete, executable implementations, but existing agents fall well short of human experts. DeepCode frames this as information-flow management and orchestrates four operations to address the challenge; the paper reports that it surpasses commercial agents and human experts on key reproduction metrics.

1 Introduction

  • Current agents remain below human-level paper reproduction: PaperBench reports 42.4% for o1 with IterativeAgent versus 72.4% for human experts [7].PaperCoder reaches a 51.14% reproduction rate on PaperBench [8].Source: §1 · p. 2
  • Paper-to-repository synthesis requires preserving scattered multimodal constraints, maintaining cross-file consistency, filling underspecified design choices, and producing executable code.Long-horizon generation can yield logic bugs, dependency conflicts, and fragile pipelines that prevent end-to-end execution.Source: §1 · p. 2
  • DeepCode treats repository synthesis as hierarchical information-flow management, combining blueprint distillation, structured code memory, retrieval-augmented knowledge injection, and closed-loop verification.These operations compress source specifications, index repository state, supply implementation patterns, and use execution feedback to correct errors.Source: §1 · p. 3
  • The paper characterizes the core conflict as information overload versus finite LLM context, proposing that systems structure, route, and compress information to maximize task-relevant signal.Naively concatenating documents and code history can saturate context, allowing redundant tokens to mask algorithmic constraints.Source: §1 · p. 3
  • DeepCode achieves state-of-the-art PaperBench performance, outperforming commercial agents and surpassing human experts on key reproduction metrics.The figure contrasts current agents’ 42% replication score with 72% for human experts, then depicts DeepCode as surpassing human performance.Source: Figure 2 · p. 2, §1 · p. 3

2 Preliminary

The preliminary section defines synthesis as turning a specification document into a complete repository, including scientific-paper reproduction and software-system generation. It then sets four objectives—preserving specifications, maintaining consistency, grounding underspecified designs, and ensuring executability—for generation under context limits.

2.1 Task Definition

  • Scientific-paper reproduction maps a paper to code for replicating its key experiments and results, while software-system generation maps design documents or natural-language requirements to multi-component repositories.The source may contain text, equations, tables, figures, and pseudocode, and its length can challenge finite context windows.Source: §2.1 · pp. 3–4
  • The target is a structured repository comprising a directory structure, interacting source files, and a dependency manifest, rather than a single code file.Cross-file interactions such as imports and function calls make coherent generation non-trivial.Source: §2.1 · p. 4

2.2 Objectives

  • The four objectives are to preserve multimodal algorithmic constraints, maintain module and type consistency, resolve underspecified designs with external patterns, and use runtime feedback to reduce bugs.The framework routes and compresses information to support these objectives under strict context-window constraints.Source: §2.2 · p. 4

3 The DeepCode Framework

DeepCode structures paper-to-repository synthesis as three phases: blueprint generation, code generation, and automated verification. It combines document distillation, selective context management and retrieval, and iterative correction to support implementation across files.

3 The DeepCode Framework

  • DeepCode divides repository synthesis into blueprint generation, code generation, and automated verification, using information-flow management to preserve task-relevant signals under context limits.The framework treats runtime feedback as a corrective signal during verification.Source: §3 · p. 4

3.1.1 Hierarchical Content Segmentation

  • Hierarchical segmentation indexes document chunks by section headings, enabling agents to retrieve relevant passages on demand rather than process the full document at once.This is intended to reduce token load and context overload.Source: §3.1.1 · p. 5

3.1.2 Multi-Agent Specification Analysis

  • The Concept Agent maps document structure and reproduction requirements, while the Algorithm Agent extracts implementation-level details such as equations, architectures, and hyperparameters.Their outputs provide complementary conceptual and technical specifications.Source: §3.1.2 · p. 6

3.1.3 Synthesizing the Implementation Blueprint

  • The Code Planning Agent reconciles conceptual and technical analyses into a self-contained blueprint covering project structure, component specifications, verification, environment, and development stages.It can query the indexed document to resolve inconsistencies.Source: §3.1.3 · p. 6

3.2.1 Stateful Generation with CodeMem

  • CodeMem builds a compressed repository representation: each file is generated using the blueprint and relevant dependency summaries, then summarized into memory to support cross-file consistency without context growing with repository size.The generation context is cleared before the newly generated file is summarized and added to memory.Source: §3.2.1 · p. 7

3.2.2 Knowledge Grounding with CodeRAG

  • CodeRAG indexes relevant source files and maps them to target blueprint files; during generation, retrieval can inject actionable implementation patterns to address implicit knowledge gaps.Retrieval is adaptive to target-file complexity and blueprint detail.Source: §3.2.2 · p. 8

3.3.1 Static Analysis and Code Quality Refinement

  • Static analysis checks the repository against the blueprint for structural discrepancies and code-quality deficiencies, then applies targeted line-level fixes rather than rewriting whole files.The analysis report identifies issues for the Modification Agent to address.Source: §3.3.1 · p. 9

3.3.2 Sandbox Execution and Functional Correction

  • Sandbox verification provisions the specified environment, executes repository entry points, and iteratively patches errors until execution succeeds or the iteration limit is reached.The framework presents this closed-loop process as a way to check functional correctness against the specification.Source: §3.3.2 · p. 9

4 Experiments

The experiments evaluate DeepCode on PaperBench against LLM agents, scientific and commercial coding systems, and human experts, then examine model backbones and component ablations. The reported comparisons favor DeepCode overall, while the analyses show that backbone and component effects vary by task and model.

4.1 Experiments Settings

  • PaperBench Code-Dev evaluates reproduction of 20 research papers; SimpleJudge scores structural and functional criteria, and each paper’s score averages three trials.The weighted Replication Score aggregates binary leaf-criterion judgments through the rubric hierarchy.Source: §4.1 · pp. 9, 11

4.2 Main Results

  • 73.5±2.8 vs. 43.3±1.1: DeepCode outscored the best LLM-agent baseline, o1 with IterativeAgent, by 70% relatively.PaperCoder scored 51.1±1.4, over 22 points below DeepCode.Source: §4.2 · pp. 11–12
  • 0.8482 vs. 0.5871: DeepCode’s five-paper average exceeded Claude Code’s, despite using the same base model as Claude Code and Cursor.DeepCode also scored 0.3997 against Codex and 0.5841 against Cursor; Table 1 covers the five-paper comparison.Source: §4.2 · p. 12, Table 1 · p. 12
  • 75.9 ± 4.5 vs. 72.4: DeepCode’s average on the three-paper subset exceeded the human experts’ best-of-3 score.Figure 4 compares DeepCode with human experts and three other baseline categories.Source: Figure 4 · p. 11, §4.2 · p. 12

4.3 Analysis on Different LLMs

  • Across three tasks, Claude-4.5-Sonnet performs best or near-best overall, while GPT-5 leads on stay-on-topic (0.81 vs. 0.72).With architecture and tooling fixed, DeepSeek-R1 consistently underperforms; scores vary across backbones and tasks.Source: §4.3 · p. 12, Figure 5 · p. 13

4.4 Ablation Studies

  • CodeRAG yields up to 70% relative gain with Gemini-2.5-Flash, but negligible gains with Claude 4.5 Sonnet.The ablation reports Flash’s baseline performance ceiling as 0.35–0.38.Source: §4.4 · p. 13, Figure 6 · p. 13
  • CodeMem raises scores from 0.33–0.43 to 0.70–0.92 on three tasks by preserving dependencies that sliding-window eviction truncates.It also improves test-time-model-adaptation from 0.62 to 0.72 and all-in-one from 0.66 to 0.76.Source: §4.4 · p. 13
  • Verification is a final pass that removes small execution deviations after earlier phases achieve technical correctness.The reported ablation describes its improvement as modest.Source: §4.4 · p. 14

5 Related Work

Related work describes general coding agents through multi-agent architectures and specialized tools, while commercial systems also face context and repository-coherence challenges. Scientific coding agents target paper-to-code and other research workflows, but generation efficiency without sacrificing code quality remains open.

5.1 General Coding Agents

  • General coding agents use multi-agent organizational or refinement architectures and specialized tools; commercial systems support complex edits but face context-window and repository-coherence challenges.Examples include ChatDev, MetaGPT, AgentCoder, MapCoder, CodeAgent, and SWE-agent.Source: §5.1 · p. 14

5.2 Scientific Coding Agents

  • Scientific coding agents generate or refine research code through structured pipelines, execution-reflection cycles, or iterative validation, but efficiency without compromising code quality remains an open challenge.The surveyed applications include paper-to-repository reproduction, experimental code generation, algorithm discovery, and experiment automation.Source: §5.2 · p. 14

6 Discussion: Challenges and Future Directions

The discussion identifies three frontiers for moving coding agents from episodic tasks toward autonomous, evolving engineering. It outlines approaches for improving efficiency, accumulating reusable experience, and adapting plans during implementation.

Agentic Capability and Computational Efficiency

  • Agentic capability is constrained by the trade-off between costly, high-latency proprietary models and efficient open-weight models with weaker reasoning for autonomous decisions.Supervised fine-tuning is limited by scarce, costly agentic trajectories; retrieved knowledge may be irrelevant, and smaller models can struggle to integrate complex contexts.Source: §6 · p. 15
  • Hybrid architectures could pair large models for high-level reasoning with efficient small models for routine implementation.The discussion also proposes distilling knowledge from large models to reduce the data bottleneck.Source: §6 · p. 15

From Episodic to Evolving Agents

  • Agents reset between projects, while reward design for complex software tasks is difficult and raw-log memories can overwhelm context.RL-based optimization is also inapplicable to closed-source models whose parameters cannot be updated.Source: §6 · p. 15
  • Post-task reflection could distill execution traces into reusable skills or heuristics for later retrieval, avoiding context explosion.The discussion presents this as a scalable alternative to extensive manual annotation and training.Source: §6 · p. 15

Dynamic Planning and Adaptability

  • Linear Plan-then-Code workflows can become fragile when specifications evolve or implementation reveals constraints that make the initial blueprint stale.Agents may then be constrained to suboptimal workarounds or failure.Source: §6 · p. 15
  • Bidirectional planning could feed execution insights back into the high-level plan so agents can adapt to unforeseen implementation constraints.The discussion identifies this feedback mechanism as important for large-scale software development.Source: §6 · p. 15

7 Conclusion

The conclusion frames document-to-repository synthesis as information-flow management under finite context limits. It reports strong PaperBench results for DeepCode and argues that hierarchical information orchestration is more decisive than indiscriminate context scaling.

7 Conclusion

  • DeepCode treats document-to-repository synthesis as an information-flow management challenge under finite context bottlenecks.The conclusion describes the framework as an autonomous approach to agentic code engineering.Source: §7 · p. 15
  • Blueprint distillation, stateful memory, conditional knowledge injection, and closed-loop verification are orchestrated to maximize signal-to-noise ratio for long-horizon tasks.Source: §7 · p. 15
  • On PaperBench, DeepCode establishes a new state of the art, outperforming leading commercial agents and surpassing PhD-level human experts in reproduction accuracy.The conclusion presents hierarchical information orchestration, rather than indiscriminate context scaling, as the decisive path toward robust autonomous systems.Source: §7 · p. 15

A Appendix

The appendix supplements the main text with quantitative evaluations, application examples, architecture details, and documentation of DeepCode's integrated tools.

Complete Results

  • Complete Results reports quantitative comparisons across benchmark models and reproducibility analyses across papers and operational scenarios.Source: A Appendix · p. 18

DeepCode Application Cases

  • Application Cases presents visualizations of backend systems, web interfaces, and the Paper2Code research-reproduction workflow.Source: A Appendix · p. 18

DeepCode Sub-Agent Details

  • Sub-Agent Details explains the specialized sub-agents' roles, responsibilities, and coordination patterns.Source: A Appendix · p. 18

MCP Toolkit in DeepCode

  • The MCP Toolkit section documents external interfaces DeepCode uses to interact with repositories, documentation, and execution environments.Source: A Appendix · p. 18

A.1 Full Results

DeepCode's appendix results report higher reproduction scores than commercial and specialized systems, with competitive performance against human experts on a three-paper subset. Additional analyses describe cross-paper and repeated-run stability.

Full-benchmark and human comparison

  • 73.5 ± 2.8 average reproduction score on the full benchmark exceeds PaperCoder's 51.1 ± 1.4, while the three-paper subset score of 75.9 ± 4.5 exceeds human Best@3's 72.4.Source: §A.1 · p. 18

Commercial-agent comparison

  • 0.8482 average reproduction score gives DeepCode the highest result across five papers, more than 0.26 above the strongest competing system.The advantage is reported across all individual papers in the subset.Source: §A.1 · p. 18

Per-paper and repeated-run analysis

  • Across diverse targets, Claude 4.5 Sonnet–based DeepCode scores typically fall in the 0.7–0.9 range with relatively small standard errors and a tight distribution of average cost across papers.The passage interprets these results as evidence of cross-task generalization and stability across repeated runs.Source: §A.1 · p. 18

A.2 Use Cases for DeepCode

DeepCode’s use cases illustrate generated backend systems, web interfaces, and research-paper-to-code artifacts.

A.2 Use Cases for DeepCode

  • DeepCode scaffolds complex, data-driven interfaces from high-level specifications, potentially reducing boilerplate and accelerating server-side infrastructure deployment.Figure 7 shows generated backend system pages.Source: §A.2 · p. 18, Figure 7 · p. 20
  • Figure 8 illustrates generated Web UIs with data visualizations and interactive forms that support rapid prototyping and user-experience iteration.The UI complements backend generation and can shorten the path from concept to a functional product.Source: §A.2 · p. 19, Figure 8 · p. 20
  • Figure 9 presents Paper2Code’s inferred code structure and a concrete code sample, turning scientific language into structured, executable artifacts.The examples include an architectural blueprint with modular components and file organization.Source: §A.2 · pp. 19–20

A.3 Sub-Agents Details of DeepCode

DeepCode uses specialized agents and shared representations to plan, synthesize, and refine code while maintaining repository-level consistency.

A.3 Sub-Agents Details of DeepCode

  • Specialized agents with standardized interfaces transform shared representations across the software lifecycle while preserving global architectural and semantic consistency.The lifecycle spans requirement understanding, architectural planning, and code synthesis.Source: §A.3 · p. 20
  • Parallel conceptual and algorithmic analyses produce partial plans that the Code Analysis agent reconciles into a roadmap constraining downstream generation.The roadmap specifies module boundaries, interfaces, dependencies, priorities, and testing hooks.Source: §A.3 · p. 20
  • Figure 9 juxtaposes an inferred code structure with a concrete code sample.Source: Figure 9 · p. 21
  • During synthesis, retrieval supplies relevant code patterns while global code memory tracks symbols, dependencies, and project conventions.Code generation queries the memory before writing and updates it with new symbols and dependencies afterward, supporting repository-level consistency.Source: §A.3 · p. 21

A.4 MCP Tool Stack in DeepCode

DeepCode’s MCP tools separate information access, semantic processing, and environment manipulation into an end-to-end development loop.

A.4 MCP Tool Stack in DeepCode

  • MCP tools are grouped into Perception & Retrieval, Cognitive Processing, and Action & Execution categories.Table 5 summarizes the integrated tools in these three categories.Source: §A.4 · p. 21, Table 5 · p. 23
  • Perception and retrieval access web and local materials, while cognitive processing builds semantic indexes and context-compatible segments.These capabilities support queries over existing artifacts and long technical materials.Source: §A.4 · p. 21
  • The tools form a loop from information retrieval and organization to code changes whose effects can be observed through commands.File I/O, command execution, and Git operations are confined to the sandboxed, path-validated Action & Execution layer.Source: §A.4 · p. 21

Related papers

Loading 2512.07921v1…