Source-linked AI summary
Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining
Zhiyuan Zeng, Yichi Zhang, Yong Shan, Kai Hua, Siyuan Fang, Zhaiyu Liu, Jiaheng Liu, Haozhe Wang, Yining Zheng, Ming Ding, Ke Shen, Ge Zhang, Wenhao Huang, Xipeng Qiu
TL;DR
Complex software repositories expose final code while omitting the long-horizon reasoning used to create it. The paper reconstructs repository-development trajectories through grounded multi-agent simulation and search-based CoT refinement, then uses them for continual pre-training. Llama-3-8B shows significant gains across long-context understanding, coding, reasoning, and agentic benchmarks.
Problem
Static repositories represent only the terminal state of software development, omitting intermediate planning, debugging, and refinement needed for complex long-horizon reasoning.
Method
The framework synthesizes repository-development trajectories with grounded multi-agent simulation and iteratively refines their CoT reasoning against ground-truth code.
Results
Llama-3-8B continuously pre-trained on reconstructed trajectories achieves significant gains across long-context understanding, coding, reasoning, and agentic benchmarks.
Takeaways & Limitations
Reconstructed development processes provide a reasoning-dense pre-training signal for software repositories beyond final code alone.
Takeaways & Limitations
All model variants use a fixed budget of 12% of 20B total tokens, so comparisons control training-token volume despite differing sample lengths.
Abstract
from arXiv · showhide
While Large Language Models (LLMs) have achieved remarkable success in code generation, they often struggle with the deep, long-horizon reasoning required for complex software engineering. We attribute this limitation to the nature of standard pre-training data: static software repositories represent only the terminal state of an intricate intellectual process, abstracting away the intermediate planning, debugging, and iterative refinement. To bridge this gap, we propose a novel paradigm: understanding via reconstruction. We hypothesize that reverse-engineering the latent agentic trajectories -- the planning, reasoning, and debugging steps -- behind static repositories provides a far richer supervision signal than raw code alone. To operationalize this, we introduce a framework that synthesizes these trajectories using a multi-agent simulation. This process is grounded in the structural realities of the source repositories (e.g., dependency graphs and file hierarchies) to ensure fidelity. Furthermore, to guarantee the logical rigor of the synthetic data, we employ a search-based optimization technique that iteratively refines the Chain-of-Thought (CoT) reasoning to maximize the likelihood of the ground-truth code. Empirical results demonstrate that continuous pre-training on these reconstructed trajectories significantly enhances Llama-3-8B's performance across diverse benchmarks, including long-context understanding, coding proficiency, and agentic capabilities.
1 Introduction
Static repositories omit the planning, debugging, and refinement underlying complex software, limiting long-horizon reasoning from code alone. The paper reconstructs these processes with grounded multi-agent simulation and search-refined reasoning, improving Llama-3-8B across diverse benchmarks.
- Static repositories encode only the terminal state of software development, omitting requirement analysis, planning, debugging, and iterative refinement.
- A multi-agent framework simulates high-level planning and file-level implementation while grounding generation in repository hierarchies and dependency graphs.
- Search-based optimization refines Chain-of-Thought steps to maximize the likelihood of ground-truth code, producing more causally complete and logically rigorous trajectories.
- Continuous pre-training Llama-3-8B on reconstructed trajectories improves performance across long-context understanding, coding, reasoning, and agentic benchmarks.
- The paper proposes understanding via reconstruction: expanding repositories into dynamic sequences of planning, reasoning, and execution.
2 Related Work
Prior synthetic-trajectory methods trade authenticity and cost against hallucination risk. This work instead reconstructs holistic repository-development trajectories, covering planning, actions, tool interactions, and evolving states.
- Unlike approaches recovering isolated reasoning steps, the framework reconstructs holistic trajectories integrating architectural planning, file-level actions, and iterative tool use.
- Existing trajectory construction uses real-world exploration for authenticity or LLM simulation for lower cost, with the latter vulnerable to hallucinations.
- The proposed method generates tool calls and outcomes with an LLM while anchoring the terminal state to a real repository as ground truth.
- Compared with prior code-synthesis datasets, this work augments entire repositories and models actions, tool interactions, and evolving development states.
3 Approach
The approach simulates repository development with coordinated agents, grounds synthetic interactions in extracted repository structure, and optimizes reasoning against ground-truth code. The resulting trajectories are flattened and selectively trained for continual pre-training.
- The pipeline has two stages: Multi-Agent Trajectory Curation and LongCoT Optimization.
- Multi-Agent Trajectory Curation: A Main Agent creates requirements and a dependency-aware file plan, then delegates each file to a specialized Sub-Agent.
- Multi-Agent Trajectory Curation: Each Sub-Agent plans a file, reads already implemented files for context, and writes the file code; thoughts, tool calls, and responses form one trajectory.
- Grounding the Simulation: Repository grounding supplies file trees, inter-file dependency graphs, and intra-file AST structure, while replacing Read and Write outputs with factual repository content.
- CoT Optimization via Search: Inference-time search samples CoT refinements, evaluates ground-truth-code perplexity, and permanently updates steps when perplexity decreases.
- Data Preparation: The data are flattened into chronological documents, with Observation tokens masked so training predicts Think and Action tokens rather than feedback.
4 Experiments
The experiments compare raw repository code with synthetic agentic trajectories, including search-optimized variants, across long-context, coding, reasoning, and software-engineering benchmarks. Reconstructed trajectories generally outperform flattened code, with optimization providing additional gains in several tasks.
- Experiment Setup: All models are continually pre-trained for 20B tokens with a 64k context window and the same 70% general-domain, 30% repository-related mixture.The repository-related portion includes a fixed 18% Prolong Repos component and a 12% experimental slot.
- Long-Context Understanding: Repo2Agent and Repo2Agent-Search consistently outperform Raw-Repos across tested Ruler context lengths.The advantage is reported at 16k, 32k, and 64k context lengths.
- Long-Context Understanding: 62.65 at 32k is Repo2Agent-Search’s Helmet average, versus 60.98 for Raw-Repo.At 16k and 32k, Repo2Agent-Search achieves peak performance; at 64k it still exceeds the primary Raw-Repo ablation.
- Coding and Reasoning: 37.20 on HumanEval is Repo2Agent-Search’s score, versus 34.76 for Raw-Repos; it also leads LongCodeBench-32k with 36.46.These results are presented as evidence that planning and refinement improve code generation over memorizing static files.
- Coding and Reasoning: Repo2Agent-Search yields the best MATH results, while the models match or slightly exceed baselines on BBH and AGI-Eval.The paper notes that absolute MATH scores remain low across models, reflecting Llama-3-8B’s limitations on complex mathematics.
- Software-Engineering Capability: Repo2Agent reaches 34.84% on Issue-Fix, while Repo2Agent-Search reaches 21.61% on Env-Setup and 24.49% on Error diagnosis.The authors describe this as a tunable balance between broad planning and logical precision.
5 Analysis on Synthetic Data
The analysis shows that reconstruction expands repositories into substantially longer, more explicit trajectories, while search optimization deepens the reasoning trace. More optimization iterations lengthen CoT and reduce the perplexity of the target code.
- Token Distribution: Sub-agent Tool-Calls and Responses contain most trajectory tokens, reflecting the detailed implementation process.The figure analysis identifies sub-agent activity as the primary concentration of tokens.
- Token Distribution: 2,300 Sub-Agent-Call-Think tokens in Repo2Agent-Search exceed the 900 tokens in Repo2Agent.The authors interpret this increase as substantial elaboration of implementation logic through search optimization.
- Information Expansion: 12,083.4 tokens is Repo2Agent-Search’s average per-repository length, versus 4,865.5 tokens for raw code.The expansion makes latent planning and execution steps explicit.
- Information Expansion: All variants use a fixed 12% of the 20B-token training budget for experimental data, separating trajectory quality from total training volume.The fixed budget is used to make the comparison fair despite longer synthetic samples.
- CoT Optimization: CoT length increases with optimization iterations, while target-code PPL steadily decreases over the same iterations.The relationship is evaluated on 100 sample trajectories over 10 iterations.
6 Conclusion
The paper concludes that understanding via reconstruction addresses limitations of static software artifacts by converting repositories into dynamic, causally rich training data. Llama-3-8B trained on these reconstructed processes shows enhanced coding, reasoning, and agentic capabilities.
- Conclusion: The proposed paradigm reverse-engineers latent agentic trajectories through grounded multi-agent simulation and search-based reasoning refinement.This transforms static repositories into dynamic training data with explicit process structure.
- Conclusion: Llama-3-8B experiments show enhanced coding, reasoning, and agentic capabilities from learning on reconstructed processes.The conclusion states the result without restricting it to a single benchmark.
Appendix
The appendix describes Ruler score aggregation across multi-key, multi-value, multi-query, and question-answering subtasks.
- Ruler: Ruler reports NIAH-Multi-Multi as the average of multi-key, multi-value, and multi-query subtasks.RULER-QA averages the Hotpot and Squad question-answering subtasks.
A Detailed Results on Long-Context Benchmarks
Agent-based reconstructed training data consistently outperforms raw-code training on long-context benchmarks, with the strongest gains appearing in complex retrieval and at extended context lengths.
- 61.80: Repo2Agent-Search achieves the highest average score at 64k, versus 61.00 for Raw-Repo and 57.10 for Prolong.
- 80.40: Repo2Agent-Search outperforms Raw-Repo’s 76.30 and Prolong’s 66.20 on 64k NIAH-Multi retrieval.
- 62.65 versus 60.98: at 32k on Helmet, Repo2Agent-Search scores higher than raw-code training.
- 99.81 at 32k and 96.00 at 64k: Repo2Agent-Search outperforms both baselines on Helmet Recall tasks.
- 57.84 versus 57.13: the method remains competitive with Prolong’s 58.10 overall average while exceeding the controlled Raw-Repo baseline.
- Repo2Agent methods consistently outperform the internal Raw-Repo baseline across tested long-context benchmarks.
C Case Study: Evolution of CoT
The case study shows search optimization transforming an initially generic CoT into a more detailed, technically justified, and project-aware implementation blueprint.
- Original CoT: The original CoT is functional but generic, listing necessary steps while providing little detail about how or why they should be performed.
- First-round optimization: After one optimization round, the CoT becomes more specific by checking both authentication_status and status_2FA in session state.
- First-round optimization: The first-round CoT adds technical rationale for one-time resource initialization and a 5-minute data-cache TTL.
- Second-round optimization: After two rounds, the CoT becomes a comprehensive implementation blueprint covering authentication, configuration, database connection, and access control.
- Second-round optimization: The optimized reasoning adds explicit requirement rationale, error handling, and implementation details such as warning users and preventing further execution.
- Second-round optimization: The final reasoning broadens from a single-file checklist to project-aware concerns including environment loading, modularity, readability, and Streamlit best practices.
D Prompt
The prompts construct synthetic repository trajectories by coordinating a planning main agent with file-level sub-agents, tools, repository context, and optimized reasoning instructions.
- Main-agent prompt: The main-agent prompt supplies repository requirements, tree structure, file paths, and file-specific requirements for planning and delegation.
- Main-agent prompt: The main agent decomposes the repository into an implementation order and calls sub-agents to generate individual files.
- Sub-agent prompt: The simulated memory is required to chronicle creating a file from scratch rather than explaining or refactoring existing code.
- Sub-agent prompt: Sub-agents receive repository and file context, then use Read to inspect dependencies and Write to generate source code.
- Reasoning instructions: The prompt requires context-driven, specific reasoning that mentions actual names, dependency interfaces, edge cases, and implementation decisions.
- CoT optimization prompt: CoT optimization rewrites a weak target block to make reasoning more precise and solution-aligned while preserving narrative flow and first-person derivation.