Source-linked AI summary
CodePlan: Repository-level Coding using LLMs and Planning
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, Shashank Shet
TL;DR
Repository-level coding requires pervasive, inter-dependent edits that localized LLM coding tools cannot directly handle within limited context. CodePlan addresses this by planning a chain of LLM-driven edits using dependency and may-impact analyses with adaptive execution. It achieved better ground-truth matching and validity checks on 5/6 repositories, while baselines passed none.
Problem
Repository-level coding tasks require coordinated edits across inter-dependent repositories, exceeding the localized scope and context capacity of existing LLM coding tools.
Method
CodePlan synthesizes and executes a multi-step edit plan using incremental dependency analysis, change may-impact analysis, and adaptive planning.
Results
5/6 repositories passed validity checks with CodePlan, compared with 0 repositories for baselines, and CodePlan better matched the ground truth.
Takeaways & Limitations
Planning enables CodePlan to coordinate repository-wide LLM edits across the evaluated C# migration and Python temporal-edit tasks.
Takeaways & Limitations
CodePlan primarily handles static-analysis relations, while dynamically typed code and dynamic dependencies remain challenging.
Abstract
from arXiv · showhide
Software engineering activities such as package migration, fixing errors reports from static analysis or testing, and adding type annotations or other specifications to a codebase, involve pervasively editing the entire repository of code. We formulate these activities as repository-level coding tasks. Recent tools like GitHub Copilot, which are powered by Large Language Models (LLMs), have succeeded in offering high-quality solutions to localized coding problems. Repository-level coding tasks are more involved and cannot be solved directly using LLMs, since code within a repository is inter-dependent and the entire repository may be too large to fit into the prompt. We frame repository-level coding as a planning problem and present a task-agnostic framework, called CodePlan to solve it. CodePlan synthesizes a multi-step chain of edits (plan), where each step results in a call to an LLM on a code location with context derived from the entire repository, previous code changes and task-specific instructions. CodePlan is based on a novel combination of an incremental dependency analysis, a change may-impact analysis and an adaptive planning algorithm. We evaluate the effectiveness of CodePlan on two repository-level tasks: package migration (C#) and temporal code edits (Python). Each task is evaluated on multiple code repositories, each of which requires inter-dependent changes to many files (between 2-97 files). Coding tasks of this level of complexity have not been automated using LLMs before. Our results show that CodePlan has better match with the ground truth compared to baselines. CodePlan is able to get 5/6 repositories to pass the validity checks (e.g., to build without errors and make correct code edits) whereas the baselines (without planning but with the same type of contextual information as CodePlan) cannot get any of the repositories to pass them.
1 INTRODUCTION
Repository-level coding tasks require coordinated edits across inter-dependent code, beyond the localized changes handled by current LLM coding tools. CodePlan frames these tasks as planning, synthesizing dependent edits and outperforming non-planning baselines on validity checks.
- Problem Formulation: Repository-level tasks include migrations, static-analysis or testing fixes, refactoring, and specification additions that propagate edit requirements across dependencies.Such propagation is typically performed manually.
- Problem Formulation: CodePlan defines repository-level coding as reaching a target repository state that satisfies a correctness oracle after executing seed and derived edit specifications.The oracle may enforce building, static analysis, typing, testing, or verification conditions.
- Proposed Solution: CodePlan synthesizes a multi-step plan whose graph links dependent code-edit obligations and adaptively extends as monitored changes reveal further obligations.Its derived specifications combine incremental dependency analysis, change may-impact analysis, and adaptive planning.
- Proposed Solution: In the API-migration example, a changed method signature identifies callers such as process as affected, enabling a derived edit that restores a repository that builds without errors.The same mechanism handles transitive propagation beyond the one-hop example.
- Contributions: The paper identifies systematic planning of inter-dependent repository edits and monitoring their effects as a previously unsolved problem.It presents repository-level coding formalization and a task-agnostic planning framework as contributions.
2 DESIGN
CodePlan treats repository-level coding as an iterative planning problem: it tracks dependencies, uses LLMs to discharge edit obligations, and adaptively propagates changes until an oracle is satisfied.
- 2.1 The CodePlan Algorithm: CodePlan takes a repository, seed edit specifications, a correctness oracle, and an LLM as inputs.
- 2.1 The CodePlan Algorithm: The algorithm maintains a dependency graph of syntactic and semantic code relationships and a directed acyclic plan graph of pending or completed edit obligations.Plan-graph edges record dependency relations that caused target obligations to be created.
- 2.1 The CodePlan Algorithm: After plan execution, the oracle supplies error locations and diagnostics as seed specifications for another round, continuing until the repository passes its checks.
- 2.1 The CodePlan Algorithm: For each pending obligation, CodePlan extracts a code fragment, gathers spatial and temporal context, invokes the LLM, merges the result, and updates repository dependencies.
- 2.3 Adaptive Planning and Plan Execution: Adaptive planning classifies each change and traverses relevant dependencies to create obligations for affected blocks, such as callers after a method-signature change.
- 2.2 Static Analysis: Static analysis is used to assess edit impact beyond the local LLM call through incremental dependency analysis.
3 IMPLEMENTATION
CodePlan implements repository-wide dependency tracking and localized LLM editing for C# and Python, with language-specific analyses supplying the dependency relations.
- 3 IMPLEMENTATION: The implementation parses repository files with tree-sitter to produce AST-like structures and identify classes, methods, imports, and expressions.The representation supports locating code blocks at multiple syntactic levels.
- 3 IMPLEMENTATION: C# dependency edges are identified from AST relationships, including caller-callee, overrides-overridden, and base-class-derived-class connections.
- 3 IMPLEMENTATION: Python dependency edges are identified with Jedi, which discovers symbol references and declarations across the codebase.
- 3 IMPLEMENTATION: GPT-4 receives temporal context, spatial context, and structured code snippets containing the edit site to generate localized edits.
- 3 IMPLEMENTATION: CodePlan currently supports C# and Python, and extending it to other languages primarily requires constructing the corresponding dependency graph relations.
4 EXPERIMENTAL DESIGN
The experiments evaluate CodePlan on diverse migration and temporal-edit repositories using oracle-based validity checks, reactive repair baselines, alternate edit models, and block-level metrics.
- 4.1 Datasets: The evaluation covers C# migration and Python temporal-edit tasks across internal proprietary and external public GitHub repositories.
- 4.1 Datasets: Source, target, and predicted repositories are compared to identify seed changes and evaluate whether generated edits match the post-commit repository.
- 4.2 Oracles and Baselines: The C# oracle is error-free compilation, while the temporal-edits oracle is Pyright static checking for Python.
- 4.2 Oracles and Baselines: Oracle-Guided Repair applies seed edits, invokes the oracle, analyzes diagnostics, and asks an LLM to patch reported locations.
- 4.2 Oracles and Baselines: CodePlan is also evaluated with Coeditor as an alternate localized edit model to test whether its planning and analysis generalize beyond GPT-4.
- 4.3 Evaluation Metrics: Matched, missed, and spurious blocks measure correct edits, omitted target edits, and unnecessary predicted edits, respectively.
5 RESULTS AND ANALYSIS
CodePlan outperforms oracle-guided repair baselines on repository-level coding tasks by planning derived edits, incorporating repository context, and iteratively refining LLM-generated changes. Across C# migration and Python temporal-edit evaluations, it more accurately identifies affected locations and produces more complete, precise repository modifications.
- C# Migration: CodePlan achieves more accurate repository edits than Build-Repair by avoiding missed and spurious changes in C# migration tasks.On internal repositories, CodePlan matched 151 blocks for Int-1 and 438 for Int-2, while Build-Repair matched 82 and 437, respectively, and introduced 13 and 25 spurious blocks.
- Multiple Iterations: CodePlan uses iterative refinement to address occasional LLM inaccuracies during repository editing.After erroneous first-iteration corrections, a subsequent iteration re-engages the LLM using identified build errors.
- C# Migration: On Ext-1, CodePlan updated 58 blocks with a perfect DiffBLEU score of 1.00, whereas Build-Repair missed six blocks and generated eight build errors.CodePlan’s change may-impact analysis identified constructor updates that did not trigger build errors but were required by the ground truth.
- Temporal Edits: CodePlan identifies more derived edit locations than Pyright-Repair on temporal edits, with higher DiffBLEU and lower Levenshtein Distance across repositories.Pyright-Repair identifies no derived edits on T-2 and T-3 because its oracle does not flag changes that may be required without violating type-checking rules.
- Coeditor Evaluation: CodePlan performs on par with Coeditor-CodePlan on T-1 but misses one edit site on each of T-2 and T-3.The comparison attributes lower alignment by Coeditor-CodePlan to differences in context-understanding abilities between gpt-4-32k and Coeditor.
- Results Discussion: CodePlan’s planning and contextual analysis support precise repository-wide changes, while oracle-guided repair can miss behaviorally necessary edits that produce no build or type-checking errors.The results attribute this difference to change may-impact analysis and the use of temporal and spatial context during planning.
6 LIMITATIONS AND THREATS TO VALIDITY
CodePlan’s validity and generality are constrained by dependency-analysis quality, limited repository coverage, LLM reliability, single-block updates, and incomplete artifact and prompting support.
- Dependency analysis: Dependency analysis is effective in statically typed languages but semantically rich relationships are harder to establish in dynamically typed languages without type hints.This limits CodePlan’s applicability to languages such as Python and JavaScript.
- Dependency analysis: The current implementation primarily models static code-block relations, leaving data flow, dynamic dispatch, algorithmic, and execution dependencies for future work.These dependencies include multithreading and distributed processing.
- Evaluation scope: Evaluation used multiple repositories across two tasks and two languages, but the number of repositories was restricted by the complexity of setting up each experiment.More experimentation is required to probe CodePlan’s strengths and weaknesses.
- Artifact coverage: Repository-level editing remains incomplete for configuration, metadata, project-setting files, external dependencies, and other non-code artifacts.A comprehensive approach would require dependency graphs covering these additional artifacts and relations.
- LLM reliability: CodePlan relies on LLM responses for change-may-impact analysis, so incorrect or spurious responses can produce erroneous repository updates.Multiple iterations sometimes rectify these issues, but may not always suffice.
- Future extensions: The update strategy handles one block at a time, although some scenarios may require or benefit from simultaneous multi-block changes.Few-shot prompting and Chain of Thought are also identified as possible extensions beyond the current zero-shot methodology.
7 RELATED WORK
Related work situates CodePlan at the intersection of LLM coding, online automated planning, incremental program analysis, contextualization, and learned edit patterns.
- LLMs for Coding Tasks: Prior LLM coding research covers program synthesis, repair, vulnerability patching, invariant inference, test generation, and multitask evaluation.These investigations generally use curated examples rather than repository-level coding tasks.
- Automated Planning: CodePlan treats repository editing as online planning because LLM edit effects are unpredictable and the state space is unbounded.It monitors edits and extends the plan through static dependency analysis.
- Analysis of Code Changes: Incremental analysis, program differencing, and change-impact analysis provide precedents for recomputing affected results and determining change effects across programs.Prior specialized techniques address dataflow, pointer, symbolic-execution, bug-detection, and type analyses.
- Spatial and Temporal Contextualization: CodePlan unifies spatial context from repository dependencies with temporal context from tracked code changes and past edits.This supplies LLMs with relevant information from other files and repository history.
- Learning Edit Patterns: Earlier approaches learn edit patterns from rewrite rules, bug fixes, type changes, API migrations, neural edit representations, examples, or IDE actions.These methods automate repetitive or context-aware edits but differ from CodePlan’s dependency-driven planning.
8 CONCLUSIONS AND FUTURE WORK
The paper concludes that CodePlan can orchestrate LLM-guided edits across complex, inter-dependent repositories, while identifying broader artifact, language, dependency, and task-specific extensions.
- Conclusions: CodePlan combines incremental dependency analysis, change may-impact analysis, and adaptive planning to coordinate multi-step LLM-guided repository edits.The evaluation covered internal and public C# and Python repositories for migration and temporal-edit tasks.
- Conclusions: The results show better ground-truth alignment than baseline methods and position CodePlan as a promising approach to repository-level coding automation.The conclusion associates this success with potential productivity and accuracy improvements in software engineering.
- Future Work: Future work will broaden support to more languages and artifacts, customize change-may-impact rules, and address dynamic dependencies.Targeted dependencies include data flow, dynamic dispatch, algorithmic assumptions, multithreading, and distributed processing.