PaperlayerView exact revision on arXiv

Source-linked AI summary

Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback

Zhangqian Bi, Yao Wan, Zheng Wang, Hongyu Zhang, Batu Guan, Fangxin Lu, Zili Zhang, Yulei Sui, Hai Jin, Xuanhua Shi

arXiv:2403.16792v3cs.CLcs.SE

TL;DR

LLM-generated code may miss project-specific information, while repository context can be too large to fit into prompts. COCOGEN uses compiler feedback and retrieved repository information to iteratively correct mismatches; experiments report over 80% relative pass rates on project-context-dependent functions and consistent gains over retrieval-augmented generation.

  • Problem

    Repository-level code generation can require project-specific APIs, classes, data structures, and type constraints that prompts do not supply, while full project context cannot fit in model input.

    Source: §1 · pp. 1–2

  • Method

    COCOGEN compiles generated code to identify context-related errors, retrieves relevant repository information, and iteratively refines the code until no errors are identified.

    Source: §1 · p. 2

  • Results

    Over 80% relative pass rates: COCOGEN improves repository-level generation over native LLM baselines and consistently enhances retrieval-augmented generation.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Project-context errors are more prevalent in project-level than function-level generation, and COCOGEN can fix compilation errors by retrieving relevant project context.

    Source: §7 · p. 9

    Successful compilation does not guarantee functional correctness or execution safety, so testing and manual review remain necessary.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown remarkable progress in automated code generation. Yet, LLM-generated code may contain errors in API usage, class, data structure, or missing project-specific information. As much of this project-specific context cannot fit into the prompts of LLMs, we must find ways to allow the model to explore the project-level code context. We present CoCoGen, a new code generation approach that uses compiler feedback to improve the LLM-generated code. CoCoGen first leverages static analysis to identify mismatches between the generated code and the project's context. It then iteratively aligns and fixes the identified errors using information extracted from the code repository. We integrate CoCoGen with two representative LLMs, i.e., GPT-3.5-Turbo and Code Llama (13B), and apply it to Python code generation. Experimental results show that CoCoGen significantly improves the vanilla LLMs by over 80% in generating code dependent on the project context and consistently outperforms the existing retrieval-based code generation baselines.

1 Introduction

Project-level code generation is challenging because prompts may omit repository-specific APIs, classes, data structures, and type constraints. COCOGEN uses compiler feedback and retrieved repository context to iteratively correct errors, improving performance by over 80% relative pass rates on project-context-dependent functions.

1 Introduction

  • Repository-level generation can require APIs, classes, data structures, and type constraints that task prompts and surrounding code do not provide.The paper notes that project context may also be too large to fit in a prompt.Source: §1 · pp. 1–2
  • In the motivating example, GPT-3.5-Turbo calls an unavailable reduce_noise method, while supplying denoising.denoise as project context fixes the mismatch.Carefully engineered prompts may resolve such errors, but users cannot always create them.Source: §1 · pp. 1–2
  • COCOGEN compiles generated code to detect context-related errors, retrieves relevant repository information, and iteratively refines the solution until no errors are identified.The approach post-processes model output rather than fitting the entire project code into the model input.Source: §1 · p. 2
  • COCOGEN is evaluated on CoderEval, a benchmark of project-context-dependent code generation tasks, using GPT-3.5-Turbo and Code Llama (13B).The paper also reports analysis of the method’s effectiveness and limitations.Source: §1 · p. 2
  • COCOGEN significantly improves repository-level code generation across different dependency levels, outperforming the baseline by over 80% relative pass rates for functions dependent on project-specific contexts; its iterative method consistently enhances vanilla retrieval-augmented generation.Experiments use GPT-3.5-Turbo and Code Llama (13B).Source: §1 · p. 2
  • The paper studies error distributions in self-contained and repository-level generation and introduces compiler-based iterative generation, verification, and retrieval.The study highlights the importance of precise, grounded program context for project-level code generation.Source: §1 · p. 2

2 Background

LLM code generation is bounded by model context windows, while repository-dependent code requires project-specific APIs, classes, data structures, and constraints. The section examines the resulting errors and how repository context can help address them.

2.1 LLM-based Code Generation

  • LLMs generate code token by token from a bounded context window, which does not encompass the entire software project codebase.Few-shot generation may also be conditioned on a fixed set of input-output exemplars.Source: §2.1 · pp. 2–3

2.2 Error Analysis in Code Generation

  • 31.6% with CodeT5 and 94.7% with Code Llama are reported on HumanEval, while prior work finds project-context-dependent code challenging for existing LLMs.The cited context-dependent cases include private APIs, classes, data structures, and type constraints.Source: §2.2 · p. 3
  • 64% of project-level errors are API errors, while 21% of function-level errors are UNDEF errors; runtime or functional errors account for 72% and 34%, respectively.Figure 2 presents error distributions for function-level and project-level generation.Source: Figure 2 · p. 3, §2.2 · p. 3
  • With 10 candidate solutions generated per task, Pass@10 was 53.57% for function-level tasks (850 solutions) and 39.73% for project-level tasks (1,450 solutions).Failed code was categorized into UNDEF, API, OBJECT, FUNC, and OTHER error types.Source: §2.2 · p. 3
  • Compiler and context feedback fix project-context errors and mitigate other compilation errors, with experiments reporting improved prediction accuracy.COCOGEN supplies relevant project context and feedback on error messages to the code LM.Source: §2.2 · p. 4

3 Methodology

COCOGEN extracts project-level code context and uses it to iteratively refine LLM-generated solutions with compiler feedback. Its workflow combines structural and semantic retrieval with compilation-based checking.

3.1 Overview

  • The workflow combines project-context extraction with iterative generation and evaluation to refine solutions toward alignment with the codebase.Source: §3.1 · p. 4

3.2 Project-Level Code Context Extraction

  • COCOGEN parses project files into ASTs and captures classes, functions, variables, and their parent-child relationships for qualified-name and semantic lookup.The captured context includes function semantics such as docstrings and relationships to enclosing classes and files.Source: §3.2 · p. 4

3.3 Retrieval-Augmented Code Generation

  • Structural search turns compiler feedback into an SQL query that retrieves relevant project context, such as implementations associated with a named class.Source: §3.3 · p. 5
  • Semantic search embeds a query, compares it with context-entry embeddings, and retrieves the top-n matches alongside structurally retrieved context.The initial query uses the task description; later searches use the error report and corresponding error line.Source: §3.3 · pp. 4–5

3.4 Refinement with Compiler Feedback

  • Compiler feedback guides retrieval and regeneration until no compiler error remains; COCOGEN does not address execution-time FUNC errors.Compilation errors account for 66% of project-level errors.Source: §3.4 · p. 5

4 Experimental Setup

The experiments evaluate COCOGEN with GPT-3.5-Turbo and Code Llama (13B) on project-dependent Python code generation. They compare it with direct generation and retrieval-based baselines using benchmark tasks and pass-rate metrics.

4.1 Models and Datasets

  • The Python CoderEval split has 230 samples across six context-dependency levels; the study focuses on 55 class-runnable, 68 file-runnable, and 23 project-runnable tasks.Source: §4.1 · p. 6

4.2 Baseline Methods

  • The comparison uses GPT-3.5-Turbo and Code Llama (13B), with Direct Generation, ReACC, and RepoCoder as baselines.RepoCoder iteratively retrieves snippets similar to previously generated code but does not use compiler feedback.Source: §4.2 · p. 6, §4.1 · p. 6

4.3 Evaluation Metrics

  • Pass@k evaluates functional correctness by unit tests, and Figure 5 presents Pass@10 for COCOGEN, RepoCoder, and No Feedback across three dependency levels.The study generates 20 solutions per task and reports k values of 1, 5, and 10.Source: §4.3 · p. 7, Figure 5 · p. 7

5 Results and Analysis

COCOGEN outperforms project-level code-generation baselines, and iterative refinement improves its performance. Ablations and error analysis show the value and limits of compiler feedback paired with repository context.

5.1 Overall Performance of COCOGEN

  • COCOGEN significantly outperforms other baselines on project-level code generation across CoderEval and CrossCodeEval, with a few Pass@1 exceptions.The authors attribute the exceptions to variation in generated solutions; contextual models also outperform the no-context direct model.Source: §5.1 · p. 7

5.2 Effectiveness of the Iterative Refinement

  • Performance improves as COCOGEN undergoes more iterations, supporting the effectiveness of iterative refinement with compiler feedback.The reported trend holds across different data splits.Source: §5.2 · p. 7

5.3 Ablation on Components

  • Compiler feedback improves accuracy, but not markedly by itself; combining it with project-specific context yields stable gains over RepoCoder’s semantic retrieval.The authors explain that highlighting compilation errors alone does not provide the related context needed to resolve project-level coding problems.Source: §5.3 · p. 7, Table 3 · p. 7

5.4 Error Analysis and Case Study

  • After one refinement iteration, COCOGEN reduces UNDEF errors from 5,133 to 1,042; RepoCoder fixes API and syntax errors but not UNDEF or OBJECT errors.The authors suggest RepoCoder’s lack of compiler feedback leaves it unaware of UNDEF and OBJECT errors.Source: §5.4 · p. 8
  • A runtime-error case stems from ambiguous task requirements and unfamiliarity with the Structure class format, despite its definition being available.The authors propose integrating documentation, web search results, and execution logs as future guidance for code generation.Source: §5.4 · p. 8, Figure 7 · p. 8

6 Related Work

Related work spans code-generation models, methods for retrieving or representing project-level context, and techniques that post-process generated code using feedback. The cited approaches range from historical language-modeling methods to retrieval and execution-based refinement.

LLM-based Code Generation

  • Code-generation research has progressed from rule-based systems and structured prediction to modern coding LLMs and commercial tools.The passage cites Woods (1973), Zelle and Mooney (1996), Zettlemoyer and Collins (2005), DeepSeek Coder (Bi et al., 2024), Code Llama (Rozière et al., 2023), Star-Coder (Li et al., 2023), GPT-3.5 (OpenAI, 2023), and GitHub Copilot.Source: §6 · p. 8

Project-level Code Generation

  • Project-level generation work tracks project changes, traverses relational databases, uses API documentation, filters repository prompts, or retrieves local and broader context.Examples include cache models, relational-database graph walking, API documentation (Zan et al., 2022; Zan et al., 2023), prompt filtering (Shrivastava et al., 2023), retrieval-augmented generation (Lu et al., 2022; Zhang et al., 2023a), and A3-CodGen (Liao et al., 2023).Source: §6 · pp. 8–9

Post-processing of LLMs for Code Generation

7 Conclusion

COCOGEN uses structurally and semantically indexed project context to address compiler-detected errors in LLM-generated code. The paper reports improved native LLM baselines, including over 80% relative pass rates for functions dependent on project-specific context.

  • COCOGEN uses structurally and semantically indexed project context to fix compilation errors and improve LLM-generated code quality.Source: §7 · p. 9
  • Project-context-related errors are more prevalent in project-level than function-level code generation.Source: §7 · p. 9
  • COCOGEN improves native LLM baselines by over 80% relative pass rates when generating functions dependent on project-specific context.Source: §7 · p. 9

Limitations

Compilation is one validation signal, but it cannot establish execution success, safety, or full functional correctness.

  • Successful compilation does not guarantee that generated programs execute successfully or are safe to run.Source: Limitations · p. 9
  • The authors call for additional testing and manual review to assess functional correctness.Source: Limitations · p. 9
  • Functional correctness also includes task requirements, preconditions and postconditions, security requirements, and robustness.Source: Limitations · p. 9

Ethics Statements

The paper reports adherence to open-access licensing and intellectual-property policies, and describes the two investigated models and their inference setup.

Ethics Statements

  • The authors state that the research uses freely available resources and complies with Creative Commons licensing and intellectual-property laws.Source: Ethics Statements · p. 9

A More Details on Investigated Large Language Models

B More Details on the Investigated Dataset

The appendix describes three Python benchmarks, evaluation metrics for code generation and line completion, and the static-analysis errors examined in CoderEval-Python.

B More Details on the Investigated Dataset

C More Details on the Evaluation Metrics

D Errors Reported by the Compiler

E More Details About the Error Cases

Compiler feedback improves repository-level generation and completion, but examples show that compilation does not catch every flawed solution and that some errors remain unrepaired.

E More Details About the Error Cases

F.1 Usefulness of Compiler Feedback

  • Pass@1 rises from 36.82 to 38.00 for RepoCoder with compiler feedback versus without it on CoderEval-Python.Source: §F.1 · p. 15

F.4 COCOGEN on Project-level Code Completion

F.2 More Results on the Ablation Study

  • Compiler feedback and structural queries improve performance in most dependency-level scenarios, with stronger results on file- and project-level tasks.Source: §F.2 · p. 15

F.3 Varying the Number of Iterations

F.5 COCOGEN on Function-level Code Generation

  • On HumanEval, six of seven unresolved compilation errors after the first iteration came from Markdown code-block markers, and one syntax error persisted through three iterations.Source: §F.5 · p. 16
  • On MBPP, two tasks account for all seven errors left after the final iteration: six from “sum” and one from “month_season.”Source: §F.5 · p. 16

F.6 Efficiency of Static Analysis

The appendix reports static-analysis latency and describes repository-context retrieval through precomputed structures or generated queries, alongside the project-database construction process.

F.6 Efficiency of Static Analysis

  • Static analysis and structural queries average 1.27 seconds per test case, with observed latency from 0.359 to 6.984 seconds.Source: §F.6 · p. 16

G.1 The Design Principle

  • For frequent errors, COCOGEN retrieves context from precomputed tables; otherwise, it asks the LLM to generate a CodeQL query over the project graph.With three iterations on CoderEval-Python, related context was retrieved for 93.2% of errors.Source: §G.1 · p. 17

G.2 Demonstration Examples of Structural Queries

  • The structural-query examples pair compiler messages with queries for four representative error types, selected randomly from the compilation log.Source: §G.2 · p. 18

H Algorithm for Project Database Construction

Loading 2403.16792v3…