Source-linked AI summary
RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, Weizhu Chen
TL;DR
Repository-level code completion needs broader repository context than unfinished in-file code alone can provide. RepoCoder combines retrieval with language-model generation in an iterative pipeline and introduces RepoEval for multiple completion granularities. It improves In-File completion by over 10% across settings and consistently enhances vanilla retrieval-augmented generation, although gains may be limited in repositories with little code duplication.
Problem
Repository-level code completion must use useful information scattered across repository files, but existing methods face limitations in completion flexibility and generalization.
Method
RepoCoder combines a similarity-based retriever and a pre-trained code language model in an iterative retrieval-generation pipeline that reuses model predictions for retrieval.
Results
RepoCoder improves In-File completion by over 10% across experimental settings and consistently enhances vanilla retrieval-augmented generation.
Takeaways & Limitations
RepoEval enables evaluation across line, API invocation, and function body completion, with unit tests improving evaluation accuracy.
Takeaways & Limitations
RepoCoder may provide limited improvement in repositories with few instances of code duplication because retrieval finds insufficient relevant information.
Abstract
from arXiv · showhide
The task of repository-level code completion is to continue writing the unfinished code based on a broader context of the repository. While for automated code completion tools, it is difficult to utilize the useful information scattered in different files. We propose RepoCoder, a simple, generic, and effective framework to address the challenge. It streamlines the repository-level code completion process by incorporating a similarity-based retriever and a pre-trained code language model in an iterative retrieval-generation pipeline. RepoCoder makes effective utilization of repository-level information for code completion and has the ability to generate code at various levels of granularity. Moreover, we propose a new benchmark RepoEval, which consists of the latest and high-quality real-world repositories covering line, API invocation, and function body completion scenarios. Experimental results indicate that RepoCoder significantly improves the In-File completion baseline by over 10% in all settings and consistently outperforms the vanilla retrieval-augmented code completion approach. Furthermore, we validate the effectiveness of RepoCoder through comprehensive analysis, providing valuable insights for future research. Our source code and benchmark are publicly available: https://github.com/microsoft/CodeT/tree/main/RepoCoder
1 Introduction
Repository-level code completion must use interrelated repository context, but existing approaches struggle with varying-length completions and generalization. RepoCoder addresses this with iterative retrieval-generation, while RepoEval benchmarks multiple completion granularities using unit tests.
- Repository-level completion must use dependencies, shared utilities, configurations, naming conventions, and coding styles distributed across files.
- Existing static-analysis and tuned-language-model approaches remain limited for varying-length completions and generalization beyond their evaluation scenarios.
- RepoCoder iteratively uses generated completions to retrieve repository snippets, narrowing the gap between retrieval context and the intended target.
- RepoEval evaluates line, API invocation, and function body completion using high-quality GitHub repositories and repository unit tests.
- Over 10% improvement over In-File completion was observed across experimental settings, while RepoCoder consistently enhanced vanilla retrieval-augmented generation.
2 Methodology
RepoCoder combines repository retrieval with language-model completion in an iterative pipeline. Each iteration uses prior generated code to refine retrieval while preserving the unfinished target code as context.
- Retrieval-Augmented Generation: The vanilla RAG pipeline retrieves repository snippets using unfinished code X, then generates a completion from the retrieved snippets and X.The repository is partitioned into code snippets, and the retriever selects relevant snippets before generation.
- Iterative Retrieval-Generation: RepoCoder reuses the previous prediction to construct a new retrieval query, narrowing the gap between retrieved context and the intended completion target.For iteration i > 1, the query incorporates X and the previous prediction ˆY i−1.
- Framework Design: The retrieval and generation models remain unchanged across iterations, and constructing the retrieval database requires neither static analysis tools nor heuristic rules.The framework permits off-the-shelf retrievers and pre-trained language models.
- Code Retrieval: The retrieval database uses a sliding window that extracts contiguous code lines, with window size Sw and sliding size Ss controlling snippet construction.The window traverses repository files and advances by a fixed number of lines.
- Code Retrieval: The initial query uses the last Sw lines of unfinished code, whereas later queries concatenate the last (Sw − Ss) lines of X with the first Ss lines of ˆY i−1.This query construction supplements unfinished-code context with generated completion context.
- Code Generation: Prompts concatenate retrieved snippets with unfinished code X, ordering snippets by similarity and including their original file paths within a limit of K snippets.The prompt template integrates repository and target-file context for generation.
3 Benchmark Construction
RepoEval is a repository-level benchmark built from curated GitHub repositories and organized around line, API invocation, and function body completion. It combines repository metadata with unit-test-based evaluation for functional correctness.
- Benchmark Scope: RepoEval covers three completion granularities: line, API invocation, and function body, using unit tests to assess completed-function correctness.Each sample records its source repository, file path, line numbers, and ground-truth completion.
- Line Completion: The line-completion dataset selects 200 qualifying lines from each of 8 repositories, producing 1600 non-repetitive test samples.Selected lines are not comments and contain at least 5 tokens.
- API Invocation Completion: The API invocation dataset samples 200 non-repetitive in-repository API invocations from each of the same 8 repositories, producing 1600 test samples.The authors characterize in-repository API completion as harder than completion involving built-in or third-party APIs because customized training data is limited.
- Function Body Completion: The function-body dataset uses smaller repositories with unit-test-covered functions and selects bodies containing 3 to 30 lines, yielding 373 test samples.Smaller repositories were chosen because executing tests is time-consuming and computationally expensive.
4 Experimental Setup
The experiments compare in-file, oracle, and RepoCoder retrieval-augmented methods using multiple pre-trained generators and retrieval iterations. Line and API completion use similarity metrics, while function completion uses repository unit tests.
- Methods for Comparison: The in-file baseline conditions a pre-trained language model on target-file context without repository retrieval.It is implemented as a zero-shot completion method using the provided in-file prompt context.
- Methods for Comparison: The oracle method retrieves snippets using the unfinished code plus the first Ss lines of ground-truth completion, providing an upper bound conditioned on retriever R and generator M.It performs a single retrieval process before generating the completion.
- Implementation Details: The main experiments use a sparse bag-of-words retriever based on Jaccard similarity and evaluate GPT-3.5-Turbo alongside CODEGEN models with 6B, 2B, and 350M parameters.The setup includes four pre-trained language models with varying code-generation capabilities.
- Hyper-parameters: Experiments use maximum combined prompt-and-output lengths of 4,096 tokens for GPT-3.5-Turbo and 2,048 for CODEGEN, with retrieved snippets occupying half the prompt length.The authors report that performance was not highly sensitive to hyper-parameter changes.
- Evaluation Metrics: Line and API completion are evaluated with Exact Match and Edit Similarity, while function body completion is evaluated by executing repository unit tests and reporting Pass Rate.Exact Match is binary, Edit Similarity uses normalized Levenshtein distance, and Pass Rate indicates whether all corresponding tests pass.
5 Experimental Results
RepoCoder improves repository-level completion across line, API invocation, and function body settings, outperforming In-File completion and vanilla RAG while remaining competitive with Oracle retrieval.
- Over 10% absolute EM and 8% ES improvements over In-File completion appear across line and API datasets and model sizes.
- With at least two iterations, RepoCoder consistently outperforms vanilla RAG across all evaluated language models.
- RepoCoder achieves competitive performance compared with Oracle retrieval on line and API invocation completion.
- A simple sparse retriever achieves equivalent performance to a UniXcoder-powered dense retriever, indicating robustness across retrieval and generation models.
- On function body completion, RepoCoder significantly improves over In-File completion across most repositories and remains competitive with Oracle.
6 Analysis
The analyses connect completion quality to retrieved-code quality and location, showing that useful repository context often contains target-like statements or API examples and comes from related files.
- 6.1 Quality of Retrieved Code: Retrieved-code quality substantially affects completion performance, with helpful snippets resembling the target or showing target API usage.
- 6.1 Quality of Retrieved Code: GT-Code generally achieves the best performance by supplying ground-truth API invocation examples.
- 6.1 Quality of Retrieved Code: RepoCoder with two iterations provides higher recall for ground-truth API invocation examples than the compared retrieval setting.
- 6.2 Locations of Retrieved Code: Retrieved snippets commonly originate from similar imports, similar names, or the current directory when Oracle or RepoCoder outperforms In-File completion.
- 6.2 Locations of Retrieved Code: Restricting retrieval to those frequent file locations degrades performance, indicating that effective retrieval is not fully captured by that restriction.
7 Related Work
Related work spans repository-context completion, large language models, and joint retrieval-generation methods, while RepoCoder addresses flexibility and retrieval challenges with an in-context iterative approach.
- Repository Context in Code Completion: Traditional code completion uses code analysis and reranking efficiently but lacks flexibility for arbitrary-granularity generation.
- Repository Context in Code Completion: Prior repository-level work uses inflexible heuristics and classifier training for prompt construction, leaving challenges in leveraging code effectively.
- Joint Modeling Retrieval and Generation: Retrieval-generation methods extend from knowledge-intensive language tasks to code generation by incorporating retrieved documents or code examples.
- Joint Modeling Retrieval and Generation: In-context approaches increasingly treat language models as fixed black boxes while using model predictions as supplementary retrieval context.
8 Conclusion and Future Work
RepoCoder uses iterative retrieval and generation to exploit repository information and improve code completion, with RepoEval experiments supporting its effectiveness over In-File completion and vanilla RAG.
- RepoCoder combines a retriever and language model in an iterative process that bridges retrieval context and the target code.
- Experiments on RepoEval show that RepoCoder consistently and significantly enhances In-File completion performance while surpassing vanilla RAG.
- The authors identify simplicity, versatility, and effectiveness as qualities supporting further improvements in usability and robustness.
Limitations
RepoCoder’s effectiveness is constrained by repository duplication, iteration selection, latency, and limited exploration of prompts, models, and baselines.
- Low code duplication can limit RepoCoder’s performance improvements because retrieval struggles to find sufficient relevant repository information.
- Determining the optimal number of iterations remains challenging because later iterations may perform unstably relative to earlier ones.
- Additional retrieval-generation steps may increase latency, creating concerns for real-time deployment with strict latency requirements.
- Improving time efficiency through optimizations and adaptive iterations is outside the scope of the current paper.
- The study does not explore prompt templates, alternative retrieval or generation models, or systematic comparisons with complex confidential commercial products.
- RepoCoder is evaluated with a sparse retriever in the main experiments, while dense-retriever results are reported for line and API invocation completion.
C Code Duplication in Repositories
RepoCoder’s gains are examined in relation to repository code duplication, with higher duplication generally associated with larger improvements but not determining them alone.
- Figure 4 measures the correlation between RepoCoder Iter-2’s absolute performance improvements over In-File and repository duplication ratios.
- The duplication ratio is defined as duplicated code lines divided by total code lines.
- “diffusers” has the highest duplication ratio and a significant RepoCoder improvement on both line and API completion datasets.
- “FedScope” and “evaluate” have similar duplication ratios but different RepoCoder performance gains.
D Failed Cases between Iterations
The analysis shows that successive RepoCoder iterations can correct previously failed cases while also losing previously correct cases, with misleading retrieval contributing to failures.
- Table 8 reports changes in correct API invocation completions across different RepoCoder iterations, where correctness is determined by an EM score of 1.
- Each RepoCoder iteration passes some cases that the previous iteration failed and fails some cases that the previous iteration passed.
- Many failures result from misleading retrieved code, such as API examples whose parameter sets differ across files.
- Some line and API completions judged incorrect by EM are functionally correct, motivating unit-test-based evaluation alongside exact matching.
- RepoCoder successfully predicted a correct completion in a qualitative comparison involving GitHub Copilot, Tabnine, and Amazon CodeWhisperer.