Source-linked AI summary
ReACC: A Retrieval-Augmented Code Completion Framework
Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, Alexey Svyatkovskiy
TL;DR
Existing code-completion methods rely mainly on context within the input file, although programmers reuse related code snippets. ReACC retrieves lexically and semantically similar external code and combines it with an autoregressive generator through stage-wise training. It achieves state-of-the-art performance on CodeXGLUE and on the evaluated Python and Java benchmarks.
Problem
Existing code-completion methods are limited to information within the input file, while external related code may provide useful context for predicting following tokens.
Method
ReACC uses a hybrid code retriever and autoregressive language model, retrieving similar code with an unfinished snippet and concatenating it with the code context for generation.
Results
ReACC achieves state-of-the-art performance on CodeXGLUE and CodeNet in Python and Java, with retrieved external code useful for auto-completing partial code.
Takeaways & Limitations
Retrieval-augmented code completion can use external code containing lexical and semantic similarity as auxiliary information for code completion.
Takeaways & Limitations
The retriever avoids BM25 hard negatives because lexically similar code with distinct semantics can help code completion, and retrieval excludes CodeSearchNet to avoid dataset duplication.
Abstract
from arXiv · showhide
Code completion, which aims to predict the following code token(s) according to the code context, can improve the productivity of software development. Recent work has proved that statistical language modeling with transformers can greatly improve the performance in the code completion task via learning from large-scale source code datasets. However, current approaches focus only on code context within the file or project, i.e. internal context. Our distinction is utilizing "external" context, inspired by human behaviors of copying from the related code snippets when writing code. Specifically, we propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval. We adopt a stage-wise training approach that combines a source code retriever and an auto-regressive language model for programming language. We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark.
1 Introduction
ReACC extends code completion beyond the input file by retrieving external code that is lexically or semantically similar, then using it with the unfinished snippet to generate completions. The framework combines a retriever and autoregressive generator and achieves state-of-the-art performance on evaluated benchmarks.
- Motivation: Existing code-completion methods use information bounded within the input file, despite programmers often reusing existing snippets.This motivates expanding the information scope to an external codebase.
- Framework: ReACC retrieves similar code from a source-code corpus using the unfinished snippet as a query, then predicts subsequent tokens by reusing the retrieved code.The retrieved code is concatenated with the unfinished context before generation.
- Framework: The framework combines a dual-encoder code-to-code retriever with an autoregressive language-model generator trained stage-wise.The generator incorporates retrieved code and code context as its input.
- Retriever: ReACC uses a hybrid retriever to capture both lexical similarity useful for copying and syntactic or semantic similarity useful for intent matching.Its retriever is also pre-trained with contrastive learning and semantic-preserving code transformations for augmentation.
- Evaluation: ReACC achieves state-of-the-art performance on CodeXGLUE and CodeNet across Python and Java, with results showing retrieved external code helps complete partial code.The evaluation covers both benchmark datasets and both programming languages.
- Contributions: The study introduces partial code-to-code search as the retrieval task and creates datasets for evaluating it.The retrieval query is an unfinished code snippet rather than a natural-language description.
2 Related Work
Prior code-completion research uses written code context, structural information, or token types, but does not leverage existing external code as auxiliary information. Retrieval-based code intelligence commonly queries with natural-language descriptions, whereas ReACC addresses the harder setting of partial code as the query.
- Code completion: Code-completion research has used neural language models, pre-training, AST completion, sketches, and methods targeting efficiency or long-range modeling.These approaches generally consume previously written code context with structural or token-level information.
- Code completion: Earlier code-completion methods do not use existing external code as auxiliary information.Their inputs remain previously written code context, sometimes augmented with AST structure or token types.
- Contrastive learning on code: Contrastive code-representation methods use compiler-based semantic-preserving transformations to generate augmented snippets.These approaches include Contra-Code and Corder.
- Retrieval for code intelligence: Code retrieval has supported tasks including general-purpose code generation, retrieve-and-edit completion, structural code search, and relevant-snippet recommendation.These methods establish retrieval as a component in several code-intelligence settings.
- Retrieval on code intelligence: Most dense code-retrieval systems use a natural-language comment as the query, whereas partial code search uses incomplete code and has few labeled semantically similar pairs.Incomplete query semantics make this retrieval scenario more difficult.
3 Approach
ReACC extends code completion beyond local context by retrieving semantically similar code and combining it with an autoregressive generator. Its retriever uses hybrid lexical and dense search, semantic-preserving augmentation, and fragment alignment before generation.
- Task Formulation: ReACC reformulates completion as retrieving a similar code fragment from a source corpus and predicting subsequent tokens from retrieved code plus unfinished context.
- Retriever: The retriever combines BM25 lexical matching with dense encoders that represent syntactic and semantic information.
- Retriever Training: Semantic-preserving augmentation generates equivalent variants through identifier renaming and dead-code insertion.
- Retriever Training: Randomly truncated code serves as the query, while the entire transformed program is used as a positive example for incomplete-semantic retrieval.
- Generator: Fragment alignment feeds the next fragment from the retrieved file, concatenated with the original context, to the decoder-only generator.
4 Experiments: Code Clone Detection
The experiments evaluate ReACC retrieval using partial code clone detection on CodeNet in Python and Java. ReACC substantially outperforms CodeBERT and GraphCodeBERT, while BM25 performs strongly, indicating that lexical similarity remains useful for semantically related code.
- Dataset: The CodeNet evaluation uses newly created zero-shot Python and Java clone-detection datasets derived from online-judge solutions.
- Training: Retrieval training uses 1.6M Java methods and 1.2M Python functions from CodeSearchNet, with GraphCodeBERT initialization and 30 training epochs per language.
- Experimental Setup: The partial-search clone-detection setup evaluates whether a retriever can find semantically equivalent programs from incomplete code queries.
- Evaluation: Models are measured with MAP@K and precision at 1, emphasizing both ranked retrieval quality and the most similar result.
- Results: ReACC significantly outperforms CodeBERT and GraphCodeBERT when retrieving semantically equivalent code from incomplete queries.
- Results: BM25 performs strongly, suggesting that semantically related code is often lexically similar and that lexical retrieval contributes substantially.
5 Experiments: Code Completion
The experiments evaluate ReACC for code completion across CodeXGLUE and CodeNet, using Python and Java settings with token- and line-level metrics. ReACC consistently benefits from retrieved external code, with performance depending on retriever configuration and retrieval-database domain.
- Experiment Setup: ReACC is evaluated for end-to-end code completion on CodeXGLUE and CodeNet in Python and Java.CodeXGLUE experiments use PY150 and GitHub Java Corpus; CodeNet provides a Python algorithm-domain setting.
- Experiment Setup: Evaluation covers token-level completion with perplexity and line-level completion with exact match and edit similarity.For token-level completion, retrieval is performed after the first 100 predicted tokens rather than at every step.
- Results: ReACC with the hybrid retriever outperforms other baselines consistently across CodeXGLUE Python and Java datasets.The comparison includes models such as CodeGPT-adapted and X-CodeGPT, which use only internal file context.
- Analysis: The PY150 improvement exceeds the Java Corpus improvement because Java has a much smaller retrieval database of 12,934 training files.The authors connect the smaller database with greater difficulty retrieving similar Java code.
- Analysis: BM25 matches the dense retriever comparably and exceeds it on perplexity and exact match, indicating value from both lexical and semantic similarity.In the specific algorithm domain, however, dense retrieval significantly outperforms BM25.
- Analysis: On CodeNet’s algorithm-domain completion dataset, ReACC improves edit similarity by 10% absolute and exact match by 18% absolute over CodeGPT-adapted.The authors report that semantic similarity is more valuable than lexical similarity in this domain.
- Ablation Study: Ablations show metric degradation when retriever or generator augmentation and training strategies are removed, with query truncation identified as most essential.Identifier renaming contributes more than dead-code insertion, while removing fragment alignment causes a slight performance decline.
6 Conclusion
ReACC augments code completion with external code retrieved from an existing codebase using semantic and lexical similarity. Its partial-code retriever and compatible generator achieve state-of-the-art performance on the CodeXGLUE code completion benchmark.
- Conclusion: ReACC retrieves semantically and lexically similar code fragments to provide external context for code completion.The framework is designed to retrieve code given a partial code snippet.
- Conclusion: A dual-encoder is pretrained for partial-code search, while the generator can use any architecture capable of code completion.The conclusion identifies retrieval and generator flexibility as central design properties.
- Conclusion: ReACC achieves state-of-the-art performance on the CodeXGLUE code completion task.The conclusion reports this benchmark-level result without specifying a numerical score.
A Predefined Dead Code
The paper defines dead-code snippets from common Python and Java statement types for semantic-preserving data augmentation. Snippets may use one statement type or combine several types.
- A Predefined Dead Code: Dead code is constructed for Python and Java from declarations, expressions, conditionals, and loops.These four statement categories are treated as common statement types in the augmentation procedure.
- A Predefined Dead Code: Generated dead-code snippets can contain one statement category or combinations of different categories.The examples use the predefined statement set to form candidate snippets.
B Qualitative Examples
The qualitative section presents generated-code examples from the PY150 test set and compares outputs from different completion models. The examples include Django and Borg code contexts, with ReACC also evaluated using Copilot as generator.
- B Qualitative Examples: Figures 5 and 6 show qualitative code-completion examples from the PY150 test set.The examples are drawn from Django and Borg repository files.
- B Qualitative Examples: ReACC + Copilot denotes the ReACC framework using Copilot as its generator.This configuration separates the retrieval framework from the underlying generation model.
- B Qualitative Examples: The examples include a Django test-case context and Borg code involving logging, data masking, model estimation, and score calculation.The displayed inputs include imports, a test class, an evaluation function, and model-processing statements.