ReACC: A Retrieval-Augmented Code Completion Framework

Introduction

• Existing approaches use information bounded to the input file, while programmers reuse repository code, motivating completion from a larger codebase.ReACC proposes retrieving snippets with similar semantics as auxiliary information.Source: §1 · p. 1
• ReACC retrieves similar code from a source corpus and uses it with the unfinished snippet to predict the following code tokens.Its two components are a dual-encoder code retriever and an autoregressive language-model generator.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The hybrid retriever combines dense semantic and sparse lexical retrieval; its dual encoder starts from GraphCodeBERT and is contrastively pretrained using transformed similar-function programs.The hybrid approach follows Karpukhin et al. (2020) and Ma et al. (2021); GraphCodeBERT is cited as Guo et al. (2020).Source: §1 · p. 2
• The decoder-only transformer concatenates retrieved code with the code context and is initialized from CodeGPT-adapted, a code-corpus adaptation of GPT-2.The cited models are CodeGPT-adapted (Lu et al., 2021) and GPT-2.Source: §1 · p. 2
• ReACC achieves state-of-the-art performance on CodeXGLUE and CodeNet in Python and Java, and experiments show retrieved external code helps complete partial code.Source: §1 · p. 2
• The authors introduce partial code-to-code search for unfinished-code queries and create datasets to evaluate that task.Source: §1 · p. 2
• The authors use semantic-preserving transformations to augment data for pretraining the code retrieval model.Source: §1 · p. 2

2.1 Code completion

• Code completion research has used N-gram language models, deep neural networks and pre-training, AST next-node prediction, code-sketch generation, and approaches to improve efficiency and long-range modeling.Examples include Hindle et al. (2016), Liu et al. (2016), Alon et al. (2020), Karampatsis et al. (2020), Liu et al. (2020), Svyatkovskiy et al. (2020, 2021), Li et al. (2018), Kim et al. (2021), Guo et al. (2021), and Clement et al. (2021).Source: §2.1 · p. 3
• None of the described completion approaches leverages existing external code as auxiliary information.Source: §2.1 · p. 3

2.2 Retrieval on code intelligence

• Contrastive learning has been applied to code-fragment representation through semantic-preserving augmentations and multimodal pretraining, with reported applications including clone detection, search, and summarization.Contra-Code (Jain et al., 2020) and Corder (Bui et al., 2021) use compiler-based transformations; SYNCOBERT (Wang et al., 2022) and UniXcoder (Guo et al., 2022) use code, comments, and AST data.Source: §2.2 · p. 3
• Code retrieval supports code search, general-purpose code generation, autocompletion, summarization, and generation through search, retrieve-and-edit, or dense retrieval methods.Examples include code search with natural-language queries (Cambronero et al., 2019), ReCode (Hayati et al., 2018), retrieve-and-edit (Hashimoto et al., 2018), Aroma (Luan et al., 2019), and RED-CODER (Parvez et al., 2021).Source: §2.2 · p. 3
• Code completion retrieval uses partial code rather than the natural-language comments commonly used as queries, making retrieval harder because semantics are incomplete and labeled similar-code pairs are scarce.Source: §2.2 · p. 3

3.1 Task Formulation

• Code files are split into equal-length fragments; given unfinished code, the retriever selects a similar fragment and the generator predicts the next tokens from both.For token-level completion, the target consists of one next token.Source: §3.1 · pp. 3–4

3.2 Retriever

• ReACC combines BM25 lexical matching with a dense DPR-style retriever whose shared-weight encoders compare partial-code queries with code fragments using in-batch InfoNCE negatives (Karpukhin et al., 2020; Ma et al., 2021; Robertson and Zaragoza, 2009; Oord et al., 2018).It avoids BM25 hard negatives because lexically similar code with distinct semantics can still help completion.Source: §3.2 · p. 4
• To train without labeled similar-program pairs, ReACC creates semantic-preserving variants using identifier renaming and dead-code insertion rather than searching for equivalent code through costly compilation and execution.Identifier candidates come from GraphCodeBERT’s top-10 predictions after identifiers are masked, preserving some lexical information; dead code is inserted after a randomly selected statement.Source: §3.2 · pp. 4–5
• Retriever training appends the API-usage sequence to code tokens, uses a random truncation as the query, and treats the entire created program as the positive example.The API sequence is included because it is highly related to snippet functionality.Source: §3.2 · p. 5

3.3 Generator

• The generator pairs the current context with the fragment following the retrieved fragment in the same file, then predicts the next code tokens.ReACC uses CodeGPT-adapted, a decoder-only transformer, in its experiments.Source: §3.3 · p. 5

4.1 Dataset

• The zero-shot Python and Java evaluation sets come from CodeNet, where solutions to the same problem count as semantically equivalent.The dataset collects solutions for thousands of problems; retriever pre-training uses 1.6M Java methods and 1.2M Python functions from CodeSearchNet.Source: §4.1 · pp. 5–6

4.2 Baseline Methods

• Baselines are CodeBERT, pretrained on CodeSearchNet NL-PL pairs across six languages, and GraphCodeBERT, which also models code data flow.Source: §4.2 · p. 6

4.3 Experiment Setup

• The GraphCodeBERT-initialized retriever is continually trained with masked-language modeling and contrastive learning, then evaluated by partial search using partial programs as queries.Training uses in-batch negatives, batch size 256, learning rate 5e-5, and 30 epochs each for Python and Java.Source: §4.3 · p. 6

4.4 Results

• In zero-shot partial search, ReACC significantly outperforms CodeBERT and GraphCodeBERT at retrieving equivalent code despite incomplete query semantics, measured by MAP@K and precision at 1.Source: §4.4 · p. 6
• BM25 performs splendidly, suggesting that semantically related code is often lexically similar and that lexical matching contributes strongly to code-to-code retrieval.Source: §4.4 · p. 6

5.1 Dataset

• The PY150 and Java Corpus datasets from CodeXGLUE provide the Python and Java code-completion evaluation datasets.The paper identifies CodeXGLUE as a benchmark spanning 14 datasets and 10 code-intelligence tasks.Source: §5.1 · p. 6

5.2 Baseline Methods

• CodeGPT and CodeGPT-adapted are baselines pretrained on CodeSearchNet’s Python and Java data; X-CodeGPT adapts eWASH to CodeGPT-adapted.CodeGPT is trained from scratch, whereas CodeGPT-adapted is initialized from GPT-2; X-CodeGPT is reproduced for comparison.Source: §5.2 · pp. 6–7

5.3 Experiment Setup

• The evaluation measures token-level completion with perplexity and line-level completion with exact match and edit similarity.For efficiency, retrieval is performed after the first 100 predicted tokens and used for subsequent prediction.Source: §5.3 · p. 7
• The PY150 and Java Corpus training sets serve as retrieval databases; CodeSearchNet is excluded to avoid duplication with CodeXGLUE’s GitHub-derived data.The hybrid retriever combines dense similarity with BM25 using α = 0.9, selected on development data for both datasets.Source: §5.3 · pp. 7–8

5.4 Results

• ReACC with the hybrid retriever consistently outperforms other baselines across CodeXGLUE datasets; hybrid performs best on almost all metrics, except exact match on the new PY150 test set.Comparison with X-CodeGPT on the new test set indicates that external context can be more useful than using the current file alone.Source: Table 3 · p. 7, Table 4 · p. 7, §5.4 · p. 8

5.5 Analysis

• On CodeNet’s algorithm-domain dataset, ReACC improves edit similarity by 10% and exact match by 18% absolute over CodeGPT-adapted.ReACC with the dense retriever significantly outperforms BM25 in this domain, where semantically similar code may be more valuable.Source: §5.5 · p. 8, Table 5 · p. 8
• Ablations show that removing retriever or generator augmentation and training strategies degrades performance; query truncation matters most, and identifier renaming helps more than dead-code insertion.Removing fragment alignment and feeding the retrieved snippet directly to the generator causes a slight performance drop.Source: §5.5 · p. 8, Table 6 · p. 8
• In qualitative examples, ReACC with CodeGPT as generator sometimes outperforms GitHub Copilot, and Copilot can benefit from ReACC’s retriever.The paper notes that Copilot uses OpenAI Codex as its backend and that Codex is more powerful than CodeGPT.Source: §5.5 · pp. 8–9

• ReACC augments code completion with external context by retrieving semantically and lexically similar code from existing codebases.Source: §6 · p. 9
• ReACC achieves state-of-the-art performance on the CodeXGLUE code completion benchmark.Source: §6 · p. 9
• A pretrained dual-encoder serves as a retriever for searching code fragments given partial code.Source: §6 · p. 9
• The framework can use any architecture capable of code completion as its generator.Source: §6 · p. 9

A Predefined Dead Code

• For Python and Java, dead-code snippets use declaration, expression, conditional, or looping statements, either individually or in combination.Examples are shown in Figure 4.Source: A Predefined Dead Code · p. 12

B Qualitative Examples

• Figures 5 and 6 show qualitative examples of generated code from different models, with ReACC + Copilot denoting ReACC using Copilot as the generator.Source: B Qualitative Examples · p. 12, Figure 5 · p. 13, Figure 6 · p. 14
• Figure 5 presents a qualitative example from the PY150 test set, using input code from Django Daydreamer’s http/base.py.Source: Figure 5 · p. 13
• Figure 6 presents a qualitative example from the PY150 test set, using input code from borg’s mul_over_alpha.py.Source: Figure 6 · p. 14