PaperlayerView exact revision on arXiv

Source-linked AI summary

RLCoder: Reinforcement Learning for Repository-Level Code Completion

Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, Zibin Zheng

arXiv:2407.19487v1cs.SE

TL;DR

Repository-level completion relies on retrieval, but lexical methods struggle with code semantics and model-based retrievers lack labeled candidate data. RLCoder trains a retriever with evaluator feedback and selective retrieval, and reports improved benchmark performance with broader applicability.

  • Problem

    BM25 struggles to capture code semantics, while model-based retrieval lacks ground-truth candidate data for training.

    Source: §I · p. 1

  • Method

    RLCoder trains a retriever without labeled data using evaluator feedback based on target-code perplexity, alongside Split-Aggregate candidates and a stop signal.

    Source: §I · pp. 1–2

  • Results

    12.2% Exact Match improvement over previous methods; RLCoder outperforms state-of-the-art methods on CrossCodeEval and RepoEval.

    Source: §I · p. 2

  • Takeaways & Limitations

    RLCoder generalizes across LLMs and programming languages and can further improve existing methods such as RepoCoder.

    Source: §I · p. 2

Abstract

from arXiv · show

Repository-level code completion aims to generate code for unfinished code snippets within the context of a specified repository. Existing approaches mainly rely on retrieval-augmented generation strategies due to limitations in input sequence length. However, traditional lexical-based retrieval methods like BM25 struggle to capture code semantics, while model-based retrieval methods face challenges due to the lack of labeled data for training. Therefore, we propose RLCoder, a novel reinforcement learning framework, which can enable the retriever to learn to retrieve useful content for code completion without the need for labeled data. Specifically, we iteratively evaluate the usefulness of retrieved content based on the perplexity of the target code when provided with the retrieved content as additional context, and provide feedback to update the retriever parameters. This iterative process enables the retriever to learn from its successes and failures, gradually improving its ability to retrieve relevant and high-quality content. Considering that not all situations require information beyond code files and not all retrieved context is helpful for generation, we also introduce a stop signal mechanism, allowing the retriever to decide when to retrieve and which candidates to retain autonomously. Extensive experimental results demonstrate that RLCoder consistently outperforms state-of-the-art methods on CrossCodeEval and RepoEval, achieving 12.2% EM improvement over previous methods. Moreover, experiments show that our framework can generalize across different programming languages and further improve previous methods like RepoCoder. We provide the code and data at https://github.com/DeepSoftwareAnalytics/RLCoder.

I. INTRODUCTION

The introduction frames repository-level completion as retrieval under constraints in code semantics, training labels, candidate construction, and selective retrieval. It presents RLCoder as a feedback-trained retriever with natural candidates and a stop signal, then reports benchmark gains and broader applicability.

I. INTRODUCTION

  • BM25 cannot capture code semantics, whereas model-based retrieval understands semantics but lacks ground-truth candidate training data.The paper notes that obtaining labeled data requires significant parsing and expert-labeling effort, limiting generalizability.Source: §I · p. 1
  • RLRetriever learns what to retrieve from evaluator feedback without labeled data, using target-code weighted perplexity as its reward.Source: §I · pp. 1–2
  • The stop signal lets the retriever decide when to retrieve and which candidates to retain, addressing cases where extra repository context can detract from completion.Source: §I · pp. 1–2
  • 12.2% Exact Match improvement over previous methods; RLCoder generalizes across LLMs and programming languages and further improves RepoCoder on CrossCodeEval and RepoEval.Source: §I · p. 2
  • Split-Aggregate candidates preserve code continuity, while fixed windows may disrupt it and dependency parsing is limited to graph context and complex scenarios.Source: §I · pp. 1–2

A. Retrieval-Augmented Generation

Repository-level code completion uses retrieval-augmented generation because repositories exceed the context models can practically use. The retrieved snippets must preserve code context and avoid misleading generation.

A. Retrieval-Augmented Generation

  • RAG retrieves relevant snippets from a knowledge base and supplies them to a generator, which produces code using the query and retrieved candidates.Repository-level systems use this approach because the repository is too large to provide directly as generation context.Source: A. Retrieval-Augmented Generation · p. 2

B. Repository-Level Code Completion

  • Repository-level completion retrieves snippets from a code repository using the unfinished code as a query, then generates target code from the query and selected snippets.Related approaches also combine in-file and cross-file context, fuse analogy and rationale context, or iteratively retrieve and generate.Source: B. Repository-Level Code Completion · pp. 2–3
  • Fixed-window candidates can split a function signature across snippets, while dependency-graph retrieval can omit other potentially useful repository code.A partial signature may confuse generation, and retrieved snippets can also mislead it: the example retrieves a “values” function and prompts a call to nonexistent “__values.”Source: B. Repository-Level Code Completion · p. 3

III. METHODOLOGY

RLCoder trains a retriever using target-code perplexity feedback, then uses it to select repository context for generation. Its data pipeline filters repositories, analyzes dependencies, selects target code, and constructs candidates.

A. Overview

  • RLRetriever learns from evaluator feedback based on weighted perplexity of target code with retrieved candidates, and inference uses its selected candidates as context for generation.Training constructs unfinished code, target code, and a candidate codebase; inference builds a codebase from the current repository.Source: A. Overview · pp. 3–4

B. Data Construction

  • The data-construction pipeline comprises repository filtering, dependency analysis, target-code selection, and candidate construction.The tasks simulate repository-level completion of a line, API, or part of a function body.Source: B. Data Construction · p. 4, Figure 4 · p. 4

1) Repository Selection:

  • The authors randomly selected 10,000 Python and Java repositories created before March 2023, requiring cross-file dependencies and exclusion from CrossCodeEval and RepoEval.The filtering was intended to prevent potential data leakage.Source: 1) Repository Selection · p. 4

2) Dependency Analysis:

  • Import-based dependency graphs yield 27,919 Python and 41,647 Java multi-file clusters, which are topologically sorted so later files contain segments dependent on other files.Single-file clusters are eliminated.Source: 2) Dependency Analysis · p. 4

3) Target Code Selection:

The data-construction procedure masks a randomly selected target segment in a dependent file and excludes that file from retrieval candidates. Candidate construction then groups code into bounded-length snippets along blank-line boundaries.

3) Target Code Selection:

  • A target segment is selected at a random position in a dependent file other than the first file in its cluster, with its length chosen randomly within file boundaries.The position avoids the beginning and ending segments to ensure sufficient context.Source: 3) Target Code Selection · p. 4
  • The selected target segment is masked to simulate unfinished code, and its containing file is excluded from the candidate set.This keeps the file containing the completion target out of retrieval candidates.Source: 3) Target Code Selection · p. 4

4) Candidate Construction:

  • Split-Aggregate candidates are formed by dividing files at blank lines and concatenating mini-blocks without exceeding a preset length threshold.The strategy uses blank lines as separators between code snippets with different semantics and usage.Source: 4) Candidate Construction · pp. 4–5

C. Reinforcement Learning-based RLRetriever Training

RLRetriever is trained with feedback based on how retrieved candidates affect target-code perplexity, rather than execution results. Its weighted perplexity emphasizes early tokens and identifiers, and candidate rewards guide retrieval choices.

C. Reinforcement Learning-based RLRetriever Training

  • Execution-based reward is difficult because repository environments are costly to establish, execution can be slow, and suitable test cases may be unavailable.Source: 1) Design of Reward · p. 5
  • The objective is to select candidates that maximize the likelihood of the target code, equivalently minimizing its negative log-likelihood and perplexity.Source: 1) Design of Reward · p. 5
  • Weighted perplexity assigns greater importance to early target-code tokens and to API or identifier tokens.Token weights depend on position and whether a token is an identifier.Source: 1) Design of Reward · p. 5
  • A candidate receives reward 1 when its weighted perplexity is no higher than that of any other candidate in the set, and 0 otherwise.Source: 1) Design of Reward · p. 5
  • The learning objective aggregates the log probabilities of selecting candidates, weighted by their rewards.The selection probability is conditioned on the context and candidate set.Source: 1) Design of Reward · pp. 5–6

2) Stop Signal Mechanism for Candidates Selection:

The stop signal mechanism lets retrieval determine how many candidates to retain, including whether cross-file context is needed.

2) Stop Signal Mechanism for Candidates Selection:

  • An empty candidate marks a cutoff: retain only retrieved candidates before it, and a first-position stop indicates the task likely needs no cross-file context.Source: 2) Stop Signal Mechanism for… · p. 6

3) Learning from Reward:

RLRetriever receives evaluator rewards and updates its parameters iteratively, with retrieval results progressively improving in quality. At inference, RLCoder constructs repository candidates, retrieves and filters them, then supplies selected candidates with unfinished code to a generator.

3) Learning from Reward:

  • Evaluator rewards update RLRetriever iteratively, progressively improving retrieval results and the quality of selected code candidates.Source: 3) Learning from Reward · p. 6

D. Code Completion with RLCoder

  • At inference, RLCoder constructs repository candidates, retrieves with trained RLRetriever, retains candidates selected by the stop signal, and gives them with unfinished code to the generator.Source: D. Code Completion with RLCoder · p. 6

1) For RLCoder:

  • The evaluation compares RLCoder with RawRAG and RepoCoder and also uses UniXcoder as a dense retriever.Source: 1) For RLCoder · p. 6

2) For RLRetriever:

The retriever evaluation includes direct generation, lexical and dense retrieval, and a supervised retriever variant. Benchmarks cover multilingual and repository-level completion tasks, measured by exact match and edit similarity.

2) For RLRetriever:

  • Retriever comparisons include no retrieval, BM25, UniXcoder, and UniXcoder-SFT, which uses the lowest-target-perplexity candidate as its training label.Source: 2) For RLRetriever · p. 7

B. Benchmarks

  • Evaluation uses CrossCodeEval's Python and Java subsets and RepoEval's line-level and API-level tasks.RepoEval also includes a function-level task, which is not used here.Source: B. Benchmarks · p. 7

C. Evaluation Metrics

  • Exact Match checks whether the entire generated snippet matches the expected code, while Edit Similarity measures similarity through edit distance.Source: C. Evaluation Metrics · p. 7

D. Experimental Details

The evaluation uses five backbone models on CrossCodeEval and RepoEval, then reports RLCoder’s overall effectiveness and training trajectory. The section also specifies the training and evaluation setup.

D. Experimental Details

  • Training initialized RLRetriever from UniXcoder, used DeepSeekCoder-1B as evaluator, and ran for up to 20 epochs with early stopping.Experiments used two 80 GB Tesla A100 GPUs; evaluation used five generator backbones.Source: D. Experimental Details · p. 7

A. RQ1: Effectiveness of RLCoder

B. RQ2: Effectiveness of RLRetriever

The section compares RLRetriever with alternative retrieval methods, tests RLCoder components, and evaluates transferring its training framework to RepoCoder.

B. RQ2: Effectiveness of RLRetriever

  • RLRetriever outperformed comparison retrievers across metrics on both benchmarks; retrieval methods beat NoRetrieval, and reinforcement learning outperformed supervised fine-tuning.Source: B. RQ2: Effectiveness of RLRetriever · p. 8

C. RQ3: Contributions of Each Component

D. RQ4: Generalizability of RLCoder

  • Integrating RLCoder training into RepoCoder improved EM by 12.4% for Python and 8.1% for Java on CrossCodeEval; the fused model was comparable, but RLCoder was chosen for its single-round completion.Source: D. RQ4: Generalizability of RLCoder · p. 9

E. Case Study

The case study shows RLCoder selecting a useful repository candidate where surface similarity misleads other methods. The conclusion presents its reinforcement-learning framework, candidate construction, and stop signal as contributions to repository-level completion.

E. Case Study

A. Code Completion

  • Code completion research spans rule-based and example-based methods, deep learning, and LLM-based retrieval-augmented generation approaches.Source: A. Code Completion · p. 10

B. Repository-Level Code Completion

  • Repository-level completion research includes dependency analysis, static analysis, iterative retrieval and generation, and tool invocation; RLCoder distinguishes itself by avoiding labeled training data and using a stop signal.Source: B. Repository-Level Code Completion · p. 11

VII. CONCLUSION

  • RLCoder combines reinforcement learning, Split-Aggregate candidate construction, and a stop signal to avoid useless cross-file context.Source: §VII · p. 11
Loading 2407.19487v1…