PaperlayerView exact revision on arXiv

Source-linked AI summary

IRCoCo: Immediate Rewards-Guided Deep Reinforcement Learning for Code Completion

Bolun Li, Zhihong Sun, Tao Huang, Hongyu Zhang, Yao Wan, Ge Li, Zhi Jin, Chen Lyu

arXiv:2401.16637v3cs.SE

TL;DR

Code completion faces exposure bias and DRL methods whose delayed rewards do not readily account for changing contexts or partial code. IRCoCo uses immediate rewards in a code-completion-specific DRL framework, and experiments report improved performance over SFT across configurations. The authors also note that exact-match-based metrics can overlook semantically equivalent code.

  • Problem

    Code-completion DRL must address changing intents that delayed rewards may miss and partial code that is difficult to evaluate for reward redistribution.

    Source: §1 · p. 3

  • Method

    IRCoCo combines an actor-critic framework with immediate rewards and reward shaping based on evaluating subsequent code-fragment completion.

    Source: §1 · p. 3, §3.4 · p. 9

  • Results

    IRCoCo consistently enhances code-completion efficacy across configurations; CodeGPT with a BLEU-trained critic improves Python Edit-Sim by 7.9% over SFT.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The authors conclude that immediate-reward DRL can improve pretrained LMs’ performance on code completion.

    Source: §9 · p. 20

    The BLEU- and Edit-Sim-based evaluator metrics overlook code semantics, while obtaining completion acceptance-rate data is challenging.

    Source: §7 · p. 19

Abstract

from arXiv · show

Code completion aims to enhance programming productivity by predicting potential code based on the current programming context. Recently, pretrained language models (LMs) have become prominent in this field. Various approaches have been proposed to fine-tune LMs using supervised fine-tuning (SFT) techniques for code completion. However, the inherent exposure bias of these models can cause errors to accumulate early in the sequence completion, leading to even more errors in subsequent completions. To address this problem, deep reinforcement learning (DRL) is an alternative technique for fine-tuning LMs for code completion, which can improve the generalization capabilities and overall performance. Nevertheless, integrating DRL-based strategies into code completion faces two major challenges: 1) The dynamic nature of the code context requires the completion model to quickly adapt to changes, which poses difficulties for conventional DRL strategies that focus on delayed rewarding of the final code state. 2) It is difficult to evaluate the correctness of partial code, thus the reward redistribution-based strategies cannot be adapted to code completion. To tackle these challenges, we propose IRCoCo, a code completion-specific DRL-based fine-tuning framework. This framework is designed to provide immediate rewards as feedback for detecting dynamic context changes arising from continuous edits during code completion. With the aid of immediate feedback, the fine-tuned LM can gain a more precise understanding of the current context, thereby enabling effective adjustment of the LM and optimizing code completion in a more refined manner. Experimental results demonstrate that fine-tuning pretrained LMs with IRCoCo leads to significant improvements in the code completion task, outperforming both SFT-based and other DRL-based baselines.

1 INTRODUCTION

IRCoCo addresses exposure bias in code completion and the mismatch between delayed-reward DRL and changing code contexts. Its immediate-reward framework is evaluated across pretrained language models, programming languages, and code-completion metrics.

1 INTRODUCTION

  • The study targets line-level completion and motivates localized fine-tuning because pretrained models are costly to train and hosted services can raise privacy concerns.Source: §1 · p. 2
  • SFT teacher-forcing can create exposure bias: at inference, reliance on prior predictions can make errors accumulate and hinder contextually appropriate tokens.Training uses ground-truth sequences, whereas testing requires predictions based on the model’s prior outputs.Source: §1 · p. 2
  • Delayed-reward DRL struggles to reflect changing completion intents, while reward redistribution based on unit tests is difficult when partial code lacks tests or complete functionality.Source: §1 · p. 3
  • IRCoCo uses immediate feedback for changing contexts and shapes rewards by evaluating how generated tokens support subsequent code-fragment completion.The framework is described as model-agnostic and designed specifically for code-completion tasks.Source: §1 · pp. 3–4
  • Across 28 configurations (7 pretrained LMs × 2 evaluators × 2 languages), IRCoCo consistently enhances code-completion efficacy; with CodeGPT and a BLEU-trained critic, Python Edit-Sim improves 7.9% versus SFT.Source: §1 · p. 4

2 MOTIVATION

The motivating example contrasts errors from SFT and delayed-reward DRL with IRCoCo’s immediate-feedback approach. The section also explains how critic rewards are designed and reports average gains over delayed-reward methods on Py150.

2.1 A Motivating Example

  • In the example, SFT predicts resp instead of return, while delayed-reward DRL moves closer to the reference but still misses the render_class-to-resp assignment.Figure 1 presents the developer-written code alongside the two CodeGPT completions.Source: §2.1 · pp. 4–5, Figure 1 · p. 5

2.2 Key Ideas

  • IRCoCo assigns the LM as actor and a trained critic to score generated tokens immediately, turning delayed sparse feedback into dense feedback as context changes.The actor uses the critic’s scores to refine its token-completion strategy.Source: §2.2 · pp. 5–6

2.3 Feasibility Analysis

  • Because partial completions often lack unit tests and token contributions are hard to assess in changing contexts, the proposed critic rewards tokens according to whether they support subsequent completions.BLEU and Edit-Sim are proposed as evaluation metrics for designing immediate rewards.Source: §2.2 · p. 6, §2.3 · p. 6
  • On Py150, immediate rewards improve Edit-Sim by 3.32%, EM by 4.56%, BLEU-4 by 3.75%, and CodeBLEU by 2.81% on average over delayed-reward methods.The passage points to Table 3 for these results.Source: §2.3 · p. 6

3 IRCOCO

IRCoCo combines supervised fine-tuning with actor-critic reinforcement learning, using evaluator-based immediate rewards to guide code completion. Its method defines the completion task, trains a quality evaluator, and jointly optimizes the model with SFT and DRL.

3 IRCOCO

  • IRCoCo fine-tunes a pretrained LM as an actor, trains an evaluator as critic, then jointly optimizes the actor with SFT and DRL.The actor samples tokens appended to the incomplete fragment, and the critic returns immediate rewards.Source: §3 · p. 6, Figure 2 · p. 7

3.1 Code Completion Task

  • Given a partial code sequence, the completion task predicts following code fragments until the end-of-sequence token is generated.Source: §3.1 · p. 7

3.2 Supervised Fine-Tuning-Based Model Training

  • SFT trains the model to minimize cross-entropy between generated code and reference code, using teacher-forcing at each decoding step.Source: §3.2 · p. 7

3.3 Quality Evaluator for Generated Code

  • The evaluator scores completions against reference code using BLEU or Edit-Sim, with higher scores indicating greater expectation of a correct completion from the current token.Its immediate reward targets anticipated benefit from the entire completion rather than only the next token’s local gain.Source: §3.3 · pp. 7–8
  • Evaluator training pairs incomplete fragments with scores from generated completions, then learns to predict those scores from the fragments by minimizing mean-square error.Source: §3.3 · pp. 8–9

3.4 Reinforcement Learning-Based Alignment of Generated Code

  • IRCoCo models token generation as a Markov decision process, rewarding each generated step through a trained evaluator and carrying the preceding token’s reward to end-of-sequence.The state includes the incomplete fragment and previously generated tokens; actions sample the next token from the vocabulary.Source: §3.4 · p. 9
  • The policy network samples synthetic completion sequences, and policy-gradient training minimizes negative expected returns.The training procedure computes immediate rewards for sampled examples and updates gradients using the corresponding losses.Source: §3.4 · p. 10
  • The final training objective sums SFT and DRL losses, combining supervised learning from data with autonomous trial-and-error learning.Source: §3.4 · p. 10

4 EXPERIMENTAL SETUP

The experimental setup evaluates IRCoCo across code-completion models and datasets, using established similarity and exact-match metrics. It specifies the research questions, data preparation, baselines, and implementation settings.

4 EXPERIMENTAL SETUP

  • The experiments address five questions on completion effectiveness, immediate rewards, learning objectives, token counts, and qualitative realism.Source: §4 · p. 11

4.1 Evaluation Datasets

  • Evaluation uses Py150 and Java Corpus, randomly splitting code fragments into an incomplete context and a 10-token reference for the completion task.Py150 contains 150,000 Python 2 files, while Java Corpus contains almost 30,000 Java files.Source: §4.1 · p. 11

4.2 Baselines

  • The six considered pretrained-LM baselines include GPT-2, CodeGPT, CodeGPT-adapt, CodeGen, StarCoder, and CodeT5+.Source: §4.2 · pp. 11–12

4.3 Evaluation Metrics

  • IRCoCo is evaluated with Edit Similarity, BLEU-4, CodeBLEU, and Exact Match Accuracy.Source: §4.3 · p. 12

4.4 Implementation Details

  • Implementation fine-tunes models before DRL alignment, trains BLEU- and Edit-Sim-based critics, and runs DRL for 10 epochs with batch size 2 and learning rate 2 × 𝑒−5.GPT-2 1.5B experiments use an RTX A6000; other experiments use two RTX 3090 GPUs.Source: §4.4 · p. 12

5 EXPERIMENTAL RESULTS

Across the experiments, IRCoCo generally improves code-completion performance, and its immediate rewards outperform alternative reward designs. Joint SFT–DRL training improves performance where either objective alone can overfit or encounter vanishing gradients; gains also hold across completion lengths and qualitative examples, though errors remain.

5.1 Effectiveness of Code Completion (RQ1)

  • IRCoCo significantly improves performance across base LMs and evaluation datasets; for CodeGPT on Py150 with a BLEU-trained evaluator, Edit-Sim rises from 60.66% to 65.44%.The same configuration raises EM from 15.65% to 21.94%, BLEU-4 from 38.10% to 43.71%, and CodeBLEU from 42.42% to 45.79%.Source: §5.1 · pp. 12–13, Table 2 · p. 13

5.2 Validity of Immediate Rewards (RQ2)

  • IRCoCo surpasses delayed, linearly attenuating, and 0-1 reward designs across metrics; on Py150 with CodeGPT and a BLEU-trained evaluator, it exceeds delayed rewards by about 3.5% in Edit-Sim.The reported comparison also gives gains of about 4.5% in EM, 3% in CodeBLEU, and 4% in BLEU-4.Source: §5.2 · p. 14, Table 3 · p. 14

5.3 Impact of Different Model Learning Objectives (RQ3)

  • Joint SFT and DRL fine-tuning improves performance, while SFT-only training overfits and DRL-only training encounters vanishing gradients.The authors report that hybrid training alleviates the gradient-vanishing problem; the table compares additional SFT, DRL, and joint training after five SFT epochs.Source: §5.3 · p. 15, Table 4 · p. 15

5.4 Quantitative Analysis (RQ4)

  • IRCoCo outperforms pretrained LMs across completion lengths on Py150 and Java Corpus, with consistent superiority at longer lengths.The reported improvement is smaller on Java, which the authors attribute to its more complex syntax and smaller training dataset.Source: §5.4 · p. 16, Figure 4 · p. 16, Figure 5 · p. 16

5.5 Qualitative Analysis (RQ5)

  • Qualitative examples show IRCoCo can produce correct completions, while other reward schemes may leave errors; complementary errors still occur in both datasets.The authors report that some IRCoCo predictions are close enough to the reference for developers to use with minor modifications.Source: §5.5 · pp. 17–18, Figure 6 · p. 17

6 DISCUSSION

The discussion explains why the study evaluates line-level code completion with similarity and accuracy metrics and does not compare against CodeRL. The task's evolving context, incomplete outputs, and lack of unit tests make CodeRL's setting and reward mechanism a poor fit.

6 DISCUSSION

  • The study uses Edit-Sim, BLEU, and accuracy to assess completion quality, and does not use pass@k because current code-completion datasets lack unit tests.The authors explain that completion aims for accurate or similar fragments that programmers can modify, not necessarily fully correct code.Source: §6 · p. 18
  • CodeRL targets code generation from natural-language descriptions, whereas this study focuses on line-level code completion with a context that changes as tokens are completed.CodeRL's encoder-decoder reward framework assumes the natural-language input remains unchanged.Source: §6 · p. 18
  • Because completion outputs may be incomplete and datasets lack unit tests, CodeRL's unit-test-based rewards do not readily apply to this task.The passage notes that unit-test datasets such as APPS address programming contests rather than code-completion objectives.Source: §6 · p. 18

7 THREATS TO VALIDITY

The authors identify limitations in automated code-completion evaluation: exact-match-style metrics can miss semantic equivalence, while acceptance-rate data are difficult to obtain. They also note that metric computation can mark equivalent expressions incorrect.

7 THREATS TO VALIDITY

  • Equivalent expressions such as “b==a” and “a==b” can be scored as incorrect by the automated metrics used in the study.The authors propose integrating human evaluations alongside standard metrics to alleviate this threat.Source: §7 · p. 18
  • Exact-match-based BLEU and Edit-sim metrics overlook code semantics, potentially undermining their reliability as measures of completion quality.The authors use these metrics to build evaluator datasets because manually constructing such datasets is impractical; acceptance rate is desirable but difficult to obtain.Source: §7 · p. 19
  • The authors choose similarity metrics despite their limitations because they align more closely with the desired acceptance rate for short industrial completions needing only minor programmer modifications.Obtaining acceptance-rate data is challenging.Source: §7 · p. 19

8 RELATED WORK

Related work spans heuristic, statistical, and SFT-based completion methods, as well as DRL for sequence generation. The authors position code completion as requiring immediate rewards because its context changes and unit tests are unsuitable for partial outputs.

8 RELATED WORK

  • Rule-based and N-gram approaches can require substantial manual expertise or high-quality, high-quantity data and may struggle with complex tasks or code semantics and context.The section contrasts these earlier approaches with later deep-learning methods.Source: §8 · p. 19
  • SFT-based methods use contextual learning, hybrid networks, or syntax trees, and pretrained LMs are fine-tuned for completion, but exposure bias remains an issue.The section also discusses code repair after completion and approaches that predict abstract-syntax-tree nodes.Source: §8 · p. 19
  • DRL has been applied to sequence generation using final metrics such as BLEU and ROUGE, actor-critic rewards, and conditional sequence modeling.The cited applications include machine translation, code summarization, image generation, and Decision Transformer.Source: §8 · p. 19
  • A prior code-completion approach combines compiler feedback with syntactic and semantic match scores to reduce reward sparsity.The method is described as guiding generated code toward closer alignment with the correct target.Source: §8 · p. 20
  • Delayed sequence-level rewards can slow convergence or encourage local optima, while unit tests are not a direct reward signal for potentially incomplete code.The section therefore identifies immediate rewards as important for code-completion generation.Source: §8 · p. 20
  • The paper does not aim to surpass closed- or open-source LLMs on natural-language-to-code tasks; it presents a fine-tuning mechanism for LMs facing exposure bias and delayed rewards.The stated mechanism may apply across models subject to those challenges.Source: §8 · p. 20

9 CONCLUSION

The conclusion presents IRCoCo as a code-completion DRL framework that combines SFT fine-tuning with DRL alignment and uses an evaluator for immediate rewards. The authors report improved pretrained-LM performance and make the experimental data and source code available.

9 CONCLUSION

  • IRCoCo combines SFT fine-tuning and DRL-based alignment to augment pretrained LMs for code completion.A code-completion quality evaluator supplies immediate rewards during generation.Source: §9 · p. 20
  • The authors report that IRCoCo improves pretrained-LM performance on code completion.The conclusion describes this as the outcome of its immediate-reward-guided DRL framework.Source: §9 · p. 20
  • Experimental data and source code are available at the repository identified by the authors.The paper provides the repository address in its data-availability statement.Source: §9 · p. 20
Loading 2401.16637v3…