PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems

Zhensu Sun, Xiaoning Du, Fu Song, Shangwen Wang, Mingze Ni, Li Li, David Lo

arXiv:2209.05948v3cs.SEcs.AI

TL;DR

Neural code completion systems often produce unhelpful suggestions that burden developers and waste computation, motivating a cost-friendly way to avoid low-return requests. The paper identifies observable prompt patterns and introduces FrugalCoder, which estimates completion quality before LCM inference; its best estimator rejects 20% of requests at 97.4% precision.

  • Problem

    Around 70% of displayed Github Copilot completions are not accepted, creating developer review costs and wasting computation and energy.

  • Method

    FrugalCoder estimates completion quality from prompts and rejects requests below a preset threshold before sending them to the LCM.

  • Results

    The best-performing estimator achieves 97.4% precision when rejecting 20% of code prompts.

  • Takeaways & Limitations

    Observable low-return prompt patterns make early rejection feasible for improving productivity and reducing LCM computational costs.

  • Takeaways & Limitations

    The estimators rely on BLEU and CrystalBLEU, whose exact-match-based measurements may not reliably capture completion semantics.

Abstract

from arXiv · show

Currently, large pre-trained language models are widely applied in neural code completion systems. Though large code models significantly outperform their smaller counterparts, around 70\% of displayed code completions from Github Copilot are not accepted by developers. Being reviewed but not accepted, their help to developer productivity is considerably limited and may conversely aggravate the workload of developers, as the code completions are automatically and actively generated in state-of-the-art code completion systems as developers type out once the service is enabled. Even worse, considering the high cost of the large code models, it is a huge waste of computing resources and energy, which severely goes against the sustainable development principle of AI technologies. However, such waste has never been realized, not to mention effectively addressed, in the research community for neural code completion. Hence, preventing such unhelpful code completions from happening in a cost-friendly way is of urgent need. To fill this significant gap, we first investigate the prompts of unhelpful code completions, called "low-return prompts". We empirically identify four observable patterns in low-return prompts, each lacking necessary information, making it difficult to address through enhancements to the model's accuracy alone. This demonstrates the feasibility of identifying such low-return prompts based on the prompts themselves. Motivated by this finding, we propose an early-rejection mechanism to turn down low-return prompts by foretelling the code completion qualities. The prompts that are estimated to receive unhelpful code completions will not be sent to the model. Furthermore, we investigated five types of estimators to demonstrate the feasibility of the mechanism. The experimental results show that the estimator can reject 20% of code completion requests with a 97.4% Precision.

1 INTRODUCTION

Large code models can generate many unhelpful completions, imposing review costs on developers and wasting computation and energy. This paper proposes FrugalCoder, an early-rejection mechanism that estimates completion quality from prompts and blocks low-return requests before LCM inference.

  • Motivation: Around 70% of displayed Github Copilot completions are not accepted, requiring review effort while wasting computational resources and energy.These unhelpful completions threaten developer productivity and conflict with sustainable AI goals.
  • Motivation: Unhelpful completions may reflect either inadequate models or prompts lacking enough information for a proper completion.The latter cannot always be solved by improving model accuracy, because some prompts are inherently uninformative.
  • Approach: FrugalCoder estimates completion quality from each prompt and blocks requests whose estimated quality falls below a chosen threshold before activating the LCM.The mechanism is designed to improve productivity and reduce LCM operating costs simultaneously.
  • Approach: Estimator design involves a trade-off: the LCM is most accurate but saves no energy, whereas lighter estimators risk usability loss or missed low-return prompts.The paper therefore investigates learning-based estimators as a cost-friendly alternative to manual rules and full LCM inference.
  • Evaluation: Across Java and Python, two LCMs, two metrics, and the HumanAccept benchmark, an encoder-only Transformer rejected 20% of prompts with 97.4% precision.The evaluation tests estimator accuracy, productivity effects, and cost-friendliness.
  • Contributions: The paper contributes a first general mechanism for rejecting unhelpful completions, observable low-return prompt patterns, lightweight estimators, and a comprehensive evaluation.The implementation artifacts and results are released for future research and industrial practice.

2 UNHELPFUL CODE COMPLETIONS AND LOW-RETURN PROMPTS

The paper defines unhelpful completions and low-return prompts, then uses a manual study to identify observable prompt patterns that can support early rejection. Its study constructs prompt–completion pairs, obtains Copilot suggestions, and examines why rejected completions arise.

  • 2.1 Unhelpful Code Completions: Low-return prompts are prompts that lead to unhelpful completions, and recognizing them before LCM inference could avoid their productivity and resource costs.The recognizer must distinguish them effectively while remaining cost-friendly.
  • 2.1 Unhelpful Code Completions: Unhelpful completions fail to improve productivity because they may mismatch developer intent or require substantial understanding and debugging effort.Prior studies surveyed Copilot feedback and tracked discarded suggestions, including misleading recommendations in failed programming tasks.
  • 2.2 Low-return Prompts: The manual study sampled Java snippets from COFIC by splitting code into prefixes and ground-truth suffixes, producing 383 prompt–completion pairs at 95% confidence and 5% margin of error.Copilot automatically generated one recommendation for each prompt, including blank completions.
  • 2.2 Low-return Prompts: Two authors judged each completion against its context and ground truth, accepting it only when it exactly matched or achieved the same intention as the next ten tokens.They jointly reviewed unhelpful cases and incrementally recorded prompt patterns without claiming a complete feature list.
  • 2.2 Low-return Prompts: Meaningless identifier names provide little information about type or intent, making desired completions difficult; mnemonic names offer clearer inference cues.The paper contrasts a vague variable name such as ss with lowerCased for suggesting toLowerCase().
  • 2.2 Low-return Prompts: The study also identifies unopen context as a low-return pattern among prompts examined for causes of unhelpful completions.The supplied passage names the pattern but does not provide its full operational explanation.
  • 2.2 Low-return Prompts: Vague intention leaves too many plausible completions when neither names nor comments reveal what the developer wants.The figure examples mark pattern instances in red, variants in green, and generated completions with gray backgrounds.

3 FRUGAL CODE COMPLETION

FrugalCoder selectively blocks code completion prompts predicted to yield low-quality suggestions, aiming to reduce unreliable outputs and computational waste while balancing accuracy with estimator cost.

  • 3 FRUGAL CODE COMPLETION: The mechanism targets low-return prompts because indiscriminate handling produces unreliable suggestions and wastes resources.
  • 3 FRUGAL CODE COMPLETION: FrugalCoder estimates completion quality before inference and blocks prompts whose estimated score falls below a configurable threshold.Blocked prompts are not sent to the LCM, and users are notified.
  • 3 FRUGAL CODE COMPLETION: FrugalCoder’s effectiveness depends on accurately identifying unhelpful completions while keeping computing consumption low.These goals create a joint optimization challenge for the estimator.
  • 3 FRUGAL CODE COMPLETION: Incorrectly rejecting a prompt can deny developers a potentially helpful completion and force them to resubmit the request.
  • 3 FRUGAL CODE COMPLETION: Scaling the estimator indefinitely is not viable because greater estimation accuracy can conflict with the cost savings FrugalCoder is intended to provide.

4 LEARNING-BASED COMPLETION QUALITY ESTIMATOR

The paper formulates completion-quality estimation as a learning problem that maps prompts to scores derived from target-model completions, while comparing estimators with different accuracy and efficiency profiles.

  • 4 LEARNING-BASED COMPLETION QUALITY ESTIMATOR: Learning-based estimators map code prompts to the quality scores of completions produced by a target LCM.The task is treated as regression because learning-based models can capture prompt semantics more flexibly than hand-written rules.
  • 4 LEARNING-BASED COMPLETION QUALITY ESTIMATOR: Because production acceptance data are unavailable, accuracy metrics such as BLEU substitute for behavioral measures of completion helpfulness.These metrics compare generated completions with known correct completions.
  • 4 LEARNING-BASED COMPLETION QUALITY ESTIMATOR: Training examples pair a randomly split code prompt with its ground-truth continuation and the target LCM’s accuracy score.The target model generates a completion, which is compared with the ground truth using a predefined accuracy metric.
  • 4 LEARNING-BASED COMPLETION QUALITY ESTIMATOR: Prompts with null ground-truth completions receive the minimum score so the estimator can recognize the Unopen Context pattern as low-return.
  • 4 LEARNING-BASED COMPLETION QUALITY ESTIMATOR: The methodology focuses on single-line completion because exact-match metrics can misrepresent quality for multiple-line predictions.
  • 4 LEARNING-BASED COMPLETION QUALITY ESTIMATOR: The study implements Transformer, recurrent, linear-regression, and AdaBoost estimators to cover different deployment requirements and cost profiles.The set includes encoder-only and decoder-only Transformer models, an LSTM, and lightweight machine-learning alternatives.

5 EXPERIMENTAL SETUP

The experiments evaluate estimator accuracy and FrugalCoder feasibility across two languages, two target LCMs, multiple metrics, and a human-annotated acceptance benchmark.

  • 5 EXPERIMENTAL SETUP: The evaluation asks whether estimators predict completion quality accurately, preserve development productivity, and reduce inference costs relative to vanilla LCMs.
  • 5 EXPERIMENTAL SETUP: Experiments use StarCoder and CodeGen2 as target LCMs for code completion in Java and Python.
  • 5.2 Datasets: Training and evaluation use COFIC for Java and CodeSearchNet for Python, with dataset-specific train, validation, and test splits.COFIC contains 849,984 Java functions; CodeSearchNet provides separate pre-split Python sets.
  • 5.2 Datasets: HumanAccept contains acceptance annotations for completions from CodeGen2 and StarCoder across Python and Java.Its four subsets correspond to each model-language combination.
  • 5.3 Evaluation Metrics: BLEU and CrystalBLEU measure generated-completion similarity and accuracy, with CrystalBLEU designed to ignore trivially shared code n-grams.
  • 5.4 Implementation Details: The target LCMs use 4-bit quantization for inference, with limits of 256 input tokens and 10 generated tokens per completion task.

6 EXPERIMENTAL RESULTS

Across three research questions, FrugalCoder’s estimators accurately identify low-return prompts, improve retained-completion quality, and operate far more cheaply than large code models.

  • RQ1: Accuracy of estimators: Estimator performance varied across LCMs and settings, motivating configuration customization and preliminary testing on relevant datasets.The authors identify a trade-off between estimator accuracy, rejection coverage, and cost-friendliness.
  • RQ1: Accuracy of estimators: ET achieved 97.4% precision when rejecting 20% of code prompts, demonstrating the feasibility of learning-based early rejection.The evaluation covered multiple estimators, languages, LCMs, and metrics; ET consistently outperformed the other estimators in the reported comparisons.
  • RQ2: Development productivity: Estimator rejection raised ET’s average BLEU from 0.441 to 0.563 and acceptance rate from 27.4% to 40.9% when rejecting 50% of prompts.Rejected prompts had much lower acceptance rates, including 1.3% for ET at a 10% rejection rate.
  • RQ2: Development productivity: Rejecting the bottom 20% of prompts left only 4.9% acceptable among ET-rejected completions while increasing displayed-completion acceptance from 27.4% to 33.0%.This result summarizes FrugalCoder’s practical productivity benefit across the evaluated settings.
  • RQ3: Cost-friendliness: Estimators required at most 3.83 GFLOPs and 5.2 ms, whereas the evaluated LCMs required substantially more computation and generation time.The study reports that rejecting even less than 1% of requests can yield net computational savings for both LCMs.

7 RELATED WORK

Neural code completion has progressed from lightweight train-from-scratch models to large pre-trained code models, while performance estimation offers a way to assess neural models without full training or testing.

  • 7 RELATED WORK: Code completion research shifted from lightweight RNN-based models to encoder-decoder and decoder-only transformers pretrained on large code corpora.LCMs are large models trained or fine-tuned specifically to generate source code.
  • 7 RELATED WORK: Large code models create substantial computational costs, motivating methods such as weight sharing, pruning, and knowledge distillation.The paper characterizes sustainable LCM development as an ongoing challenge requiring community effort.
  • 7 RELATED WORK: Performance estimation predicts a neural model’s performance without training or testing it, and has been applied to bypass computation limits and accelerate model selection.Prior examples estimate translation-model or hyperparameter performance.

8 DISCUSSION

The discussion identifies limits in estimator evaluation, threshold selection, manual annotation, and metric reliability, while noting broader potential applications for FrugalCoder.

  • 8 DISCUSSION: Estimator effectiveness was tested on only two LCMs, and small HumanAccept groups may bias evaluation despite statistically determined sampling.The approach is theoretically applicable to other LCMs, but broader experimental validation remains outstanding.
  • 8 DISCUSSION: FrugalCoder requires a pre-set rejection threshold, but selecting an appropriate threshold remains non-trivial in complex production environments.The authors call for further study of threshold settings for optimal prevention of unhelpful prompts.
  • 8 DISCUSSION: Manual analyses by two authors may introduce judgment and subject-selection biases, although disagreements were discussed and resolved and the dataset was released.The paper presents these steps as mitigation and replication support rather than eliminating the underlying threat.
  • 8 DISCUSSION: Beyond productivity and efficiency, FrugalCoder may potentially help address adversarial attacks if adversarial examples exhibit patterns resembling low-return prompts.The paper presents this as potential future application rather than an evaluated capability.
  • 8 DISCUSSION: Estimator training relies on BLEU and CrystalBLEU because large-scale helpfulness labeling is infeasible, but exact-match metrics may not reliably capture code semantics.Acceptance is described as the preferred quality indicator, yet only a small annotated evaluation group was available.

9 CONCLUSION

The paper identifies low-return prompts as a source of unhelpful completions and proposes FrugalCoder, an early-rejection mechanism that estimates completion quality before activating an LCM.

  • 9 CONCLUSION: Low-return prompts threaten developer productivity and waste computational resources in neural code completion.The conclusion frames the problem as both a usability and sustainability concern.
  • 9 CONCLUSION: Manual inspection found four recurring low-return prompt patterns, supporting prompt-based recognition of unhelpful completions.These patterns motivate identifying low-return prompts before invoking the completion model.
  • 9 CONCLUSION: FrugalCoder uses lightweight learning-based estimators to reject lower-return prompts without activating the LCM, and experiments found accurate blocking across threshold settings.The authors identify more effective and lightweight estimators as a future direction for industrial application.
Loading 2209.05948v3…