PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Autocompletion from Real-World Datasets

Gareth Ari Aye, Seohyun Kim, Hongyu Li

arXiv:2011.04542v1cs.SE

TL;DR

Traditional code-completion evaluations use committed source code, but their results may not reflect real-world completion behavior. This paper trains comparable language models on logged completion examples and finds stronger benchmark and deployment outcomes, while examining why the data differ.

  • Problem

    Committed-code benchmarks may not reliably measure real-world autocompletion performance because models score substantially worse on logged completion events than on source-control identifiers.

  • Method

    The study compares identical n-gram and Transformer-style language models trained on committed code, real autocompletion examples, and working-version snapshots.

  • Results

    Training on real autocompletion examples yields the best performance on the autocompletion evaluation dataset and the highest tool usage in a live A/B test.

  • Takeaways & Limitations

    Real-world completion logs provide a more behaviorally aligned basis for evaluating and improving autocomplete models than committed-code corpora alone.

  • Takeaways & Limitations

    The experiments evaluate only n-gram and Transformer models, so stronger or different model families are not directly tested.

Abstract

from arXiv · show

Code completion is a popular software development tool integrated into all major IDEs. Many neural language models have achieved promising results in completion suggestion prediction on synthetic benchmarks. However, a recent study When Code Completion Fails: a Case Study on Real-World Completions demonstrates that these results may not translate to improvements in real-world performance. To combat this effect, we train models on real-world code completion examples and find that these models outperform models trained on committed source code and working version snapshots by 12.8% and 13.8% accuracy respectively. We observe this improvement across modeling technologies and show through A/B testing that it corresponds to a 6.2% increase in programmers' actual autocompletion usage. Furthermore, our study characterizes a large corpus of logged autocompletion usages to investigate why training on real-world examples leads to stronger models.

I. INTRODUCTION

Existing code-completion research largely trains and evaluates on committed source code, but this study tests whether real-world completion data better reflects developer behavior and improves deployed tool usage.

  • I. INTRODUCTION: Real-world evaluation exposes a substantial gap: a state-of-the-art model’s accuracy falls from 0.462 on version-control identifiers to 0.203 on logged autocompletion events.This gap motivates evaluating models on production completion behavior rather than only synthetic or held-out source-code identifiers.
  • I. INTRODUCTION: Models trained on real completion examples improve prediction accuracy across neural and count-based language models compared with models trained on version-controlled code.The comparison targets training-corpus selection while holding the broader modeling goal constant.
  • I. INTRODUCTION: A/B tests with thousands of programmers show statistically significant increases in autocompletion usage when real-world-trained models rank IDE suggestions.The deployment study connects offline corpus choice to actual tool usage.
  • I. INTRODUCTION: Working IDE code differs from committed code through lower parseability, more debugging and ad hoc testing idioms, and longer accepted-suggestion tokens.These differences help characterize why source-control identifiers can be an insufficient benchmark for completion behavior.
  • I. INTRODUCTION: Training on working snapshots or recently edited version-control files does not improve performance, indicating that recency bias alone does not explain real-world examples’ advantage.The study therefore distinguishes active completion context from merely newer source snapshots.

A. Datasets

The study compares identical language models trained on committed source code, logged completion acceptances, and working-file snapshots to isolate the effect of training-corpus choice.

  • A. Datasets: The Baseline dataset contains nearly one million committed source files and over three hundred million Hack tokens, representing already-written, reviewed code.Its construction reflects the source-control corpora commonly used in prior work.
  • A. Datasets: The Autocompletion dataset contains over three million logged acceptance events with surrounding context from Hack programmers’ IDE activity.It directly samples applied suggestions from real developer behavior.
  • A. Datasets: The Edit dataset contains nearly two million working snapshots of modified Hack files, enabling a test of whether freshness explains Autocompletion’s gains.Only a small subset of each snapshot is typically modified, unlike directly logged completion events.
  • A. Datasets: Experiments use n-gram, whole-token Transformer, and subtoken Transformer models to test whether corpus effects generalize across modeling technologies.The study focuses on training-corpus selection rather than introducing new model configurations.

1) N-gram model:

The study uses n-gram and Transformer language models to predict code tokens, with Transformer self-attention supporting long-distance dependencies. It also evaluates BPE tokenization and trains models on multiple corpora, including combined real and artificial data.

  • 1) N-gram model:: N-gram models estimate next-token probabilities by counting how often a token follows a preceding subsequence.KenLM provides modified Kneser-Ney smoothing to address unseen or infrequent n-grams.
  • 2) Transformer:: Transformers use self-attention to model long-distance relationships between code tokens, which is useful for non-local variable and function definitions.The architecture passes token representations through self-attention blocks before producing next-token probabilities.
  • 3) Byte pair encoding:: BPE represents code with partial tokens, and the evaluation finds significant accuracy improvements over whole-token encoding.Subtoken vocabularies can represent rare names through combinations of more common partial names.
  • Dataset setup: The study trains models on Baseline, Autocompletion, Edit, and combined Baseline ∪Autocompletion corpora using separate train, validation, and test splits.Each dataset is divided in an 8:1:1 ratio, while preprocessing selects corpus-specific vocabularies.
  • Training objective: Models estimate p(t_i|t_1, t_2, ..., t_{i−1}), pad short sequences, and train Transformer variants for up to 15 epochs with early stopping.Sequences shorter than 100 tokens, or 300 tokens for Transformer + BPE, are right-padded.

D. Offline Evaluation

Offline evaluation tests models on held-out Baseline and Autocompletion samples using top-1 accuracy and mean reciprocal rank. The evaluation considers only the ten highest-ranked suggestions because larger suggestion lists are impractical in an IDE.

  • Evaluation datasets: Models are evaluated on held-out test samples from both Baseline and Autocompletion datasets.Baseline evaluation randomly selects identifier tokens, whereas Autocompletion evaluation uses completion tokens accepted by real IDE users.
  • Metrics: Top-1 accuracy measures whether the model correctly predicts the next token, while MRR gives partial credit based on the correct token’s reciprocal rank.For example, a correct token ranked second contributes 0.5 to MRR.
  • Evaluation scope: Offline evaluation considers only the top ten predictions because displaying more suggestions is impractical in the IDE context.Predictions beyond rank ten receive zero under this evaluation procedure.

E. A/B Testing

The study tests whether model training data affects real-world autocompletion usage through live A/B tests with thousands of developers. Developers are randomly assigned to Baseline, Autocompletion, or Edit models, and daily accepted suggestions are recorded.

  • A/B testing: Thousands of developers participate in live A/B tests measuring whether model rankings change autocompletion usage.The experiments record completion acceptance events to compute population statistics for each group.
  • Experimental groups: Developers are randomly assigned to Baseline, Autocompletion, or Edit models, with Baseline serving as the control group.The tested models rank suggestions within the IDE rather than being evaluated only offline.
  • Usage measurement: The outcome is the number of completion suggestions accepted by each developer on each day.Acceptance events are used to compare autocompletion behavior across experiment groups.

III. ANALYSIS

Models trained on real-world autocompletion examples substantially outperform models trained on version-control code when evaluated on real completion events. This offline advantage extends to statistically significant increases in IDE suggestion acceptance, while dataset differences in vocabulary, token length, token kind, and recency help explain the performance gap.

  • RQ1: 0.462 to 0.203 top-1 accuracy is the Transformer drop when evaluation changes from held-out version-control identifiers to real-world autocompletion events.The n-gram model drops from 0.183 to 0.081, while Transformer + BPE drops from 0.542 to 0.312.
  • RQ2: 0.081 to 0.144 top-1 accuracy for n-gram models and 0.203 to 0.331 for Transformer models result when training changes from Baseline to Autocompletion on real-world examples.Transformer + BPE also improves from 0.312 to 0.353.
  • RQ2: Training on Autocompletion alone produces the best overall performance, whereas unions of training corpora do not generally offset concept drift despite providing more data.Baseline training remains best on held-out Baseline examples, illustrating that matched train-test corpora favor in-domain accuracy.
  • RQ3: 6.2% more suggestions per day are accepted when Transformer models trained on Autocompletion rather than Baseline rank IDE completions.The corresponding n-gram comparison yields a 5.7% increase; both improvements are statistically significant, with p-values of 0.04 and 0.094 respectively.
  • RQ3: Models trained on Edit do not increase daily suggestion acceptance relative to Baseline in either the n-gram or Transformer A/B tests.The reported p-values are 0.069 and 0.184 respectively.
  • RQ4: 27.04% to 33.85% is the increase in Baseline-corpus OOV rate when evaluation uses Autocompletion rather than Baseline sequences, while BPE reduces OOV rates below 1%.These vocabulary mismatches help explain the cross-corpus accuracy degradation and BPE’s smaller relative drop.
  • RQ4: 12% longer accepted completion tokens, lower context-token OOV rates, and fewer local-variable targets distinguish Autocompletion examples from version-control data.Accepted completions average 14.31 versus 12.78 characters; context-token OOV rates are 18.9% versus 25%, and local-variable rates are 30.13% versus 35.34%.
  • RQ4: Recent modification distinguishes all logged Autocompletion examples from potentially outdated committed code, but recency bias alone cannot explain their superior performance.Models trained on other recency-biased datasets do not show the same consistent advantage.

IV. THREATS TO VALIDITY

The study’s validity is bounded by its corpus handling, model choices, and language coverage. These constraints may affect how broadly its findings generalize.

  • a) Offline evaluation on one language: The study evaluates only one programming language, so differences in token distributions and development practices across languages remain outside the experiments.The authors believe the findings may apply to other languages, but this is not directly tested.
  • b) Data duplication: Not deduplicating training and test examples may overestimate model strength when code sequences are duplicated across datasets.The authors expect this effect to be less pronounced for autocompletion because copy-paste events would not typically coincide with completion.
  • c) Evaluation against other models: The experiments compare only n-gram and Transformer models, leaving the performance of stronger completion models untested.The paper’s focus is training-corpus selection, and similar corpus effects across both model types motivate broader applicability.

V. RELATED WORK

Related work spans statistical, neural, and industrial approaches to code completion. It also establishes the concern that synthetic benchmark accuracy may not predict real-world performance.

  • V. RELATED WORK: Code-completion research has progressed from n-gram and AST-based probabilistic models to RNNs and more recent Transformer architectures.These approaches model or rank likely next code tokens for software development tools.
  • V. RELATED WORK: GPT-2-based tools, including TabNine™, apply Transformer next-token prediction to code completion.The cited examples include Radford et al. (2019) and TabNine™.
  • V. RELATED WORK: Industry-focused studies evaluate completion models on both open-source and internal company data, while prior work warns that synthetic accuracy may not transfer to real-world use.This paper extends that concern by studying training on developer activity data internal to a large software company.

VI. FUTURE WORK

Future work considers richer context and input signals for real-world completion. The proposed directions address information that current models omit or that may behave differently outside synthetic benchmarks.

  • a) Copy mechanism: Future studies should examine how copy mechanisms affect performance when models move from synthetic benchmarks to real-world completion examples.Copying novel tokens from context is especially useful because local references are common in software.
  • b) Taking information after the cursor: Models currently use context before the cursor, but future work should investigate whether tokens after the cursor can provide an additional signal.The paper notes that code is often not written sequentially, making right-side and below-cursor context potentially relevant.
  • c) Incorporating signal from the autocompletion prefix: Future models could incorporate the autocompletion prefix, which IDEs use to filter suggestions but the study’s models ignore for the current token.The paper notes that this signal is unavailable from version-control data because the programmer’s typed prefix is unknown.

VII. CONCLUSION

The conclusion argues that committed-code evaluation can misrepresent real-world code-completion performance. Training on logged autocompletion activity improves evaluation and live usage outcomes.

  • VII. CONCLUSION: Accuracy drops sharply when n-gram and Transformer models are evaluated on production autocomplete logs rather than artificial committed-code examples.This disparity indicates that traditional evaluation methods are unreliable for measuring practical completion quality.
  • VII. CONCLUSION: Training on the Autocompletion corpus produces the best performance on the Autocompletion evaluation dataset among the tested corpora.The study constructs Baseline, Autocompletion, and Edit corpora to address concept drift.
  • VII. CONCLUSION: Models trained on Autocompletion data achieve the highest tool usage in a live A/B test involving thousands of programmers at Facebook.The live deployment connects corpus choice with observed usage of the autocomplete tool.
Loading 2011.04542v1…