PaperlayerView exact revision on arXiv

Source-linked AI summary

IntelliCode Compose: Code Generation Using Transformer

Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan

arXiv:2005.08025v2cs.CLcs.SE

TL;DR

Existing IDE completion commonly focuses on methods, APIs, or arguments rather than arbitrary code sequences. IntelliCode Compose addresses this gap with a multilingual transformer-based completion system trained on large source-code data and optimized for deployment. Its best reported Python model reaches 86.7% average edit similarity and 1.82 perplexity, while online surfacing depends partly on user typing speed and network reliability.

  • Problem

    Most existing code completion systems recommend methods, APIs, or arguments but leave broader method-call and whole-line completion limited.

  • Method

    IntelliCode Compose uses GPT-C, a generative transformer trained from scratch on multilingual source code, to generate arbitrary code-token sequences.

  • Results

    86.7% average edit similarity and 1.82 perplexity are reported for Python by the best model.

  • Takeaways & Limitations

    IntelliCode Compose generates syntactically correct multilingual code sequences and can complete a whole line in a couple of keystrokes.

  • Takeaways & Limitations

    Online surfacing rate depends partly on user typing speed and network reliability, and telemetry may understate users’ experienced surfacing rate.

Abstract

from arXiv · show

In software development through integrated development environments (IDEs), code completion is one of the most widely used features. Nevertheless, majority of integrated development environments only support completion of methods and APIs, or arguments. In this paper, we introduce IntelliCode Compose $-$ a general-purpose multilingual code completion tool which is capable of predicting sequences of code tokens of arbitrary types, generating up to entire lines of syntactically correct code. It leverages state-of-the-art generative transformer model trained on 1.2 billion lines of source code in Python, $C\#$, JavaScript and TypeScript programming languages. IntelliCode Compose is deployed as a cloud-based web service. It makes use of client-side tree-based caching, efficient parallel implementation of the beam search decoder, and compute graph optimizations to meet edit-time completion suggestion requirements in the Visual Studio Code IDE and Azure Notebook. Our best model yields an average edit similarity of $86.7\%$ and a perplexity of 1.82 for Python programming language.

1 INTRODUCTION

Existing completion systems mainly recommend methods, APIs, or arguments, while IntelliCode Compose generates arbitrary code-token sequences and can complete whole lines. The system combines GPT-C with multilingual training and deployment optimizations for edit-time use.

  • Motivation: Most existing systems recommend method or API calls and arguments, often after the method name is already typed.This leaves method-call completion to developers.
  • Contribution: IntelliCode Compose generates arbitrary token types, including variables, methods, APIs, arguments, punctuation, keywords, and delimiters.It is designed to generate syntactically correct code across multiple programming languages.
  • Model: GPT-C is a multilayer generative transformer trained from scratch on a large unsupervised multilingual source-code dataset.The paper compares multilingual and monolingual models with an n-gram baseline.
  • Results: 86.7% average edit similarity and 1.82 perplexity were achieved for Python by the best model.These metrics are reported as the paper’s best model results.
  • Data: Over 1.2 billion lines of Python, C#, JavaScript, and TypeScript source code were collected from more than 52,000 GitHub projects.The dataset contains over 4.7 million source-code files.
  • Deployment: The system uses client-side caching and optimized decoding and computation to support edit-time completion.The deployed model is retrained using the entire dataset.

5 PREPROCESSING

The preprocessing pipeline normalizes source code, tokenizes it into subtokens and structural control tokens, and avoids high-level program representations during inference. It also addresses vocabulary coverage and sensitive literals in mined source code.

  • Representation: The system uses lexical tokens and does not use ASTs, CSTs, or control-flow graphs during inference.The paper cites added overhead, dependencies, reduced coverage, and the need for syntactically complete snippets as reasons.
  • Normalization: Custom tokenizers normalize styles, extract token types and subtokens, and encode sequences for both training and inference.The normalized representation is regenerated with a common style.
  • Tokenization: Subtoken encodings reduce vocabulary storage and improve robustness to out-of-vocabulary identifiers, methods, and APIs.The paper considers BPE and identifier splitting based on casing conventions.
  • Structural tokens: Special tokens represent file boundaries, line endings, and Python indentation scope.The tokens include <BOF>, <EOF>, <EOL>, <INDENT>, and <DEDENT>.
  • Sensitive data: Mining large public code repositories can expose sensitive information embedded in string literals, comments, or configuration files.The paper presents this as a production-level data-ingestion concern.
  • Literal handling: Frequent numeric and string literals are preserved as typed literal tokens, while identifier names remain context-dependent.The exact retained counts and dataset percentiles are reported in Table 2.

6 MODEL TRAINING

The paper trains GPT-C models for code completion using distributed optimization, extensive hyperparameter tuning, and multilingual source-code data. Training uses scalable data parallelism and model configurations selected for monolingual and multilingual settings.

  • Distributed training: GPT-C training uses synchronous data-parallel distribution with local gradient accumulation, cosine learning-rate decay, and warm-up during initial epochs.These choices support convergence in the distributed training regime.
  • Distributed training: Tree-like allreduce gives logarithmic synchronization complexity, while the number of mini-batches decreases linearly with worker count.The resulting scaling model combines constant per-batch computation with logarithmic synchronization overhead.
  • Hyperparameter selection: The model architecture, tokenization, and training procedure expose numerous numerical and categorical hyperparameters for tuning.Examples include learning rate, transformer depth, embedding dimension, architecture, and source-code normalization.
  • Model configurations: The best monolingual GPT-C models use 24 layers, 16 attention heads, and a 50,000-token BPE vocabulary, while the multilingual model uses 26 layers and 60,000 subtokens.Both configurations use 16 attention heads.
  • Optimization: GPT-C is trained with Adam, weight decay, a 6.25×10^-5 base learning rate, cumulative batch size 128, and categorical cross-entropy loss.The learning rate decays by 0.98 per epoch.
  • Training evaluation: Figure 4 compares epoch duration against worker-GPU count and plots training loss across epochs for monolingual and multilingual models.Its top panel includes experimental, semi-empirical scaling, and ideal-scaling curves.

7 SEQUENCE DECODING

Sequence decoding represents possible code completions as a subtoken tree and uses beam search to rank candidate paths. Batched inference and cached transformer states reduce the computation required for real-time decoding.

  • Completion-tree decoding: Each inference call produces a vocabulary probability vector, which defines an N-ary subtoken tree rooted at the final context subtoken.A completion is a path from the root to a terminal node, with depth determined by the desired sequence length.
  • Inference evaluation: Table 5 compares inference speed across beam widths k, sequence lengths L, and beam-search configurations.The table is organized around search scenarios with different widths, lengths, and setup choices.
  • Beam search: Beam search aggregates candidates at each decoding step and retains the top k paths until a preset length or break token is reached.Break tokens include <EOL> and language-specific tokens that commonly precede line endings.
  • Efficient decoding: Batched beam search reduces model inference calls for a sequence of length L from L×k to L.The method aggregates the top k candidates before each batched decoding step.
  • Efficient decoding: Caching transformer attention keys and values from previous tokens speeds inference by 10%.The improvement is most apparent for large completion lengths L.

8 CLIENT-SIDE POST-PROCESSING

Client-side caching and post-processing make cloud-served completion responsive and usable in the editor. The system prunes cached completion trees as typing continues, applies confidence-based early stopping, and converts special tokens into printable suggestions.

  • Completion caching: A response time below 100 ms is necessary to avoid perceived delay in the user experience study.This requirement motivates client-side caching for cloud-based deployment.
  • Completion caching: After non-alphanumeric input, server suggestions are stored in a trie keyed by preceding code and traversed greedily by highest score.Character-level pruning lets the client update suggestions as typing continues.
  • Early stopping: Traversal stops when no child score reaches the parent score multiplied by the ratio R.This preserves accuracy when multiple suggestions have similar confidence.
  • Early stopping: The relaxation factor α controls completion length: lower values produce longer suggestions, while values near 1.0 produce shorter suggestions.The curvature factor κ controls how quickly R increases.
  • Suggestion processing: The client ignores <BOF> and <EOF>, truncates at <EOL>, and replaces <STR_LIT> and <NUM_LIT> with default literals.Visual Studio Code can expose placeholders for navigation with the TAB key.
  • Suggestion processing: A trie stores strings as paths whose nodes represent substrings.This data structure supports the cached completion-tree representation.

9 MULTILINGUAL MODEL

The paper develops multilingual GPT-C variants for Python, C#, JavaScript, and TypeScript, comparing language-conditioning strategies and selecting MultiGPT-C for deployment. Language-specific control codes and an auxiliary classification objective provide the selected model’s multilingual supervision.

  • 9 MULTILINGUAL MODEL: Multilingual GPT-C models share a sub-token vocabulary across Python, C#, JavaScript, and TypeScript.The models are trained using a shared vocabulary extracted with BPE tokenization.
  • 9 MULTILINGUAL MODEL: The language-agnostic baseline underperforms significantly relative to monolingual models for each programming language.This approach disregards language-type information during training.
  • 9 MULTILINGUAL MODEL: Language-type embeddings add a learned language representation to token and position embeddings during the forward pass.The combined initial representation is h0 = We·C + Wp + Wl.
  • 9 MULTILINGUAL MODEL: Language-specific control codes prefix each training sample with its programming language to constrain generation.Prefixes identify Python, C#, JavaScript, or TypeScript sequences.
  • 9 MULTILINGUAL MODEL: MultiGPT-C combines language modeling with multiple-choice language classification and is selected as the multilingual deployment candidate.The classification objective provides additional supervision beyond token prediction.

10 EVALUATION

The evaluation measures pretrained language-model quality, completion similarity, syntactic correctness, and online usage. Results show strong Python performance, broadly comparable multilingual edit similarity, improvements for JavaScript and TypeScript, and telemetry affected by user behavior and networking.

  • 10 EVALUATION: Perplexity evaluates GPT-C pretraining, while ROUGE and Levenshtein edit similarity evaluate offline completion quality.Lower perplexity corresponds to higher probability assigned to true tokens; edit similarity captures approximate matches developers may accept.
  • 10 EVALUATION: Syntax errors are assessed by parsing file-level context together with each generated completion using tree-sitter.The experiment removes the end-of-line token from beam-search break tokens because completions may be partial statements.
  • 10 EVALUATION: Python achieves the best monolingual validation performance for edit similarity and ROUGE-L precision and recall.The paper relates this result to the naturalness and predictability of conventional, familiar Python code.
  • 10 EVALUATION: The multilingual model has comparable edit similarity and ROUGE-L precision but significantly lower ROUGE-L recall for C#.For JavaScript and TypeScript, all metrics improve with the multilingual model.
  • 10 EVALUATION: Over 150,000 requests produced a 9.2% surfacing rate and a 10% click-through rate.These figures correspond roughly to suggestions shown every 11 characters and users committing displayed completions.
  • 10 EVALUATION: Online surfacing rate depends on model accuracy, typing speed, and network reliability, while typing momentum lowers observed click-through.Users often type past several characters before examining or committing a suggestion.

11 KNOWLEDGE DISTILLATION

Knowledge distillation reduces GPT-C depth to accelerate inference while trading away some completion quality. Distilling the 26-layer teacher to 12 or 8 layers produces progressively larger speedups and metric losses.

  • 11 KNOWLEDGE DISTILLATION: Knowledge distillation trains smaller student models to reproduce a larger teacher’s outputs, targeting lighter and faster inference.The paper reduces transformer blocks while retaining the block architecture and embedding layers.
  • 11 KNOWLEDGE DISTILLATION: The experiments use 8- and 12-layer students initialized from a pretrained 26-layer teacher.The teacher supplies pretrained weights and biases for student initialization.
  • 11 KNOWLEDGE DISTILLATION: 2.7× inference speedup from 26 to 12 layers costs 6% edit similarity and 5% ROUGE-L precision.These results are reported for JavaScript and TypeScript relative to the monolingual teacher.
  • 11 KNOWLEDGE DISTILLATION: 4.5× inference speedup from 26 to 8 layers costs 8% edit similarity and 9% ROUGE-L precision.The more aggressive compression yields a larger speedup and larger quality losses.

12 MODEL DEPLOYMENT

IntelliCode Compose uses a two-layer cloud service that separates server-side model inference from a client-side completion provider. This architecture supports centralized hardware control and client-side interaction with the web service.

  • 12 MODEL DEPLOYMENT: The service separates server-side model inference from a client-side completion provider to minimize inference time.The server runs the model, while the client monitors inputs, communicates with the service, and post-processes outputs.
  • 12 MODEL DEPLOYMENT: The server module runs as a containerized Python web application on Azure Kubernetes Service.It exposes an HTTPS endpoint and performs inference with PyTorch and ONNX Runtime.
  • 12 MODEL DEPLOYMENT: Graph-level optimizations include constant folding and operator fusion for layer-normalization and GELU subgraphs.These optimizations are applied in the server-side inference module.
  • 12 MODEL DEPLOYMENT: The client-side completion provider is a TypeScript Visual Studio Code extension.It monitors user input and handles communication with the cloud service and output post-processing.

13 RELATED WORK

Prior code-completion systems mainly targeted specific completion types, while IntelliCode Compose extends sequence completion toward longer, broader code suggestions. Related systems include statistical, neural, AST-based, and ranking approaches, with evaluation also comparing model sizes and multilingual settings.

  • Related approaches: Prior intelligent completion research covered statically and dynamically typed languages using BMN, n-gram, and RNN-based approaches.These approaches leveraged the sequential nature of source code.
  • Evaluation context: The evaluation includes monolingual, multilingual, zero-shot, pretrained GPT-2, and distilled models across multiple programming languages and model sizes.The reported tables separately evaluate multilingual performance by programming language and compare distilled models with their teacher.
  • Existing code completion systems: AST-based neural completion and static-analysis ranking approaches targeted practical IDE deployment while improving computational speed and memory efficiency.The AST-based system was deployed in the IntelliCode extension.
  • Sequence completion: Tabnine used GPT-2 to rank code-sequence suggestions but did not target sequences of 20–30 characters or whole-line completion.The authors report no awareness of another currently deployed tool doing so.

14 CONCLUSIONS

IntelliCode Compose is a deployed, general-purpose code-completion system that generates syntactically correct sequences across token types and programming languages. Built around GPT-C, it achieves strong Python results while addressing practical training, deployment, latency, and multilingual-modeling challenges.

  • CONCLUSIONS: IntelliCode Compose generates syntactically correct code sequences across arbitrary token types and multiple programming languages, including whole-line completions.It is presented as a deployed general-purpose AI-powered completion system.
  • CONCLUSIONS: GPT-C is a multi-layer generative pretrained transformer for code, trained from scratch on source-code data as a GPT-2 variant.The system is built around GPT-C.
  • CONCLUSIONS: 86.7% average edit similarity and 1.82 perplexity are reported for Python by the best model.The system also targets at most 100 ms per-call inference through deployment and caching work.
  • Future work: Future work targets completion personalization, fine-tuning on custom user code, automatic program repair, and code search.These directions extend source-code language-model pretraining beyond code completion.
Loading 2005.08025v2…