PaperlayerView exact revision on arXiv

Source-linked AI summary

InCoder: A Generative Model for Code Infilling and Synthesis

Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, Mike Lewis

arXiv:2204.05999v3cs.SEcs.CLcs.LG

TL;DR

Existing code generators are primarily left-to-right, limiting direct code infilling and editing despite the importance of repeated code refinement. InCoder trains an autoregressive model with causal masking to move masked spans to the sequence end, enabling bidirectional-context infilling alongside ordinary synthesis. It substantially improves zero-shot infilling over left-to-right-only approaches while retaining comparable left-to-right program-synthesis performance, with limitations in evaluation scope and metrics noted by the authors.

  • Problem

    Left-to-right code models are less directly applicable to editing tasks requiring bidirectional context, including bug fixing, comment generation, and variable renaming.

  • Method

    InCoder randomly replaces code spans with sentinel tokens, moves them to the sequence end, and trains on the resulting sequences for autoregressive infilling.

  • Results

    Zero-shot infilling with bidirectional context substantially outperforms left-to-right-only approaches on challenging tasks while retaining comparable performance on standard program-synthesis benchmarks.

  • Takeaways & Limitations

    A single causal-masked model supports both zero-shot code editing through infilling and standard left-to-right program synthesis.

  • Takeaways & Limitations

    Multi-line infilling performance increases with more right context because fewer removable lines make those examples easier to infill.

Abstract

from arXiv · show

Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. The InCoder models and code are publicly released. https://sites.google.com/view/incoder-code-models

1 INTRODUCTION

InCoder unifies left-to-right program synthesis with code editing by training an autoregressive model to infill masked spans using bidirectional context. Zero-shot infilling improves challenging editing tasks while preserving comparable left-to-right synthesis performance.

  • Existing code models generate left-to-right, limiting their direct applicability to editing tasks such as bug fixing, comment addition, and variable renaming.
  • InCoder randomly masks code spans, moves them to the document end, and trains on the resulting permuted sequences.Special sentinel tokens mark the original span locations.
  • The same model supports program synthesis without sentinel tokens and code editing through infilling.
  • Zero-shot infilling with bidirectional context substantially outperforms left-to-right-only approaches across tasks including type prediction, variable renaming, comment generation, and missing-line completion.On several tasks, it reaches performance comparable to state-of-the-art models fine-tuned for those tasks.
  • Causal masking preserves left-to-right generation ability, with similar performance to standard language models on program-synthesis benchmarks.

2 INFILLING AND SYNTHESIS VIA CAUSAL MASKING

Causal masking converts span infilling into autoregressive generation by moving masked text to the sequence end, enabling conditioning on both left and right context. At inference, sentinel tokens support arbitrary code insertion while ordinary prompting retains left-to-right generation.

  • Causal models support autoregressive document generation but cannot directly infill, whereas masked models use both left and right context for masked regions.
  • Training: Training samples one or more contiguous spans, replaces them with sentinel tokens, and moves each span to the document end.The number of spans follows a truncated Poisson distribution with mean one, typically producing few spans but allowing up to 256.
  • Training: The training objective maximizes the autoregressive log probability of the permuted sequence while excluding mask sentinel tokens from the cross-entropy loss.
  • Inference: At inference, inserting sentinel tokens at desired locations lets the model generate replacement code until an end-of-mask token or task-specific stopping criterion.
  • Inference: Without sentinel tokens, the model performs standard left-to-right generation; with them, it infills code using bidirectional context.

3 MODELS

InCoder-6.7B is the primary model, trained on permissively licensed code and StackOverflow content spanning Python and 28 total programming languages. In practice, inference uses an extra sentinel token to counter a context-window-induced infill-length bias.

  • The primary model is INCODER-6.7B, a 6.7B-parameter Transformer language model.
  • The training corpus combines permissively licensed public code from GitHub and GitLab with StackOverflow questions, answers, and comments.
  • The corpus focuses on Python while including code from 28 total languages and StackOverflow content from all available languages.
  • Inference uses an extra sentinel token to counter an infill-length bias caused by the Transformer's fixed context-window size.

4 INFILLING EXPERIMENTS

InCoder is evaluated on zero-shot code-infilling tasks spanning code completion, docstring generation, return-type prediction, and variable renaming. Across these settings, causal-masked infilling benefits from right-sided context and outperforms left-to-right baselines, while approaching fine-tuned performance on docstring generation.

  • Evaluation setup: The benchmark covers zero-shot insertion of code lines, return-type hints, docstrings, variable names, and missing code tokens.Tasks are formulated as filling one or more masked-out regions of code.
  • Evaluation setup: The three inference methods compare causal-masked infilling with single-candidate and reranked left-to-right generation using the same InCoder-6.7B model.The shared model avoids confounding performance differences caused by changing the underlying model.
  • Infilling lines of code: CM infilling substantially improves function test pass rate and exact match over both left-to-right baselines on single-line and multi-line HumanEval infilling.The benchmark averages examples across masked positions throughout each function.
  • Infilling lines of code: CM infilling’s gains over left-to-right baselines increase as more right-sided function context becomes available.Figure 2 reports pass rate by the fraction of function lines provided to the right, with 95% bootstrap confidence intervals.
  • Docstring generation: Zero-shot InCoder docstring generation approaches pretrained code models fine-tuned on 250K task examples, using BLEU evaluation.The comparison uses CodeXGLUE Python docstring generation and reports smoothed 4-gram BLEU scores.
  • Return type prediction: Causal-masked infilling substantially improves return-type prediction over left-to-right inference and outperforms the supervised TypeWriter model.The evaluation covers two datasets and benefits from conditioning on the function body.
  • Variable name prediction: Variable renaming benefits from right context: reranking uses it for selection, while causal-masked infilling uses it for both proposing and selecting names.Both approaches outperform the left-to-right single baseline, and causal-masked infilling substantially outperforms left-to-right reranking.

5 ABLATION EXPERIMENTS

The ablations examine how causal masking, model size, and training data affect left-to-right synthesis performance. Larger models and StackOverflow data improve results, while causal masking does not hurt standard generation at the 1.3B scale.

  • Objective: 1.3B causal-masked models slightly outperform standard left-to-right models on HumanEval and MBPP pass@1.Both models use the same training data.
  • Model size: Larger models consistently improve performance when training data is fixed.This pattern appears for both causal-masked and standard language-modeling objectives.
  • Effects of data: Training on multiple languages slightly reduces performance on the Python evaluations compared with more Python-focused data.The comparison uses the multi-language plus StackOverflow setting against Python-focused ablations.
  • Effects of data: Including StackOverflow data substantially improves performance on both HumanEval and MBPP.The comparison is between rows 4 and 5 of the ablation study.

6 QUALITATIVE EXAMPLES

The qualitative examples illustrate InCoder's infilling and left-to-right generation capabilities across code editing, metadata conditioning, and technical-language tasks.

  • Examples include docstring generation, metadata conditioning, and inferring class attributes from class usage.
  • The model also demonstrates comment-conditioned code editing, StackOverflow title and tag generation, and Chinese-English technical-jargon translation.

7 RELATED WORK

Related work spans code language models, infilling systems, code-assistance methods, and program synthesis. InCoder combines decoder-only generation with arbitrary-span infilling through causal masking.

  • Language Models for Code: Decoder-only code models support zero-shot program synthesis through left-to-right generation, while InCoder additionally infills arbitrary spans.
  • Infilling Models: Real-world editing tasks require left and right context, but standard left-to-right models cannot directly infill and masked models mainly target short spans.
  • Machine Learning for Code Assistance: Code-assistance research includes type inference, test generation, program repair, and correctness verification.
  • Machine Learning for Program Synthesis: Program synthesis systems generate programs from specifications such as examples, partial implementations, or natural-language descriptions.
  • Table 5 compares models by parameter size, training objective, training data, and compute across validation, HumanEval, and MBPP results.

8 CONCLUSION

The conclusion reports strong zero-shot performance on practical code infilling and editing tasks while retaining comparable left-to-right synthesis ability. It identifies scaling, fine-tuning, supervised editing, and iterative decoding as future directions.

  • Causal masking enables strong zero-shot performance on challenging code infilling and editing tasks.
  • Ablation and comparison experiments find comparable left-to-right synthesis performance to similarly resourced models.
  • The authors expect performance to increase with more parameters, data, and training steps.
  • Fine-tuning is presented as a route to better conditioning on natural-language instructions and human intent.
  • The model provides a foundation for supervised infilling and editing and for iterative decoding that refines its own output.

A.1 CODE DATA

The corpus combines permissively licensed code from public repositories with StackOverflow content, then applies deduplication, decontamination, and quality filtering. Metadata attributes support conditioned generation and prediction across code and questions.

  • Sources: The corpus contains permissively licensed code from GitHub and GitLab repositories, covering 28 languages alongside StackOverflow questions, answers, and comments.The primary focus is Python, while the collection also includes JavaScript, Jupyter Notebook files, and other languages.
  • Deduplication: Deduplication removes files with exactly matching alphanumeric-token sequences, reducing the corpus from 1 TB to 250 GB.The matching scheme uses file extension, token count, and an MD5 hash in a Bloom filter.
  • Decontamination: Decontamination removes repositories overlapping validation and test sets used by CodeSearchNet and related CodeXGLUE evaluations.This reduces overlap between pretraining data and benchmark evaluation data.
  • Filtering: Filtering excludes unusually long, low-alphanumeric, or apparently automatically generated files.The filters target both structural anomalies and files containing phrases associated with automatic code or documentation generation.
  • Metadata: Metadata includes filenames, extensions, repository sources, star-count buckets, StackOverflow tags, and vote-count buckets for attribute-conditioned generation and prediction.Attributes are arranged so they can be placed at either document end, while StackOverflow comments follow the questions or answers they annotate.
  • Corpus statistics: Figure 3 summarizes the post-deduplication and post-filtering corpus by total file size for common languages identified by file extension.The figure provides the corpus composition view used to characterize language coverage.

B.1 MODEL

INCODER is a Transformer language model trained with causal masking that moves randomly masked spans to the sequence end, enabling infilling while retaining left-to-right generation. Its evaluations examine model scaling, synthesis, and several infilling configurations and comparisons.

  • Model: INCODER-6.7B is a 6.7B-parameter Transformer trained for 24 days on 248 V100 GPUs.The model uses the dense 6.7B architecture described by Artetxe et al.
  • Inference: The infilling distribution uses left context, a mask sentinel, right context, and a final mask sentinel to indicate whether omitted content follows.The extra sentinel provides a size hint for the generated region within the context window.
  • CodeXGLUE cloze: On the CodeXGLUE max/min cloze task, masking a larger region with right-side context performs comparably to scoring the entire sequence left to right.A single-token causal-masked infill improves over left context alone but does not match full left-to-right scoring.
  • Tokenization: Breaking tokenization around the infilled token decreases performance across scoring methods.Using the original tokenization improves infilling slightly, but still does not match full left-to-right scoring.
  • Type prediction: Zero-shot infilling outperforms left-to-right baselines on the TypeWriter prediction set and becomes comparable to supervised TypeWriter when return checks are used.The return-check postprocessing predicts None when functions lack non-trivial return statements.

C.6 COMPARISON TO LEFT-TO-RIGHT GENERATIVE MODELS ON CODE SYNTHESIS

INCODER-6.7B is evaluated on standard HumanEval and MBPP program-synthesis benchmarks using zero-shot prompting and sampling-based pass-rate metrics. Its HumanEval performance is roughly comparable to similarly scaled left-to-right models, while its demonstrations cover diverse conditioned generation tasks.

  • Benchmarks: HumanEval and MBPP evaluate whether models can generate Python programs from natural-language descriptions, using functional test cases to measure pass rates.HumanEval prompts include function signatures and docstrings, while MBPP prompts use docstrings without function signatures.
  • Evaluation: INCODER-6.7B is evaluated zero-shot on HumanEval with 200 sampled completions and on MBPP with one sampled candidate.HumanEval reports pass@1, pass@10, and pass@100; MBPP reports pass@1.
  • Benchmark comparison: INCODER-6.7B achieves roughly comparable HumanEval performance to CodeGen-Multi, a similarly sized model trained on a similar amount of Python code.The comparison also includes AlphaCode’s 1.1B decoder-only model with a similar amount of Python training data.
  • Comparison caveat: The authors caution that differences in architectures, datasets, training procedures, and experimental setups make rigorous cross-paper comparison impossible.The published comparison therefore provides a broad performance reference rather than a controlled experiment.
  • Conditioned generation: The model demonstrates attribute-conditioned left-to-right generation, including producing Python or Shell completions from the same comment when file-extension metadata differs.Other examples show docstring generation, attribute inference, interactive refinement, question metadata prediction, and Chinese-English jargon translation.
Loading 2204.05999v3…