PaperlayerView exact revision on arXiv

Source-linked AI summary

On Learning Meaningful Code Changes via Neural Machine Translation

Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabriele Bavota, Denys Poshyvanyk

arXiv:1901.09102v1cs.SEcs.CLcs.LG

TL;DR

The paper addresses the lack of evidence about which meaningful code changes DL can learn and apply. It trains an NMT model on before/after method pairs from pull requests and evaluates exact reproduction alongside transformation types. In a narrow setting, the model reproduces developer changes in up to 36% of cases and learns varied refactorings and bug fixes, while the evidence is limited to Java programs.

  • Problem

    The paper investigates which code changes DL can learn and automatically apply, addressing limited empirical and qualitative evidence.

  • Method

    The study mines reviewed pull requests, extracts before/after method pairs, abstracts and filters them, and trains NMT to translate pre-change methods into post-change methods.

  • Results

    Up to 36% of developer-implemented transformations are reproduced, with performance varying from 16% to 36% by beam-search output count.

  • Takeaways & Limitations

    The model learns a wide variety of meaningful transformations, especially refactorings and bug-fixing activities, supporting further research on automatic code-change learning.

  • Takeaways & Limitations

    The empirical evaluation uses Java programs only, although the authors state that the infrastructure can be instantiated for other programming languages.

Abstract

from arXiv · show

Recent years have seen the rise of Deep Learning (DL) techniques applied to source code. Researchers have exploited DL to automate several development and maintenance tasks, such as writing commit messages, generating comments and detecting vulnerabilities among others. One of the long lasting dreams of applying DL to source code is the possibility to automate non-trivial coding activities. While some steps in this direction have been taken (e.g., learning how to fix bugs), there is still a glaring lack of empirical evidence on the types of code changes that can be learned and automatically applied by DL. Our goal is to make this first important step by quantitatively and qualitatively investigating the ability of a Neural Machine Translation (NMT) model to learn how to automatically apply code changes implemented by developers during pull requests. We train and experiment with the NMT model on a set of 236k pairs of code components before and after the implementation of the changes provided in the pull requests. We show that, when applied in a narrow enough context (i.e., small/medium-sized pairs of methods before/after the pull request changes), NMT can automatically replicate the changes implemented by developers during pull requests in up to 36% of the cases. Moreover, our qualitative analysis shows that the model is capable of learning and replicating a wide variety of meaningful code changes, especially refactorings and bug-fixing activities. Our results pave the way for novel research in the area of DL on code, such as the automatic learning and applications of refactoring.

I. INTRODUCTION

This study investigates whether NMT can learn and reproduce meaningful code changes made during reviewed pull requests. It combines method-level before/after examples with quantitative exact-match evaluation and qualitative analysis of learned transformations.

  • Motivation: The paper addresses limited empirical evidence about which non-trivial code changes DL can learn and automatically apply.Prior work automated tasks including bug fixing, but qualitative investigation of output meaningfulness remained limited.
  • Data: 78,981 merged pull requests from three Gerrit repositories provide the source material for the study.The repositories cover Android, Google Source, and Ovirt, spanning 339 subprojects.
  • Data: 239,522 method pairs represent code before and after pull-request changes and serve as training examples.Fine-grained source-code differencing extracts method-level edit operations from the mined pull requests.
  • Method: The Encoder-Decoder RNN learns to translate pre-pull-request methods into their post-pull-request versions.The broader pipeline mines, pairs, abstracts, filters, trains on, and concretizes method transformations.
  • Evaluation: The model reproduces developers’ code transformations in 16-36% of cases, depending on the number of beam-search guesses.Evaluation compares model output with the manually implemented change for the same code component, while qualitative analysis develops a taxonomy of learned transformations.
  • Findings and scope: The learned transformations include meaningful refactorings and bug fixes, although the study is restricted to small and medium-sized existing methods.The paper excludes code changes involving creation or deletion of entire methods or files.

C. Code Abstraction & Filtering

The paper reduces source-code vocabulary through abstraction and filtering so NMT can learn transformations from method pairs. It retains reusable structure and frequent tokens while restricting examples to changes representable by the model and to methods of at most 100 tokens.

  • Motivation: Source-code vocabularies reach 42k-267k tokens per dataset and exceed 370k when datasets are combined.Novel identifiers and literals make source code’s open-vocabulary problem more pronounced than in natural-language text.
  • Abstraction: Identifiers and literals are replaced with reusable category-specific IDs, while the 300 most frequent values retain their original tokens.Lexing and Java parsing identify token roles and literal types, and a mapping preserves values for later concretization.
  • Abstraction: Reusing IDs across method pairs reduces the overall vocabulary to Java keywords, top-300 identifiers/literals, and reusable IDs.The reduced vocabulary is intended to help the model learn transformation patterns across contexts.
  • Filtering: The filtering step removes target methods containing tokens outside Java keywords, frequent values, or identifiers and literals already available in the input.This avoids requiring the model to synthesize unseen identifiers or literals.
  • Filtering and datasets: Abstracted pairs with identical before and after code are removed because they contain no learnable transformation.The datasets are then randomly split into 80% training, 10% validation, and 10% test sets after duplicate removal.
  • Filtering: Method pairs longer than 100 tokens are discarded after partitioning examples into small methods of at most 50 tokens and medium methods of 50-100 tokens.The study therefore focuses on short method-level transformations.

D. Learning Code Transformations

The model translates abstracted pre-PR methods into post-PR methods, using an attention-based encoder-decoder and beam search to generate multiple candidate transformations.

  • The NMT model translates abstracted code before a pull request into abstracted code after the change.Training uses paired sequences amb and ama representing pre-PR and post-PR methods.
  • An RNN encoder-decoder with attention encodes the input sequence and predicts target tokens through a learned conditional distribution.The encoder produces states, while attention weights different input parts during token prediction.
  • Beam Search Decoding: Beam search retains k highest-probability hypotheses at each step instead of selecting only the most probable token.Each hypothesis expands into |V| possible next-token candidates, producing k · |V| possible hypotheses before pruning.
  • Beam Search Decoding: The decoder continues until hypotheses reach the sequence-end token, treating the k final sequences as candidate patches.

3) Hyperparameter Search:

The study tested multiple encoder-decoder configurations and selected checkpoints using validation performance while avoiding overfitting.

  • Hyperparameter Search: Ten encoder-decoder configurations varied RNN cell type, layer count, unit count, and embedding size.The tested cells were LSTM and GRU; layers, units, and embedding sizes used the combinations reported by the authors.
  • Hyperparameter Search: Models were trained for up to 60k epochs, with checkpoints selected before over-fitting the training data.

learn meaningful code changes?

The evaluation tests whether NMT can learn developer-performed code transformations and reproduce them exactly, while also examining the kinds of meaningful changes it generates.

  • RQ1: The study trains and evaluates eight models, one for each method-pair dataset, using training, validation, and unseen test sets.Validation selects the best configuration before evaluation on unseen test instances.
  • RQ1: A prediction succeeds when one of k beam-search outputs exactly matches the developers’ post-PR abstract code.
  • RQ2: The qualitative analysis examines successfully predicted transformations to construct a taxonomy of meaningful code changes.Authors manually described changes, validated them, and iteratively refined the taxonomy.
  • RQ1: Generating multiple translations significantly increases perfect predictions across models, with the combined-dataset model averaging one successful transformation in three.Ten transformations can be generated in less than one second on a consumer-level GPU.
  • RQ1: Up to 21% of transformations are perfectly predicted with one translation, rising to up to 36% with 10 guesses.The combined All dataset reaches 21.16% for small methods and 16.21% for medium methods with a single guess.

performed by the model?

The model reproduced diverse transformations organized into bug-fixing, refactoring, and other categories, including several inheritance-related refactorings and combined changes.

  • Taxonomy: The taxonomy analyzes 722 perfect predictions produced by the All model with beam size 10, corresponding to 216 unique AST operation types.
  • Taxonomy: The taxonomy contains bug-fixing, refactoring, and other transformation sub-trees, with 793 classified changes because predictions may contain multiple change types.Twenty-seven transformations were excluded because the authors disagreed about their type.
  • Refactoring: Refactoring transformations alter internal structure to improve non-functional attributes without changing external behavior.
  • Inheritance: Inheritance-related refactorings add final, remove redundant super, or make methods abstract by deleting their bodies.
  • Inheritance: The model also learned changes such as adding final to an Android method and removing an unnecessary super specifier in Ovirt.
  • Inheritance: One prediction combined adding abstract with deleting a method body, reproducing a multi-operation developer change.The developer rationale was that all child classes override the method, so the abstract base class needs no implementation.

2) Methods Interaction:

The model learned refactorings that alter method interaction, naming, and visibility, including broader return types, renamed identifiers, and access-modifier changes.

  • Methods Interaction: Broader generic return types can avoid unchecked warnings and simplify code usage.The post method changes from RestModifyView<P,I> to RestModifyView<P,?>.
  • Naming: The model correctly learned renamings of methods and parameters intended to improve expressiveness and clarity.Examples include correcting OnSucess to OnSuccess and renaming type to method.
  • Methods Interaction: The model learned both broadening and narrowing method visibility through access-modifier changes.Broadening enabled external use for a new feature, while narrowing reflected a method used only in one external location where it was unnecessary.

5) Readability:

The model learned readability-oriented transformations, including lambda conversion and merging equivalent exception handlers, producing more compact or clearer code.

  • Readability: The learned readability changes included braces, statement merging, this-qualifier adjustments, diamond operators, redundant-else removal, and boolean-expression simplification.These transformations target clearer, less verbose, or more maintainable source code.
  • Readability: The model converted an anonymous class into a lambda expression, making the resulting Java 8 syntax more compact and readable.The transformation occurred during Ovirt’s transition to Java 8.
  • Readability: The model merged two catch blocks when both exceptions led to the same executed code.This transformation is meaningful when the caught exceptions have identical handling.

D. Bug Fix

The model learned bug-fixing transformations involving exception handling and conditional statements, including resource-safe blocks, narrowed try scopes, and null checks.

  • Exception: The model wrapped DatagramSocket creation and use in try-with-resources to avoid resource leakage.The construct automatically invokes close() or propagates an exception.
  • Exception: The model moved ConcurrentHashMap instantiation outside a try block so the test specifically exercised the expected NullPointerException.The original test could also pass if construction threw the exception.
  • Conditional statements: The model added a null check before accessing mCallback, replicating the developer’s fix for a NullPointerException.The callback is invoked only when mCallback is not null.

0 Change comparison operand [6]. public void reset(int i) {

The model learned bug fixes that change comparison operands, return values, synchronization usage, and method-invocation parameters.

  • Change comparison operand: Changing >= to > fixed FieldPacker.reset() so the buffer could point to its final entry.The model replicated this developer-implemented conditional fix.
  • Values: The model learned return-value changes used to fix bugs, from flipping boolean constants to returning a field holding the current value.One example replaces return "refs/my/config" with return ref.
  • Lock mechanism: The taxonomy also included synchronized-keyword changes, but these transformations were not discussed because of limited space.The changes included removing or relocating synchronized between method signatures and code blocks.
  • Methods invocation: The model swapped assertEquals arguments to correct a test that passed actual before expected.The intended parameter order is (long expected, long actual).

E. Other

The model learned a wide variety of meaningful code changes, predominantly refactorings and bug fixes, but the observed scope was constrained by limited method context and recurring transformation patterns.

  • The NMT model learned and automatically applied a wide variety of code changes, mostly involving refactoring and bug-fixing activities.
  • New feature implementations were not observed among the learned changes.
  • Limited method size and the rarity of recurring patterns across classes and systems may explain why new feature changes were absent.
  • The taxonomy was manually defined, double checked, and discussed by all authors to mitigate subjectivity bias.
  • Java-only data and pull requests from three Gerrit repositories limit the direct generalizability of the findings to other languages and projects.
  • The study focused on existing methods of no more than 100 tokens rather than creating new methods.

VI. RELATED WORK

Related work has applied deep learning broadly across software-engineering tasks, while this study focuses on learning code transformations from developers’ pull requests using neural machine translation.

  • Deep learning has been applied to code completion, naming, comments, defect prediction, bug localization and fixing, clone detection, code search, and API-template learning.
  • Prior work used representation learning and neural models for clone detection, code similarity, code suggestion, code generation, and software-risk prediction.
  • Deep learning has also supported bug localization, bug triaging, effort estimation, source-code summarization, and natural-language code search.
  • Other applications include method and class naming, API-usage generation, and automatic GUI generation for mobile applications.
  • This work applies neural machine translation to learn code transformations from real developer pull requests and create a meaningful taxonomy.
  • The models perfectly predicted code transformations in up to 21% of cases with one translation and up to 36% with 10 guesses.
Loading 1901.09102v1…