PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatically Generating Commit Messages from Diffs using Neural Machine Translation

Siyuan Jiang, Ameer Armaly, Collin McMillan

arXiv:1708.09492v1cs.SEcs.CL

TL;DR

Commit messages support program comprehension and maintainability, but programmers often neglect them, while existing generators focus on what changed rather than why. This paper generates short rationale-focused messages from software changes and uses quality assurance to identify low-quality outputs; results show both close matches and substantial noise.

  • Problem

    Commit messages help program comprehension and maintainability by summarizing what changed and why, but programmers often neglect writing them, and prior research focused exclusively on what changed.

  • Method

    The approach generates short messages summarizing high-level rationale across software artifacts and uses a QA filter that labels messages with human-study scores of zero or one as bad.

  • Results

    BLEU scores for NMT2 were 32.81 on Test1 and 23.10 on Test2, while human evaluation found many messages closely matching references alongside substantial low-quality output.

  • Takeaways & Limitations

    The method supplements existing generators with concise, high-level rationale summaries and can flag cases where it does not produce good messages.

  • Takeaways & Limitations

    The approach was evaluated only on Java projects in Git repositories, which may not represent all commits.

Abstract

from arXiv · show

Commit messages are a valuable resource in comprehension of software evolution, since they provide a record of changes such as feature additions and bug repairs. Unfortunately, programmers often neglect to write good commit messages. Different techniques have been proposed to help programmers by automatically writing these messages. These techniques are effective at describing what changed, but are often verbose and lack context for understanding the rationale behind a change. In contrast, humans write messages that are short and summarize the high level rationale. In this paper, we adapt Neural Machine Translation (NMT) to automatically "translate" diffs into commit messages. We trained an NMT algorithm using a corpus of diffs and human-written commit messages from the top 1k Github projects. We designed a filter to help ensure that we only trained the algorithm on higher-quality commit messages. Our evaluation uncovered a pattern in which the messages we generate tend to be either very high or very low quality. Therefore, we created a quality-assurance filter to detect cases in which we are unable to produce good messages, and return a warning instead.

I. INTRODUCTION

Commit messages support software-evolution comprehension but are often neglected, while existing generators mainly describe what changed rather than why. This paper adapts NMT to learn short, high-level rationale from diffs and evaluates a QA filter for unreliable predictions.

  • The paper trains an NMT algorithm on over 2M pairs of diffs and human-written commit messages from 1k popular GitHub projects.
  • Commit messages record feature additions and bug repairs while helping programmers understand a change’s high-level rationale.
  • Existing generators effectively summarize what changed and where, but generally lack short, high-level descriptions of purpose.
  • A V-DO filter selects higher-quality commit messages matching an acceptable verb/direct-object pattern for NMT training.
  • The approach produces short summaries across many software-artifact types, complementing techniques that generate exhaustive code-change descriptions.
  • A QA filter replaces likely poor predictions with warnings, reducing poor predicted messages by 44% while mistakenly removing 11% of high-quality predictions.

II. RELATED WORK

Prior work generates commit messages from code changes or related documents, while this technique translates diffs directly into natural-language sentences. It complements code-change summarizers by producing one-sentence descriptions for both code and non-code changes.

  • Existing commit-message techniques use code changes, related software documents, or combinations of both as inputs.
  • This technique uses git diff outputs as inputs and translates diffs into natural-language commit-message sentences.
  • Compared with code-change summarizers, it generates one-sentence headlines rather than multi-line summaries containing pseudocode and template text.
  • The technique summarizes both code and non-code changes appearing in diffs.
  • Unlike Code-NN, which summarizes code snippets, this work targets changes represented at the diff level.

III. BACKGROUND

Commit messages are pervasive, typically short, and contain both descriptions of changes and explanations of why they were made. The paper uses attentional RNN Encoder-Decoder NMT to model these learned message patterns.

  • Studies found that 99.1% of sampled commits had non-empty messages, while another corpus contained over 2M messages from 1k projects.
  • Human-written commit messages are short: one study reported an average of 1.1 lines, and another found 82% contained one sentence.
  • Commit messages contain information about both what changed and why the change was made.
  • NMT models translation as conditional probability from a source sequence to a target sequence using encoder and decoder recurrent neural networks.

1) Encoder:

The encoder reads a variable-length source sequence and produces hidden representations, while the decoder uses the encoded context to generate a target sequence and its symbol probabilities. Both networks are jointly trained, with attention supporting long diffs.

  • Encoder: The encoder reads a variable-length source sequence one symbol at a time and updates a fixed-length hidden state.
  • Encoder: A nonlinear recurrent function computes each encoder state, and the final state represents the complete input sequence.
  • Decoder: The encoder-generated final state serves as the decoder’s context vector, while the decoder stops after predicting an end-of-sequence symbol.
  • Decoder: The decoder begins with a start symbol and generates the target sequence by computing hidden states and next-symbol probabilities.
  • Training: Encoder and decoder parameters are jointly trained to maximize conditional log-likelihood over source–target training pairs.
  • Attention: Attention is used because diffs are long, allowing the model to incorporate context when generating target symbols.

1) Encoder:

The approach preprocesses diff–message pairs and uses an attentional NMT model whose bidirectional encoder represents input sequences in both directions. Data preparation removes noisy or unsuitable commits before tokenization.

  • 1) Encoder:: The bidirectional RNN encoder reads the input sequence forward and backward, generating hidden states from both directions.The forward RNN reads x_1 through x_T, while the backward RNN reads the sequence in reverse.
  • 1) Encoder:: The decoder computes each hidden state from its previous state, previous output symbol, and a context vector.The context vector is designed to introduce the input context's impact on the next output symbol.
  • 1) Encoder:: The context vector is a weighted combination of encoder hidden states, with weights trained jointly with the other model components.The input length is T, and h_i is generated by the encoder.
  • 1) Encoder:: The dataset preparation uses first commit-message sentences as targets, removes unique issue and commit identifiers, and excludes merges, rollbacks, and diffs larger than 1MB.These steps reduce vocabulary growth and avoid translating very large diffs that NMT is not suited to handle.
  • 1) Encoder:: After filtering, 1.8M commits remained, and extracted messages and diffs were tokenized by whitespace and punctuation without splitting CamelCase identifiers.Identifiers such as class and method names were treated as individual words.

2) Setting Maximum Sequence Lengths for NMT Training:

The authors set separate maximum lengths for diff inputs and commit-message targets because their sequence lengths differ substantially. Pilot results favored a 100-token source limit, while filtering ultimately left 75k commits for modeling.

  • 2) Setting Maximum Sequence Lengths for NMT Training:: The source and target sequences receive separate maximum lengths because their typical lengths differ substantially.The paper notes that standard NMT sequence limits are often 50 to 100 tokens.
  • 2) Setting Maximum Sequence Lengths for NMT Training:: 30 tokens is the maximum target length because 98% of first commit-message sentences contain fewer than 30 tokens.The target count includes words and punctuation.
  • 2) Setting Maximum Sequence Lengths for NMT Training:: 100 tokens is the maximum source length, although optimizing the maximum diff length remains future work.The authors report that this setting outperformed source limits of 50 and 200 tokens in pilot studies.
  • 2) Setting Maximum Sequence Lengths for NMT Training:: 75k commits remained after applying the 30-token target and 100-token source limits.These limits were applied after earlier dataset filtering and preprocessing.
  • 2) Setting Maximum Sequence Lengths for NMT Training:: The V-DO filter addresses inconsistent and poorly written messages by selecting commit messages with a verb/direct-object pattern.The pattern was chosen because a previous study found it in 47% of commit messages.
  • 2) Setting Maximum Sequence Lengths for NMT Training:: 32k commit messages were identified as “dobj” sentences, and the data were split into 3k test, 3k validation, and 26k training commits.The split was random.

5) Selecting Vocabularies:

The training setup uses selected diff and message vocabularies with an attentional RNN encoder–decoder, standard optimization and validation procedures, and an ensemble of saved models for evaluation.

  • 5) Selecting Vocabularies:: The training vocabulary contains all 16k distinct message tokens and the 50k most frequent diff tokens.Tokens outside the diff vocabulary occurred only once in the training set.
  • 5) Selecting Vocabularies:: Nematus implements the attentional RNN encoder–decoder used to train the commit-message generation model.The authors selected Nematus because it was robust, easy to use, and performed well in a WMT 2016 comparison.
  • 5) Selecting Vocabularies:: Training minimizes cross-entropy using stochastic gradient descent with Adadelta, minibatches of 80, 512-dimensional embeddings, and 1024-unit hidden layers.Adadelta automatically adapts the learning rate.
  • 5) Selecting Vocabularies:: The model is validated every 10k minibatches by BLEU, uses early stopping, and evaluates an ensemble of the last four saved models.Models are saved every 30k minibatches, with maximum limits of 5k epochs and 10M minibatches.
  • 5) Selecting Vocabularies:: The training used 26k pairs with 3k validation pairs, stopped at 210k minibatches, took 38 hours, and saved seven models.Training ran on an Nvidia GeForce GTX 1070 with 8GB memory.
  • 5) Selecting Vocabularies:: Testing used the last four saved models as an ensemble on a standard 3k-example test set and took 4.5 minutes.The same GPU was used for testing and training.

A. Baseline: MOSES

MOSES serves as the baseline for translating diffs into commit messages, using a 3-gram language model trained with KenLM. BLEU measures generated–reference similarity over the test set.

  • A. Baseline: MOSES: MOSES is the baseline system because it is widely used for evaluating machine-translation systems.The authors use it specifically for RQ1's comparison with the NMT model.
  • A. Baseline: MOSES: The baseline translates diffs to commit messages with a 3-gram language model trained using KenLM.This follows the procedure used in prior work by Iyer et al.
  • A. Baseline: MOSES: BLEU evaluates similarity between generated messages and reference messages using modified n-gram precisions.The metric is intended for assessing an entire test set rather than an individual sentence.
  • A. Baseline: MOSES: BLEU scores range from 0 to 100 percent, and the evaluation uses the default maximum n-gram order of 4.The brevity penalty depends on generated and reference lengths.

C. RQ1: Compared to the Baseline

NMT1 substantially outperformed the MOSES baseline on BLEU, with performance varying by diff length. An unfiltered model improved on V-DO-style commits but performed worse on commits outside that pattern.

  • 31.92 BLEU versus 3.63 for MOSES shows that NMT1 generated messages more similar to reference messages.The paper attributes MOSES's weaker performance partly to difficulty handling very long source sequences with short target sequences.
  • Diffs longer than 75 tokens achieved the highest BLEU score among the tested length groups.The authors suggest this may reflect the larger number of long diffs in the dataset.
  • Modified 4-gram precision increased from 7.6 for 25–50-token diffs to 42.3 for diffs longer than 75 tokens.For other length ranges, p4 increased more modestly, from 3.1 to 4.5 and from 4.5 to 7.6.

2) Results:

The results examine how performance relates to diff length and evaluate generated messages through human judgments of semantic similarity. The survey used programmers and students who scored message pairs without knowing their source.

  • The test-set analysis compares BLEU performance across groups of diffs with different lengths.The corresponding figure shows the distribution of diff lengths in the test set.
  • The training-set analysis reports the distribution of diff lengths used to train the model.The paper presents this distribution alongside the test-set distribution when discussing length-based BLEU results.
  • The human evaluation complemented BLEU by measuring semantic similarity at the individual-message level.The authors note that BLEU measures textual similarity and is not recommended for evaluating individual sentences.
  • 20 participants evaluated generated/reference message pairs for similarity on a 0–7 scale during 30-minute survey sessions.Participants included two computer science Ph.D. students and 18 professional programmers with 2 to 14 years of experience.
  • Survey pages presented one message pair at a time in random order, with an optional box for participants to justify their scores.Participants did not know who or what generated the messages, and qualitative analysis of comments was outside the study's scope.

C. Results

Human evaluation found that generated messages clustered at very high or very low similarity scores, motivating a QA filter that identifies likely poor predictions. The filter used human-study labels and diff features, then achieved moderate detection performance under 10-fold cross-validation.

  • Human Evaluation: 983 generated messages received human-study scores, with 248 scored 0 and 234 scored 7.Scores were conservative rounded-down medians on a 0–7 similarity scale.
  • QA Filter: The QA filter labeled scores 0 or 1 as bad and all higher scores as not bad.The labels came from evaluated messages and their corresponding diffs.
  • QA Filter: The filter represented diffs with term frequency/inverse document frequency features and trained a linear SVM using stochastic gradient descent.At prediction time, tf/idf features extracted from a diff were passed to the trained SVM.
  • Cross-Validation Evaluation: 10-fold cross-validation yielded 44.9% precision and 43.8% recall for detecting diffs associated with bad generated messages.Each fold trained on nine partitions and tested on the remaining partition.
  • Cross-Validation Evaluation: When scores 6 or 7 defined good messages, the QA filter reduced 44% of bad messages while removing 11% of good messages.The evaluation also showed reductions across the full 0–7 score range.

VIII. EXAMPLE RESULT

The paper frames its contribution as learning short, high-level rationale from prior repository history rather than generating insights for completely novel changes. An example illustrates this capability, while the evaluation and discussion identify important scope and data limitations.

  • Example Result: The NMT model generated a high-rated message for replacing deactivate() with close().The corresponding diff made the replacement evident by removing the deactivate() call and adding the close() call.
  • Threats to Validity: The study used only Java projects in Git repositories, which may not represent all commits.The authors identify extension to other programming languages as future work.
  • Threats to Validity: Human-written training messages may omit useful information that should appear in commit messages.Improving those human-written messages was outside the paper’s scope.
  • Threats to Validity: The human study had a limited number of participants, so the fairness of every final score could not be guaranteed.The study mitigated this partly by using multiple evaluators for many messages.
  • Discussion and Conclusion: The technique learns short commit messages from previously described repository changes rather than providing new insights for completely novel changes.The authors position it as automating messages whose rationale is represented in repository history.
  • Evaluation: The evaluation combined automated metrics with a human-evaluator experiment to verify and analyze generated messages.The authors also released the implementation and data for replication and further research.
  • Discussion and Conclusion: The NMT process performed well under selected conditions but poorly under others, producing both closely matching and low-quality messages.The authors created a QA filter to warn when the model could not generate a good message.
  • Discussion and Conclusion: The authors do not claim the work is definitive or complete, and expect future improvements to target commit types and change-type detection.They suggest that prediction quality may differ across software change types.
Loading 1708.09492v1…