PaperlayerView exact revision on arXiv

Source-linked AI summary

ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking

Shangqing Liu, Cuiyun Gao, Sen Chen, Lun Yiu Nie, Yang Liu

arXiv:1912.02972v2cs.SE

TL;DR

Commit-message generation has to bridge code changes and natural-language descriptions, while prior approaches either omit AST structure or rely on retrieval. ATOM combines AST-based generation with retrieved candidates through hybrid ranking; experiments report improved BLEU-4 performance over state-of-the-art approaches.

  • Problem

    Prior commit-message generation does not explicitly use AST structure, while generation and retrieval each have limitations in relating code changes to messages.

    Source: §1 · p. 1

  • Method

    ATOM encodes ASTs for code changes and uses hybrid ranking to prioritize between generated and retrieved commit messages.

    Source: §1 · p. 2

  • Results

    ATOM increases state-of-the-art approaches by 30.72% in BLEU-4 and improves its generation module by 42.99% through hybrid ranking.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Quantitative and qualitative experiments, including human evaluation, demonstrate ATOM’s effectiveness and usefulness for commit-message generation.

    Source: §1 · p. 2

    ATOM requires more preprocessing and model tuning because extracting AST paths and running its generation and ranking pipeline are more complex and time-consuming.

    Source: §5.6.1 · p. 13

Abstract

from arXiv · show

Commit messages record code changes (e.g., feature modifications and bug repairs) in natural language, and are useful for program comprehension. Due to the frequent updates of software and time cost, developers are generally unmotivated to write commit messages for code changes. Therefore, automating the message writing process is necessitated. Previous studies on commit message generation have been benefited from generation models or retrieval models, but the code structure of changed code, i.e., AST, which can be important for capturing code semantics, has not been explicitly involved. Moreover, although generation models have the advantages of synthesizing commit messages for new code changes, they are not easy to bridge the semantic gap between code and natural languages which could be mitigated by retrieval models. In this paper, we propose a novel commit message generation model, named ATOM, which explicitly incorporates the abstract syntax tree for representing code changes and integrates both retrieved and generated messages through hybrid ranking. Specifically, the hybrid ranking module can prioritize the most accurate message from both retrieved and generated messages regarding one code change. We evaluate the proposed model ATOM on our dataset crawled from 56 popular Java repositories. Experimental results demonstrate that ATOM increases the state-of-the-art models by 30.72% in terms of BLEU-4 (an accuracy measure that is widely used to evaluate text generation systems). Qualitative analysis also demonstrates the effectiveness of ATOM in generating accurate code commit messages.

1 INTRODUCTION

The introduction motivates automated commit-message generation and presents ATOM as a model combining AST-based generation with retrieval and hybrid ranking. Its benchmark results and experiments support the approach’s effectiveness.

1 INTRODUCTION

  • Rule-based methods may miss cases, NMT treats code as a flat token sequence, and retrieval can perform poorly on programs unlike its database.The introduction illustrates retrieval returning a message unrelated to the code change.Source: §1 · p. 2
  • Hybrid ranking selects the most accurate message from generated and retrieved candidates to improve relevance to the code change.ATOM also includes a retrieval module that finds messages using text similarity.Source: §1 · p. 2
  • ATOM outperforms state-of-the-art approaches by 30.72% in BLEU-4 and improves its generation module by 42.99% through hybrid ranking.The evaluation uses a new benchmark of approximately 160k commits.Source: §1 · p. 2
  • AST2seq encodes ASTs from code changes to better capture code semantics in generated messages.The paper identifies AST as a code structure not explicitly used by prior commit-message generation studies.Source: §1 · pp. 1–2
  • The authors introduce a cleaned benchmark of approximately 160k function-level commits from 56 Java projects and make the code and benchmark public.The benchmark filters meaningless commits, including empty, non-ASCII, and merge commits.Source: §1 · p. 2
  • Quantitative and qualitative experiments, including human evaluation, are reported as demonstrating ATOM’s effectiveness and usefulness.The introduction also reports a BLEU-4 improvement over state-of-the-art approaches.Source: §1 · p. 2

2 MOTIVATION AND BACKGROUND

The section defines commits and diffs, motivates AST-based code representation, and introduces neural-network mechanisms relevant to the approach.

2.1 Commit, diff, and Commit Messages

  • A commit pairs a developer-written message with a diff, which records additions and deletions across code versions.A diff may contain multiple chunks identifying changed files and modified lines.Source: §2.1 · p. 3

2.2 Motivation

  • Sequence-based commit-message studies omit hierarchical code structure, while AST use is challenging because changed-code structure cannot be obtained simply by parsing functions.The section motivates bidirectional LSTM and attention to represent long AST paths and focus on important paths.Source: §2.2 · p. 3

2.3 Abstract Syntax Tree (AST)

  • An AST is a tree-structured, high-level representation whose leaves can encode identifiers and literals and whose non-leaves encode syntax.JavaParser provides 106 different non-leaf node types in this account.Source: §2.3 · p. 3, Figure 4 · p. 3

2.4 Encoder-Decoder Model

  • An encoder-decoder passes encoder feature vectors to a decoder to generate a target sequence.The described NMT structure usually consists of two RNNs with LSTM cells and attention.Source: §2.4 · p. 3

2.4.1 Recurrent Neural Network (RNN)

  • RNNs pass information between time steps but can struggle with long-term dependencies and long sequences.LSTM and GRU are given as proposed variants.Source: §2.4.1 · p. 3

2.4.2 Long Short-Term Memory (LSTM)

  • LSTM memory cells use gates to selectively forget information and control reading and writing, helping preserve long-term dependencies.The section describes LSTM-based RNNs as widely used in sequence models.Source: §2.4.2 · p. 3

2.4.3 Attention Mechanism

  • Attention lets each decoder output access the full input sequence and dynamically focus on relevant elements.It maps decoder time steps to encoder hidden states.Source: §2.4.3 · pp. 3–4

2.5 Convolutional Neural Network (CNN)

  • CNN-based neural ranking models can learn high-level sentence-matching patterns.The section describes convolution, pooling, and fully connected layers as components of CNNs.Source: §2.5 · p. 4

3 OUR APPROACH (ATOM)

The section introduces ATOM and states that it details the approach’s modules.

3 OUR APPROACH (ATOM)

  • The section presents an overview of ATOM and details each of its modules.Source: §3 · p. 4

3.1 Overview

The overview describes separate preprocessing for code and messages, followed by generation, retrieval, and ranking modules.

3.1 Overview

  • Preprocessing extracts AST paths for code changes from completed repository functions and uses the lemmatized first message sentence as the target.Source: §3.1 · p. 4
  • AST2seq encodes diff AST paths with a BiLSTM and uses an attention-based decoder to generate a message.Source: §3.1 · p. 4
  • The retrieval module matches a diff against training-set diffs and returns a message based on cosine similarity.Source: §3.1 · p. 4
  • A CNN adaptively ranks the generated and retrieved messages, and ATOM uses its trained generation and ranking modules at prediction time.Source: §3.1 · p. 4

3.2 Preprocessing Module

ATOM preprocesses diffs into added and deleted code representations and prepares the first commit-message sentence as the generation target. For code changes, it retrieves complete functions, parses their ASTs, and extracts paths between changed tokens.

3.2.1 Code Changes

  • Added and deleted diff tokens are separated by their signs and tokenized, with punctuation and other meaningless tokens removed.The resulting token lists are denoted W+ and W-.Source: §3.2.1 · p. 4
  • Completed functions are retrieved from the repository using diff file paths and modified line numbers, then parsed to obtain ASTs.These functions provide the compilation units used to extract AST paths based on the diffs.Source: §3.2.1 · p. 4
  • For each pair of changed-code tokens, ATOM extracts an AST path using the shortest distance between their leaf nodes.A path contains the two tokens and the intervening non-leaf nodes.Source: §3.2.1 · p. 4, §3.2.2 · p. 4

3.2.2 Commit Messages

  • The first commit-message sentence is used as the target sequence, with underscores split, file names and digits replaced by placeholders, and words lemmatized.The first sentence is selected because it often summarizes the commit.Source: §3.2.2 · p. 4

3.3 Generation Module

AST2seq represents code changes with AST paths rather than treating diffs as flat token sequences, then uses attention and a message decoder to generate commit messages. Its encoder combines path structure with endpoint-token information.

3.3 Generation Module

  • AST2seq extracts AST paths based on diffs to represent code changes, addressing sequence approaches’ neglect of code structure and their limitation on longer diffs.Its pipeline encodes each path, attends to relevant paths, and decodes a corresponding commit message.Source: §3.3 · p. 5

3.3.1 AST Encoder

  • A bidirectional LSTM encodes each AST path, while a fully connected layer combines its path representation with representations of the endpoint leaf tokens.Endpoint token values are split into subtokens and their embeddings summed; added and deleted path vectors are concatenated to represent the diff.Source: §3.3.1 · p. 5

3.3.2 Attention

  • During decoding, Luong attention learns a weight distribution over added and deleted AST paths to focus on paths that capture change information.The context vector is computed from the path representations and the decoder’s current hidden state.Source: §3.3.2 · p. 5, §3.3.3 · p. 6

3.3.3 Message Decoder

  • At each decoding step, the context vector and decoder hidden state are combined to predict the current message token, using softmax cross-entropy loss.The loss is defined over the message vocabulary, with the true message token as the target.Source: §3.3.3 · p. 6

3.4 Retrieval Module

The retrieval module finds a training-set commit message for each validation or test diff by matching diffs using token-based TF-IDF cosine similarity.

3.4 Retrieval Module

  • For each validation or test diff, ATOM retrieves the training-set commit message belonging to the first-ranked diff match.Training diffs are indexed, and similarity is computed from token TF-IDF scores using cosine similarity.Source: §3.4 · p. 6
  • TF-IDF uses token counts and the number of diffs containing each token to characterize tokens across the diff collection.The passage defines token frequency using counts in a diff and inverse document frequency using the collection’s diff count.Source: §3.4 · p. 6
  • The retrieved message is passed to the ranking module as a candidate alongside the message produced by the generation module.The ranking stage chooses between these two candidates.Source: §3.4 · p. 6

3.5 Hybrid Ranking Module

The hybrid ranking module scores a retrieved and a generated commit-message candidate against the diff, then predicts which candidate is more relevant. It trains its ConvNet score using BLEU-4 scores against the ground-truth message as targets.

3.5 Hybrid Ranking Module

  • ATOM ranks the first-retrieved and generated messages by learning relevance scores from their match with the diff, using a ConvNet rather than typical classifiers such as XGBoost or LSTM.The authors report that ConvNet outperforms those typical classifiers in experiments.Source: §3.5 · p. 6

3.5.1 Similarity Matching Matrix

  • A similarity matching matrix built from separate diff and message token embeddings represents their interaction and serves as the ConvNet input.The matrix has dimensions (Ld, Ly), the lengths of the diff and message.Source: §3.5.1 · p. 6

3.5.2 ConvNet Model

  • The ConvNet applies convolution and max-pooling operations to the matching matrix, then uses a fully connected layer to compute a diff–message relevance score.The relevance-score loss is optimized with mean square error.Source: §3.5.2 · pp. 6–7

3.5.3 Training For ConvNet

  • BLEU-4 scores against the ground-truth message serve as training targets for the retrieved and generated candidates, allowing end-to-end ConvNet training.A higher predicted score indicates greater relevance of a candidate message to the diff.Source: §3.5.3 · p. 7

4 EXPERIMENTAL EVALUATION

ATOM is evaluated on a filtered dataset of Java commits against retrieval- and generation-based baselines, with experiments examining its modules, AST path counts, and ranking methods. ATOM outperforms the baselines, while the analyses identify useful settings and complementary roles for retrieval and generation.

4.1.1 Experimental Benchmark

  • The benchmark contains approximately 160,000 commits from 56 popular Java projects, split into training, validation, and test sets at 80%, 10%, and 10%.The authors filter noisy or unsuitable commits, retain Java-file commits, and remove messages longer than 20 words and duplicate commit contents.Source: §4.1.1 · p. 7

4.1.2 Experimental Settings; 4.1.3 Evaluation Metrics

  • Experiments use BLEU-N, ROUGE-L, and Meteor to compare generated messages with ground truths.The model settings include a maximum of 80 added and deleted AST paths and beam search with width 5.Source: §4.1.3 · p. 7, §4.1.2 · p. 7

4.3.1 What is the performance of ATOM comparing with baseline approaches?

  • ATOM improves baselines by 30.72% in BLEU-4, 44.89% in ROUGE-L, and 35.26% in Meteor.The comparison results are reported in Table 1.Source: Table 1 · p. 8, §4.3.1 · p. 8

4.2 Comparison Methods

  • ATOM is compared with retrieval-based NNGen and generation-based NMT, Ptr-Net, CODISUM, and Commit2Vec.The authors reproduce NNGen and Commit2Vec and use available source code or published default settings for other baselines.Source: §4.2 · p. 8

4.3.2 What is the impact of individual modules on the performance of ATOM?

  • Hybrid ranking improves over the individual modules by 12.76%–56.58% across BLEU-4, ROUGE-L, and Meteor; retrieval and generation contribute 55.66% and 44.34% of test outputs, respectively.Across 14,674 test samples, the authors describe the two modules as complementary.Source: §4.3.2 · p. 9, Table 2 · p. 9

4.3.3 How accurate is AST2seq under a different number of paths?

  • 80 AST paths is optimal in the experiment; using 30 paths lowers BLEU-4 to 4.27, while 80 paths yields BLEU-4 7.35.Increasing the path count beyond 100 does not continuously improve scores, and the paper notes a training burden from large path sets.Source: §4.3.3 · p. 9, Table 3 · p. 9

4.3.4 What is the impact of different ranking methods?

  • ConvNet outperforms XGBoost, SVR, GRU, and LSTM variants for ranking candidate messages.The authors attribute ConvNet's performance to its similarity-matching matrix for capturing relevance between diffs and candidate messages.Source: §4.3.4 · p. 9, Table 4 · p. 10

4.4 Human Study

Six Java-experienced students rated messages from ATOM, NNGen, and NMT(Luong) for 100 sampled commits each. ATOM received the highest human scores, and the examples also identify a case where generated messages miss code-change detail.

4.4.1 Survey Design

  • Participants rated 100 randomly selected test commits each, scoring NNGen, NMT(Luong), and ATOM messages from 0 to 4 while unaware of each message's source.The questionnaire showed the diff and reference message, randomly ordered the generated messages, and included commit ids for lookup.Source: §4.4.1 · p. 10, §4.4 · p. 10, §4.4.2 · p. 10

4.4.2 Survey Results

  • In human evaluation, ATOM averaged 1.75, exceeding NNGen (1.59) and NMT(Luong) (1.32), and produced more messages scored ≥3 than either baseline.The evaluation yielded 600 pairs of scores, each containing corresponding scores for NMT(Luong), NNGen, and ATOM.Source: §4.4.2 · p. 10
  • ATOM has the highest reported inter-rater agreement at 62.60%, compared with 51.78% for NNGen and 45.32% for NMT(Luong).The authors conclude from the human study that ATOM produces messages more semantically related to the ground truths.Source: §4.4.2 · p. 10

4.5 Examples

  • In one example, all generated messages omit that the change concerns the “jmstype header”; the authors suggest code text such as logs may be underused.They identify text-mining techniques, including part-of-speech analysis, as future work for capturing textual information in diffs.Source: §4.5 · p. 11

5 DISCUSSION

The discussion examines AST2seq’s handling of code changes, the benchmark and model design, parameter and OOV behavior, complexity, dataset partitioning, and threats to validity.

5.1 Strengths of AST2seq

  • 11.37 BLEU-4 is achieved for diffs within 10 lines, while performance on 100+ line diffs declines by only 2.13 BLEU-4, 1.92 ROUGE-L, and 2.65 Meteor.AST2seq samples paths between AST leaf nodes, with up to 80 paths each for added and deleted ASTs.Source: §5.1 · p. 11, Table 7 · p. 12

5.2 Our Benchmark

  • The benchmark covers 56 popular Java projects, filters noisy and duplicate commits, and is publicly available for research.It includes commit information and functions altered by commits, supporting research beyond commit message generation.Source: §5.2 · pp. 11–12

5.3 Novelty of ATOM

  • AST2seq uses AST paths from changed fragments rather than whole-function ASTs, handling added and deleted code separately before concatenating their representations.The authors describe this as the first use of ASTs for commit message generation.Source: §5.3 · p. 12

5.4 Influence of Parameter Setting

  • Embedding size 128 gives ATOM its best performance, while decoder hidden size affects performance more than encoder hidden size.The authors set both hidden sizes to 256 based on the experiments.Source: §5.4 · pp. 12–13, Figure 10 · p. 13

5.5 OOV Issue

  • Only 1,930 of 14,674 test samples contain OOV tokens, leading the authors to suppose that a copy mechanism might not improve ATOM much.The training vocabulary contains 90,969 unique tokens from diffs and commit messages.Source: §5.5 · p. 13

5.6.1 Model Complexity

  • ATOM requires more preprocessing and tuning than sequence-based approaches, but its generation process is relatively low-cost once the models are fixed.ATOM takes more time to train and test than NMT(Luong); NNGen takes 308 seconds to test and requires no training.Source: §5.6.1 · p. 13, Table 8 · p. 13

5.6.2 Dataset Partition

  • ATOM outperforms baselines on project- and timestamp-based splits, although scores decline; the authors recommend commit-based splitting as the best practice.They caution that project-based splitting may not suit commit message generation.Source: §5.6.2 · pp. 13–14, Table 9 · p. 14, Table 10 · p. 14

5.7 Threats to Validity

  • The authors could not verify ATOM on Jiang’s dataset because it lacks commit IDs needed to extract added and deleted ASTs.They also note that human-evaluation judgments may not fully align with the evaluation criterion.Source: §5.7 · p. 14

6 RELATED WORK

Prior commit message generation work uses rules, retrieval, or deep learning, while code summarization offers related structural and learning-based approaches. ATOM combines AST encoding with retrieval and generation through hybrid ranking.

6.1 Code Commit Message Generation

  • Rule-based methods rely on manually specified rules, retrieval systems reuse messages from similar diffs, and deep-learning methods translate code changes into messages.The paper notes limitations including rule coverage, name consistency in retrieved messages, and an unbridged code–language gap in neural approaches.Source: §6.1 · pp. 14–15
  • ATOM combines AST-based code representation with retrieved and generated messages, using hybrid ranking to select more accurate commit messages than prior approaches.Source: §6.1 · p. 15

6.2 Code Summarization

  • Code summarization includes rule-, retrieval-, and learning-based approaches, but this paper targets code changes rather than summaries of whole functions.Related learning-based work includes AST paths for code snippets and AST-based models for other languages and tasks.Source: §6.2 · p. 15

7 CONCLUSION

ATOM represents code changes with AST paths and uses hybrid ranking to prioritize retrieved or generated commit messages. Experiments on the authors’ benchmark show a 30.72% increase over state-of-the-art approaches in BLEU-4.

7 CONCLUSION

  • ATOM encodes AST paths of diffs to represent code changes for commit message generation.Source: §7 · p. 15
  • The hybrid ranking module prioritizes the most accurate message from retrieved and generated candidates.Source: §7 · p. 15
  • 30.72% BLEU-4 increase over state-of-the-art approaches is reported on the authors’ benchmark.Source: §7 · p. 15

Related papers

Loading 1912.02972v2…