PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic Generation of Pull Request Descriptions

Zhongxin Liu, Xin Xia, Christoph Treude, David Lo, Shanping Li

arXiv:1909.06987v1cs.SE

TL;DR

More than 34% of PR descriptions in the collected dataset are empty, motivating automatic generation from commit messages and added source-code comments. The paper frames generation as summarization and proposes a sequence-to-sequence approach with a pointer generator and ROUGE-optimized reinforcement learning. It outperforms two baselines on ROUGE and human evaluation, while noting that preprocessing cannot capture all non-summary content in free-form descriptions.

  • Problem

    More than 34% of PR descriptions in the collected dataset are empty, despite their role in describing PR changes or motivations.

    Source: §I · p. 1

  • Method

    The approach summarizes PR commit messages and added source-code comments with an attentional encoder-decoder, using a pointer generator and a differentiable reinforcement-learning loss to address OOV words and optimize for ROUGE.

    Source: §I · p. 2

  • Results

    ROUGE-1, ROUGE-2, and ROUGE-L improve over two extractive baselines by 11.6%, 25.4%, and 12.2%, respectively, and human evaluation finds the approach significantly better.

    Source: §I · p. 2

  • Takeaways & Limitations

    The approach is intended to replace empty PR descriptions and assist developers in writing them.

    Source: §I · p. 1

    Because PR descriptions are free-form and preprocessing heuristics cannot filter all non-summary information, the training data may retain such content.

    Source: B. Threats to Validity · p. 10

Abstract

from arXiv · show

Enabled by the pull-based development model, developers can easily contribute to a project through pull requests (PRs). When creating a PR, developers can add a free-form description to describe what changes are made in this PR and/or why. Such a description is helpful for reviewers and other developers to gain a quick understanding of the PR without touching the details and may reduce the possibility of the PR being ignored or rejected. However, developers sometimes neglect to write descriptions for PRs. For example, in our collected dataset with over 333K PRs, more than 34% of the PR descriptions are empty. To alleviate this problem, we propose an approach to automatically generate PR descriptions based on the commit messages and the added source code comments in the PRs. We regard this problem as a text summarization problem and solve it using a novel sequence-to-sequence model. To cope with out-of-vocabulary words in software artifacts and bridge the gap between the training loss function of the sequence-to-sequence model and the evaluation metric ROUGE, which has been shown to correspond to human evaluation, we integrate the pointer generator and directly optimize for ROUGE using reinforcement learning and a special loss function. We build a dataset with over 41K PRs and evaluate our approach on this dataset through ROUGE and a human evaluation. Our evaluation results show that our approach outperforms two baselines by significant margins.

I. INTRODUCTION

More than 34% of PR descriptions in the collected dataset are empty, motivating automatic generation from commit messages and added source-code comments. The paper treats this as summarization and reports better ROUGE and human-evaluation results than two baselines.

I. INTRODUCTION

  • Over 34% of the 333,001 PR descriptions collected from 1K engineered Java projects are empty.The proposed approach can replace empty descriptions or assist developers as they write them.Source: §I · p. 1
  • Unlike commit messages or release notes, PR descriptions summarize multiple related commits for developer readers, and prior work had not focused on generating them.The paper treats explicit summarization of commits in a PR as distinct from recovering traceability links.Source: §I · p. 1
  • The approach treats commit messages and added source-code comments as an article, and a PR description as its summary, using an attentional encoder-decoder model.The model learns from existing PRs to generate descriptions for new ones.Source: §I · p. 2
  • The task presents two challenges: software identifiers create out-of-vocabulary words, and maximum-likelihood training does not directly optimize the ROUGE evaluation metric.The fixed-vocabulary attentional model cannot produce OOV words; ROUGE allows different word orders and correlates highly with human evaluation.Source: §I · p. 2
  • A pointer generator copies OOV words from the input, while reinforcement learning with a differentiable RL loss directly optimizes for ROUGE during training.The dataset contains over 41K PRs, and the approach is evaluated using both ROUGE and human evaluation.Source: §I · p. 2
  • ROUGE-1, ROUGE-2, and ROUGE-L improve over the two extractive baselines by 11.6%, 25.4%, and 12.2%, respectively.Human evaluation also finds the approach significantly better than the baselines and able to generate more high-quality descriptions.Source: §I · p. 2

II. MOTIVATION AND PRELIMINARY

This section motivates PR-description generation with an example, describes how generated descriptions could be used, and formulates the task as sequence-to-sequence summarization.

A. Motivating Example

  • In the Pitest example, two commits contribute to a PR description that states both the changes and their motivation; one commit adds a code comment.The commit messages indicate changes, while the added comment supplies motivation.Source: A. Motivating Example · pp. 2–3, Table I · p. 3

B. Usage Scenario

  • Generated descriptions could replace empty PR descriptions, help readers quickly grasp key ideas, and assist developers in writing descriptions after lengthy work.The paper also identifies possible uses in maintenance, program comprehension, and reviewer recommendation.Source: B. Usage Scenario · p. 3

C. Problem Formulation

  • The task is formulated as sequence-to-sequence learning: the source is commit messages plus added code comments, and the target is the PR description.The paper writes this as finding a function f such that f(w) = y.Source: C. Problem Formulation · p. 3

D. Attentional Encoder-Decoder Model

The attentional encoder-decoder encodes a source sequence and uses attention to guide token generation. The following approach section elaborates the pointer generator and reinforcement-learning loss.

D. Attentional Encoder-Decoder Model

  • The encoder and decoder use distinct recurrent neural networks; the encoder’s final hidden state initializes the decoder.A shared embedding layer maps input tokens, and the encoder processes the source sequence into hidden states.Source: D. Attentional Encoder-Decoder Model · p. 3
  • At each decoding step, attention weights encoder states into a context vector that is combined with the decoder state to produce a vocabulary distribution.The attention distribution indicates the importance of each encoding step; the vocabulary distribution determines which token to output.Source: D. Attentional Encoder-Decoder Model · pp. 3–4
  • During training, the attentional model minimizes the negative log-likelihood of the reference sequence.The probability for the next reference token is conditioned on prior decoder inputs and the source sequence.Source: D. Attentional Encoder-Decoder Model · p. 4

III. APPROACH

  • The approach section elaborates the pointer generator and the reinforcement-learning loss.Source: §III · p. 4

A. Pointer Generator

The pointer generator lets the model copy source-sequence words, addressing software-specific OOV terms. The RL loss uses a ROUGE-based reward to guide training, combined with maximum-likelihood loss to support readability.

A. Pointer Generator

  • The pointer generator can copy source-sequence words, allowing the model to produce OOV terms while retaining vocabulary-based generation.The copy probability is computed from attention over the source sequence.Source: A. Pointer Generator · p. 4

B. RL Loss

  • Because negative log likelihood penalizes literal sequence differences while ROUGE tolerates word-order variation, the training objective may not select the best ROUGE descriptions.The authors note ROUGE has been shown to correlate highly with human evaluation.Source: B. RL Loss · p. 4
  • The authors use self-critical sequence training and an RL loss to optimize directly for non-differentiable ROUGE scores.The reward is defined using ROUGE-L F1.Source: B. RL Loss · p. 4
  • The hybrid objective combines maximum-likelihood and RL losses because using only the RL loss can harm generated-text readability.The RL loss weights sequence likelihood by the sampled sequence's reward relative to a greedy baseline.Source: B. RL Loss · p. 5

A. Data Collection

The authors collected merged PRs from engineered Java projects and constructed paired source and target sequences from their descriptions, commit messages, and added code comments. Preprocessing removes noise and standardizes selected tokens.

A. Data Collection

  • The dataset contains 333,001 merged PRs from 1,000 engineered Java projects selected from 22,700 repositories with merged PRs.The projects were selected from 95,804 Java repositories classified as engineered software projects.Source: A. Data Collection · p. 5

1) Preprocess text:

  • Preprocessing removes HTML comments, checklist paragraphs, URLs, internal references, signatures, emails, mentions, and Markdown headings from texts.It also standardizes hashes, version strings, and numbers, and removes non-ASCII tokens.Source: 1) Preprocess text · p. 5

2) Construct target sequence:

  • Target sequences contain preprocessed PR descriptions; descriptions marked non-ASCII, shorter than five tokens, or consisting only of punctuation are removed as trivial.The target sequence consists only of the PR description.Source: 2) Construct target sequence · p. 5

3) Construct source sequence:

  • Source sequences order commits by creation time, place their messages first, then append added-comment paragraphs separated by special tokens.Copyright, license, Java-doc function-signature, and punctuation-only comments are filtered; valid commit messages and comment paragraphs are preprocessed.Source: 3) Construct source sequence · pp. 5–6
  • PRs exceeding the target- or source-sequence length constraints are labeled Long-desc PRs or Long-source PRs, respectively.The passage defines these labels but does not state the length thresholds.Source: 3) Construct source sequence · p. 6

4) Filter PRs:

Filtering left 41,832 PRs for the dataset, and the evaluation section specifies ROUGE-N and ROUGE-L metrics, emphasizing F1 for comparisons.

4) Filter PRs:

  • 41,832 PRs remained after filtering empty or trivial descriptions, PRs outside the valid-commit range, and sequences exceeding length limits.The valid-commit range was 2–20; maximum source and target lengths were 400 and 100, respectively.Source: 4) Filter PRs · p. 6

V. EVALUATION

  • The evaluation section presents its metrics and baselines, then research questions and experiment results, followed by a human evaluation.Source: §V · p. 6

A. Evaluation Metrics

  • The evaluation uses ROUGE-1, ROUGE-2, and ROUGE-L, with F1 scores preferred for comparing approaches because they balance precision and recall.ROUGE-N uses n-gram overlap; ROUGE-L uses longest common subsequences.Source: A. Evaluation Metrics · p. 6

B. Baselines

The paper compares two extractive baselines, LeadCM and LexRank, and describes the model split, architecture, training schedule, and decoding setup. The supplied passages do not report baseline or model scores.

B. Baselines

1) LeadCM:

  • LeadCM returns the first 25 source tokens, effectively taking the earliest commit messages as the generated description.The 25-token limit is the median PR-description length in the dataset.Source: 1) LeadCM · p. 6

2) LexRank:

  • LexRank ranks source sentences by centrality, then concatenates the highest-ranked sentences and keeps the first 25 tokens.Its sentence centrality is computed with PageRank on a sentence-similarity matrix.Source: 2) LexRank · p. 7

C. Experiment Settings

  • The data split is 80% training, 10% validation, and 10% testing; the model uses 128-dimensional embeddings and 256-dimensional hidden states.Training proceeds from maximum-likelihood loss to hybrid loss, and decoding uses beam search of width four with a duplicate-trigram filter.Source: C. Experiment Settings · p. 7

D. RQ1: The Effectiveness of Our Approach

Attn+PG+RL outperforms extractive baselines on ROUGE, while its pointer generator and RL loss each contribute to performance. The authors attribute its advantage over extractive methods partly to selecting salient content and generating concise, rephrased descriptions.

D. RQ1: The Effectiveness of Our Approach

  • Attn+PG+RL beats LexRank on all ROUGE metrics by 32.35%–138.61% and exceeds LeadCM in precision and F1, with F1 gains of 3.54, 4.53, and 3.52 points.The compared results are in Table III; the three F1 gains correspond to the three ROUGE metrics.Source: D. RQ1: The Effectiveness of Our… · p. 7, Table III · p. 8
  • On a test example, the model identifies the salient phrase “tomcat 0 support in the s-ramp,” while LexRank selects the wrong sentences and LeadCM outputs every commit message.The example contains three commit messages and multiple added code comments.Source: D. RQ1: The Effectiveness of Our… · p. 7, Table IV · p. 8
  • Unlike extractive baselines, the abstractive model can rephrase content and vary description length, reducing unimportant phrases and yielding high precision.LexRank and LeadCM extract sentences and cannot rephrase them or dynamically choose how many tokens to generate.Source: D. RQ1: The Effectiveness of Our… · p. 7
  • LeadCM has slightly higher ROUGE-1 and ROUGE-L recall than our approach because it generates as many tokens as possible up to 25, whereas our approach tends to produce short, accurate descriptions.Source: D. RQ1: The Effectiveness of Our… · p. 7
  • LexRank’s assumption that sentences resembling many others are important can fail when duplicate comments make unimportant content appear salient.In the cited test example, two identical source comments are ranked as the most important sentences.Source: D. RQ1: The Effectiveness of Our… · p. 7

E. RQ2: The Effects of Main Components

  • Across the reported evaluation metrics, Attn+PG outperforms Attn by 29.15% to 70.73%, and the authors attribute improved handling of OOV words and PR-description generation to the pointer generator.In one test result, Attn produces “[UNK]” for “fulltextonline” and “webpagearchived”; Attn+PG and Attn+PG+RL generate both words correctly.Source: E. RQ2: The Effects of Main Components · p. 8
  • Compared with Attn+PG, Attn+PG+RL improves all recall and F1 metrics by more than 5.8% but slightly reduces precision as it generates longer descriptions.Attn+PG+RL produces 5.19 more tokens on average; the authors characterize the precision reduction as the expense of improved recall.Source: E. RQ2: The Effects of Main Components · p. 8
  • The authors conclude that the pointer generator and RL loss both help Attn+PG+RL outperform Attn and Attn+PG.The component comparisons are presented in Table V.Source: E. RQ2: The Effects of Main Components · p. 8

F. Human Evaluation

Six Ph.D. students evaluated descriptions for 100 randomly selected test PRs, with three evaluators assigned to each PR. The proposed approach received higher human scores than both baselines, although generating descriptions from scratch may sometimes omit important words.

1) Procedure:

  • Six Ph.D. students with 1–5 years of Java experience evaluated descriptions for 100 randomly selected test PRs, with three evaluators assigned to each PR.The PRs were split evenly into two groups, each assessed by three different evaluators.Source: F. Human Evaluation · p. 8, 1) Procedure · p. 8
  • Evaluators scored semantic similarity to the reference from 0 to 7, with higher scores indicating closer similarity.Each PR description received three evaluator scores, averaged into a final score.Source: 1) Procedure · p. 9

2) Results:

  • Evaluators assigned the proposed approach more scores of 4 or higher and fewer scores below 4 than the baselines, with far more scores between 6 and 7.The distribution of final scores is shown in Figure 4.Source: 2) Results · p. 9, Figure 4 · p. 9

VI. DISCUSSION

  • Because the model generates descriptions from scratch, it may sometimes fail to generate important words.The authors note that this may explain why its descriptions with average scores between 0 and 1 slightly outnumber those of the baselines.Source: 2) Results · p. 9

2) Results:

  • Attn+PG+RL’s mean human score was 3.27, versus 2.73 for LeadCM and 2.14 for LexRank; its improvements over both baselines were significant at p < 0.01.The authors used Wilcoxon signed-rank tests at the 95% confidence level on 100 final scores per approach.Source: 2) Results · p. 9

A. Where Does Our Approach Perform Badly

Poor ROUGE-L cases often involve reference content absent from the source, while validity is bounded by Java-only data and the challenges of filtering free-form descriptions.

A. Where Does Our Approach Perform Badly

  • Poor ROUGE-L performance often occurs when references emphasize context, implementation details, or subjective content that is absent from the source sequence.A typical example combines motivation, implementation details, and test results missing from the source, leaving little overlap with the generated description.Source: A. Where Does Our Approach Perform Badly · pp. 9–10
  • In one subjective-reference example, the generated description better captured the change's meaning than the reference, which the authors considered unhelpful for understanding it.The reference was “we did make awesome, shall we land it,” while the generated text described refactoring common pieces into httpobject.Source: A. Where Does Our Approach Perform Badly · p. 10
  • Missing key phrases in the source can also hurt performance, though this is less common; the approach captures key phrases better than the two baselines.Source: A. Where Does Our Approach Perform Badly · p. 10

B. Threats to Validity

  • Validity is bounded by Java-only data, and non-summary or subjective content in PR descriptions may affect effectiveness; heuristic preprocessing cannot capture every non-summary pattern.The authors identify improved preprocessing as a more suitable direction for future work.Source: B. Threats to Validity · p. 10
  • Human ratings may be unfair despite averaging three evaluators, while baseline summaries capped at 25 tokens may produce incomplete sentences and lower human ratings.The authors state that the baseline-length threat does not affect ROUGE scores, which also confirm their approach's superiority.Source: B. Threats to Validity · p. 10

A. Documenting Software Changes

Prior work documents software changes at commit and release scales, studies pull-request workflows, and summarizes other software artifacts. The paper treats PR descriptions as a distinct multi-commit generation task and reports gains over two baselines.

A. Documenting Software Changes

  • Prior commit and release documentation tools use analyses, templates, or machine learning; they complement PR generation, which summarizes related commits for a distinct audience.Source: A. Documenting Software Changes · pp. 10–11

B. Understanding Pull Requests

C. Summarizing and Documenting Other Software Artifacts

  • Other work summarizes code, bug reports, reviews, discussions, and development activity; this approach instead generates PR descriptions from commit messages and added source-code comments.Source: C. Summarizing and Documenting Other… · p. 11

VIII. CONCLUSION AND FUTURE WORK

  • Experiments on over 41K PRs and human evaluation show that the approach outperforms two competitive baselines.Source: §VIII · p. 11
  • Future work will test usefulness on PRs without descriptions and explore adding artifacts such as diffs and bug reports to infer implementation details and motivation.Source: §VIII · p. 11

Related papers

Loading 1909.06987v1…