PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes a Good Commit Message?

Yingchen Tian, Yuxia Zhang, Klaas-Jan Stol, Lin Jiang, Hui Liu

arXiv:2202.02974v1cs.SEcs.HC

TL;DR

Commit messages help developers communicate what changed and why, but quality varies and prior work offers limited semantic guidance. This study defines and analyzes good messages in five active open-source projects, finding that around 44% lack What or Why information and that Bi-LSTM models performed well at identifying well-written messages.

  • Problem

    Commit messages vary in quality, while developers may lack time or guidance and existing quality work has not addressed message semantics.

    Source: §1 · p. 2

  • Method

    The authors defined good messages as containing What and Why, analyzed a sample from five Java open-source projects, developed an expression taxonomy, and trained classifiers to identify good messages.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Around 44% of sampled commit messages lacked “why” or “what” information, and the Bi-LSTM models performed well at identifying well-written messages.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings can help developers write good messages and help researchers construct higher-quality datasets before automatic message generation.

    Source: §7 · p. 11

    The findings may not generalize beyond five popular Java GitHub projects to other projects or programming languages.

    Source: §6 · p. 11

Abstract

from arXiv · show

A key issue in collaborative software development is communication among developers. One modality of communication is a commit message, in which developers describe the changes they make in a repository. As such, commit messages serve as an "audit trail" by which developers can understand how the source code of a project has changed-and why. Hence, the quality of commit messages affects the effectiveness of communication among developers. Commit messages are often of poor quality as developers lack time and motivation to craft a good message. Several automatic approaches have been proposed to generate commit messages. However, these are based on uncurated datasets including considerable proportions of poorly phrased commit messages. In this multi-method study, we first define what constitutes a "good" commit message, and then establish what proportion of commit messages lack information using a sample of almost 1,600 messages from five highly active open source projects. We find that an average of circa 44% of messages could be improved, suggesting the use of uncurated datasets may be a major threat when commit message generators are trained with such data. We also observe that prior work has not considered semantics of commit messages, and there is surprisingly little guidance available for writing good commit messages. To that end, we develop a taxonomy based on recurring patterns in commit messages' expressions. Finally, we investigate whether "good" commit messages can be automatically identified; such automation could prompt developers to write better commit messages.

1 INTRODUCTION

Commit messages help collaborators understand what changed and why, but limited time, motivation, and guidance make quality difficult to ensure. This study defines well-written messages, measures missing information, analyzes expression patterns, and tests automatic identification.

Motivation

  • Commit messages can preserve the context and rationale of changes for collaborators and future developers, especially after original developers leave a project.For long-lived projects, messages may be the only remaining source of information about what changed and why.Source: §1 · p. 1

Problem

  • Developers may lack time or motivation to write good messages, while existing syntactic quality checks do not assess message semantics or clarify what information to include.Prior studies found many messages empty or very short, and only 10% contained normal descriptive English sentences.Source: §1 · p. 2
  • Prior automatic generators depend on historical messages that may be low quality, while existing evaluations often remove trivial messages without checking content quality.This can threaten training quality and may yield artificially high precision when generated messages resemble low-quality test messages.Source: §1 · p. 2

Study findings

  • Around 44% of sampled messages lack “why” or “what” information, and the study’s Bi-LSTM models performed well at identifying well-written messages.The authors argue that low-quality messages in unfiltered training data threaten automatic commit-message generation.Source: §1 · p. 2

2 RELATED WORK

Prior work uses commit messages to study software changes and to generate new messages, but research on message content and quality has been limited.

Commit-message uses

  • Commit messages have supported classification of code changes into maintenance types, including adaptive, corrective, and perfective changes.This stream uses messages to classify changes and assist software maintenance.Source: §2 · p. 2

Commit quality

  • Studies have measured commit quality through message characteristics and found associations between unusual messages and build failures.Agrawal et al. reported declining commit quality over time; Santos et al. linked unusual messages with build failures.Source: §2 · p. 2

Automatic generation

  • Automatic generators range from templates to information retrieval and deep learning, with many reusing prior messages that may vary in quality.Template-based approaches lack flexibility and cannot convey intent that exists only in the developer’s mind before it is written.Source: §2 · p. 3

Content and quality

  • Research on message content has focused on specific aspects, including refactoring documentation and syntactic quality measures, leaving broader content patterns less explored.The paper positions its distribution, expression-category, and identification analyses as complements to this work.Source: §2 · p. 3

3 STUDY DESIGN

The study combines manual labeling, thematic analysis, and machine-learning classification to examine message quality, expression patterns, and automatic identification. It focuses on five active Java open-source projects and defines a good message as containing both what changed and why.

3.1 Data Collection and Preprocessing

  • Five active, popular Java open-source projects were selected from highly ranked GitHub projects, and 29,348 manually written English commit messages remained after cleaning.The collection included 41,886 commits through February 2021 before bot messages were excluded.Source: §3.1 · p. 3

3.2 Identifying Well-Written Messages

  • The authors developed the What-and-Why standard by reviewing research and developer guidance, consulting experienced developers, and treating trivial or diff-inferable information as present.They classified messages into four types and counted issue or pull-request links as Why information.Source: §3.2 · p. 4
  • A clustered random sample of 1,649 messages was independently labeled by two authors; 52 non-atomic commits were removed, and agreement was high.Cohen’s kappa was 0.91; a developer survey yielded almost 85% labeling accuracy.Source: §3.2 · pp. 4–5

3.3 Characterizing Well-Written Messages

  • The authors used thematic analysis on 252 messages containing both What and Why, with independent coding followed by discussion and theme refinement.They also classified maintenance activities using Mockus and Votta’s corrective, adaptive, and perfective categories.Source: §3.3 · p. 5

3.4 Automatic Identification

  • Three classifiers identify Why, What, and messages containing both, allowing separate feedback about which key element is missing.The two best-performing component classifiers were combined into C-Good.Source: §3.4 · p. 5

3.4.1 Data Preparation.

  • The authors tried random oversampling, SMOTE, and ADASYN to address class imbalance, selecting the technique with highest accuracy for each classifier.The imbalance concern was that models might undervalue minority categories.Source: §3.4.2 · pp. 5–6

3.4.2 Identification.

  • Commit messages were embedded with BERT, then several classification algorithms were evaluated and the best-performing approach selected.Candidates included Bi-LSTM, LSTM, MLP, logistic regression, random forest, KNN, gradient boosting, and decision tree.Source: §3.4.2 · p. 6

4 RESULTS

The results examine commit-message quality by quantifying it, characterizing how developers express Why and What, and assessing whether classifiers can identify missing information and well-written messages.

4.1 Quality Distribution of Commit Messages

  • ca. 56% of messages were well-written on average, leaving ca. 44% needing improvement across five sampled OSS projects.The 1,597-message sample showed that well-written messages dominated four projects; messages containing neither Why nor What made up ca. 1%–8% and fell into five categories: single-word, submit-centered, scope-centered, redundant, and irrelevant.Source: Figure 3 · p. 6, §4.1 · p. 6

4.2 A Taxonomy of Commit Messages

  • Analysis of 252 messages yielded five Why expression categories with 18 subcategories and four What expression categories.The taxonomy captures recurring ways developers describe rationales and change contents; messages could use multiple categories, though no more than two per dimension and at most 9% did so.Source: §4.1 · p. 6, §4.2.1 · p. 7, §4.2.2 · p. 8

4.2.1 Why Expression Categories.

  • Why expressions comprise Describe Issue, Illustrate Requirement, Describe Objective, Imply Necessity, and Missing Why.Developers may state rationales directly or indirectly, or omit them when the reason is common sense or inferable; Describe Issue includes error scenarios, issue reports, and shortcomings, while Missing Why includes routine changes such as tests, typo fixes, text edits, annotation changes, refactoring, and version management.Source: §4.2.1 · pp. 7–8

4.2.2 What Expression Categories.

  • Commit messages convey what changed by summarizing code-object changes, describing implementation principles, illustrating function, or omitting what changed; over half of commits identify the changed code object.Implementation Principle appeared in 6 of 252 commits (2.4%); Illustrate Function in 65 of 252; Missing What in 19 (7.5%).Source: §4.2.2 · pp. 8–9

4.2.3 Linking Maintenance Dimensions to Commit Messages.

  • Why preferences varied by maintenance activity: Describe Issue for corrective, Imply Necessity for adaptive, and Missing Why for perfective changes.For corrective changes, Describe Issue reached 45.7% and code-object summaries described What in 58.6%; Imply Necessity reached 39.7% for adaptive changes, and Missing Why reached 34.2% for perfective changes. Missing What was unusual across all three activity types.Source: §4.2.3 · pp. 9–10, Table 3 · p. 9

4.3 Automatically Classifying Good Messages

  • Across eight techniques, Bi-LSTM performed best for C-Why and C-What, with accuracies of 84.7% and 91.0%; C-Good reached 75.9%.Performance was estimated using ten-fold cross-validation on 1,597 messages.Source: Table 5 · p. 10, Table 4 · p. 10, §4.3 · p. 10
  • C-Why achieved 76.5% precision and 70.9% recall, C-What 78.2% precision and 64.5% recall, and C-Good 81.6% precision and 74.0% recall.The classifiers indicate whether “Why” or “What” is missing or whether a message is well written, giving developers a hint about what to revise.Source: §4.3 · p. 10
  • Compared with an unfiltered dataset averaging 56% good messages, C-Good can identify and construct a higher-quality commit-message dataset.Source: §4.3 · p. 10

5 IMPLICATIONS

The findings offer developers guidance for writing more informative commit messages and show that message quality matters to generation research and dataset construction. The authors also propose automated checks and a classifier to support these aims.

5 IMPLICATIONS

  • For corrective tasks, developers can use Describe Issue to explain Why and Object of change to summarize the key code objects.Describe error scenario can prompt developers to include reproduction steps and background.Source: §5 · pp. 10–11
  • The authors designed two tools that check whether a message contains both Why and What, helping developers identify what to add.Source: §5 · p. 11
  • About 44% of messages were poor quality across five OSS projects, and uncurated training data can perpetuate low-quality generated messages.The authors say this exacerbates the problem of poorly written commit messages.Source: §5 · p. 11
  • C-Good identifies good messages with 81.6% precision and could help researchers build high-quality datasets from historical VCS data.The authors call for a standard dataset to support comparisons among generation methods and improve generated-message quality.Source: §5 · p. 11

6 THREATS TO VALIDITY

The study notes threats from bot detection, manual labeling, dataset scale and sampling, classifier inputs, and the limited project and language scope. These constraints qualify how broadly its taxonomy and classification findings can be generalized.

6 THREATS TO VALIDITY

  • Newer, more human-like bots may remain in the dataset, although no new bot patterns appeared among 1,649 manually analyzed messages.The dataset filtered messages from known bots using established patterns.Source: §6 · p. 11
  • Multiple messages came from the same authors among 339 developers, potentially limiting the diversity of the message taxonomy.Two authors labeled messages independently, and an experienced qualitative researcher helped resolve disagreements; reported agreement was 0.91.Source: §6 · p. 11
  • Random ten-fold cross-validation did not account for developers potentially writing better messages over time.The dataset contained 1,597 messages sampled from five projects; its limited scale reflected the complexity of manual labeling and analysis.Source: §6 · p. 11
  • The classifiers used message text as input, while format-related elements were replaced with unified tokens during preprocessing.The authors state this preprocessing ensured syntactic features were considered during classification.Source: §6 · p. 11
  • The findings may not generalize beyond five popular GitHub projects implemented in Java, where message-writing patterns may differ.The authors say the classification tools can be adapted to other projects and languages by replacing the datasets.Source: §6 · p. 11

7 CONCLUSION

The study examines commit-message quality and expression patterns, connects expression categories with maintenance activities, and builds models to identify good messages. Its findings can help developers write good messages and researchers create higher-quality datasets for message generation.

7 CONCLUSION

  • Low-quality commit messages in OSS projects threaten the effectiveness of existing automatic commit-message generation tools.Source: §7 · p. 11
  • The study explored the distribution and expression patterns of well-written messages, linked expression categories to maintenance activities, and constructed automatic identification models.Source: §7 · p. 11
  • The findings can help developers write good commit messages and researchers construct high-quality datasets before automatic message generation.Source: §7 · p. 11

Related papers

Loading 2202.02974v1…