Source-linked AI summary
ComBeautify: Leveraging Code Change Summaries for Conventional Commit Generation
Rishi Raj, Aashray Tandon, Nikolay Matyunin, Jibesh Patra
TL;DR
Commit messages often lack clear technical explanations, while CCS type selection is difficult because types are numerous and project-specific. ComBeautify summarizes changes and learns project conventions from commit history; project-specific learning outperforms generalized models for classification and message generation.
Problem
CCS provides no explicit type definitions, and manually defining types is time-consuming and non-scalable, especially as types and their semantics vary across projects.
Source: §2.3 · p. 9, Baselines · p. 14Method
ComBeautify summarizes commit changes and combines the summary with the original message, using fine-tuned LLMs to generate the CCS type and description while learning project conventions from commit history.
Source: §1 · p. 4, §8 · p. 22Results
Project-specific learning outperforms generalized models for commit classification and message generation.
Source: §1 · p. 4Takeaways & Limitations
Learning conventions from each project's commit history offers an alternative to manually curated type definitions.
Source: §1 · p. 4, Baselines · p. 14The project-specific evaluation covers twenty open-source projects, which may not constitute a representative sample.
Source: §6 · p. 20
Abstract
from arXiv · showhide
Modern software development relies on version control systems such as Git to track changes and facilitate collaboration. While commit messages are crucial for conveying developer intent, they do not affect program functionality; as a result, developers often lack motivation to write meaningful ones. The Conventional Commits Specification (CCS) addresses this with a structured format and guidelines that help developers express the intent of a change, improving clarity and enabling automation. Yet, developers frequently struggle to express intent while adhering to the format. In particular, they may misclassify commits due to the large number of possible categories or the lack of project-specific guidance. We introduce ComBeautify, an approach that automatically generates CCS-compliant commit messages. Our hybrid approach leverages context-preserving summarization of Git diffs with large language models (LLMs) to improve commit classification accuracy under CCS. Rather than relying on manually curated category definitions, we learn them from historical commits. We also propose a method for automatically generating the CCS description component. Our empirical evaluation across 13 languages, 6,500 open-source repositories, and over 20 million commits provides long-term insights into CCS adoption, revealing increasing compliance. Our evaluation on twenty popular repositories reveals the limitations of approaches guided by generic category definitions, and shows that ComBeautify, when trained on project-specific data, outperforms baselines by up to 3x. Additionally, using a dataset of 2,000 manually curated commits, we validate our summarization approach, achieving up to 95.25% accuracy and surpassing baselines. These findings emphasize the need for adaptable, context-aware AI methods for commit classification and CCS-compliant commit message generation.
1 Introduction
Commit messages communicate developer intent, but low motivation and unclear guidance often leave them informal or incomplete. ComBeautify uses change summaries and project-specific conventions to generate structured, CCS-compliant messages.
1 Introduction
- Developers may lack motivation to write effective messages, while time constraints, self-explanatory assumptions, and unclear guidance contribute to missing information.Tian et al. (2022) define high-quality messages as stating both what changed and why.Source: §1 · pp. 2–3
- CCS structures messages as a type and description, with optional scope, body, and footer, making intent clearer and enabling automated tools to use message information.The example uses the type “fix” and scope “router” to explain a bug fix.Source: §1 · pp. 3–4
- ComBeautify combines a summary of code changes with the original message and learns project-specific conventions to generate CCS types and descriptions.Figure 1 illustrates an informal httpd message transformed into a CCS-compliant message with a clear type and description.Source: Figure 1 · p. 2, §1 · p. 4
- The paper analyzes CCS adoption and presents an approach for generating compliant messages from non-conforming commits aligned with project conventions.The introduction also identifies a case study of how developers use commit structure for downstream purposes.Source: §1 · p. 4
- The case study examines uses of commit structure including changelog generation and semantic-version bumps.These are presented as examples of purposes for which developers leverage the available structure of a commit.Source: §1 · p. 4
2 Approach
ComBeautify generates the mandatory type and description of a conventional commit from a non-compliant commit. It summarizes code diffs and combines that summary with the original message, then uses project-history-trained models to generate each component.
2 Approach
- ComBeautify generates the mandatory type and description, while leaving the optional body and footer out of scope.The generated components are combined with the changed files to form a CCS-compliant commit.Source: §2.1 · p. 5, §2.5 · p. 10
2.1 Overview
- The pipeline aggregates file-level diffs, summarizes them, appends the original message, and sends the result to separate type and description generators.The summary substitutes for directly supplying potentially large diffs to the generators.Source: §2.1 · p. 5
2.2 Creating Commit Summary
- LLM summaries convert potentially large, code-level diffs into brief natural-language accounts that capture key changes for message generation.The reported average summary is 688 characters, about 30 times shorter than the original diff.Source: §2.2 · p. 6
2.3 Generating Type
- Project-specific type generation addresses ambiguous categories by learning each project's conventions from historical commits rather than relying on universal definitions.The paper notes that type meanings can vary by project and common development tasks are assigned inconsistently across types.Source: §2.3 · pp. 8–9
2.4 Generating Description
- A dedicated fine-tuned LLM generates the concise description from the commit-intent representation; its training examples must include a body.The body is optional in CCS, but is a precondition for selecting commits to fine-tune the description model.Source: §2.4 · p. 9
2.5 Inference
- At inference, separate fine-tuned models predict the type and description from the same commit-intent representation, then combine them with the changed files.The two outputs form the mandatory components of the resulting conventional commit.Source: §2.5 · p. 10
3 Experimental Setup
The study measures CCS adoption across a large, multilingual repository dataset and reports adoption patterns by repository, year, language, and commit type. It also describes the models and dataset used for this analysis.
3.1 Benchmark
- The benchmark covers 6,500 open-source repositories across 13 languages, selected as the top 500 repositories by GitHub stars for each language.Repositories with multiple languages were assigned according to GitHub's primary-language label.Source: §3.1 · p. 10
- The collection pipeline yielded approximately 21.5 million non-merge commits from activity since January 2017 and identified CCS messages by matching the prescribed message structure.It attempted shallow cloning first, then used full cloning or GitHub's REST API as fallbacks.Source: §3.1 · p. 10
3.2 Pre-trained LLMs within ComBeautify
- Code Llama Instruct 7B generated context-aware diff summaries, while separate Code Llama models were fine-tuned for type and description generation.The type model used base Code Llama 7B; the description model was based on Code Llama Instruct 7B.Source: §3.2 · p. 10
3.3 Empirical Study
- 35.9% of commits were CCS-compliant in 2025, up from 16.3% in 2017; repository compliance also varied, with 19.2% of repositories at or above 70%.The dataset contains 20.76 million commits, of which about 5.1 million were CCS commits (24.58%).Source: §3.3 · p. 11
- TypeScript had the highest average CCS compliance at 53.9%, followed by Go at 44.4%, Rust at 38.9%, and JavaScript at 34.9%.Dart, C, C++, Python, Shell, and Kotlin showed moderate adoption; Java, C#, and Swift showed comparatively lower compliance.Source: §3.3 · p. 11
- The ten most frequent CCS types included fix (13.4%), chore (10.3%), and feat (8.3%), within a long-tailed distribution of over 215,000 distinct types.The specification's lack of restrictions allows extensive use of custom CCS types.Source: §3.3 · p. 12
- The adoption findings show rising compliance alongside substantial variation across languages, repositories, and CCS types, motivating project-aligned automated message generation.The study reports that 1,247 repositories (19.2%) had at least 70% compliance.Source: §3.3 · pp. 11–12
4 Evaluation
The evaluation tests ComBeautify's type and description generation, examines uses of conventional commits, and assesses the contribution of code-diff summarization. Across the reported comparisons, project-specific type generation beats the baselines, and summarization improves type-generation performance.
4.1 RQ1: Effectiveness of ComBeautify in Generating Type
- The type-generation evaluation compared six configurations across twenty projects, using 1,400 training, 200 validation, and 400 evaluation commits per project.The configurations included retrieval-based generation, zero-shot LLMs, manual type definitions, a cross-project model, and project-specific ComBeautify.Source: §4.1 · p. 12, Baselines · pp. 13–14
- In the deno project, ComBeautify outperformed NNGen, Copilot, and Gemini by factors of 1.6, 1.3, and 1.26, respectively, and beat the cross-project global model across the evaluated projects.ComBeautify also matched or exceeded the manual-definition variant across projects; on Cockroach it improved over that variant by a factor of 34.Source: Table 2 · p. 13, Baselines · p. 14
4.2 RQ2: Effectiveness in Generating Description
- The fine-tuned description-generation model outperformed the much larger general-purpose models, the code-focused model, and the state-of-the-art baseline.The evaluation used 2,000 CCS-compliant commits, split into 1,400 training, 200 validation, and 400 evaluation commits.Source: §4.2 · p. 15, Table 3 · p. 15
4.3.1 Automatically generating CHANGELOGs
- The evaluated GitHub Actions include tools for enforcing Conventional Commit pull requests and for generating changelogs; Release Please also performs semantic version bumps.The Conventional Changelog action groups commits by CCS type and extracts their one-line descriptions for changelogs.Source: §4.3.1 · p. 16, Table 4 · p. 17
- ComBeautify's breaking-change model achieved 88% accuracy, compared with Gemini 2.0 at 83.25%, after commit-message hints were removed.The evaluation used a 1:3 ratio of breaking to non-breaking commits.Source: §4.3.1 · p. 17
4.3.2 Automatically determining a semantic version bump
- A breaking change in the Oh My Posh release notes after version 25.23.3 was followed by a major-version increment in the next release.The paper notes that GitHub Actions can also be used to bump semantic versions.Source: §4.3.2 · p. 17, Figure 5 · p. 18
4.3.3 Impact on Commits
- CCS-compliant commits averaged 466 lines, compared with 679 for non-compliant commits, and were 30% smaller on average.The comparison sampled 100,000 commits in each group.Source: §4.3.3 · p. 18
4.4 RQ4: Utility of Code Summarization
- Adding the LLMsummary component outperformed both the non-summarized approach and zero-shot GPT-4o for generating CCS types.The authors conclude that a natural-language summary performs better than a potentially arbitrarily large diff in this evaluation.Source: §4.4 · pp. 18–19, Figure 6 · p. 19
5 Practical Applicability of ComBeautify
The paper envisions ComBeautify in developer tools and repository workflows, with one-time project-specific training followed by service-based use.
5 Practical Applicability of ComBeautify
- Potential uses include generating CCS commits from draft messages, automating versioning and changelog creation, and classifying legacy commits for repository mining.The authors envision training once on a sample of past project commits, then handling developer requests through a dedicated workstation service.Source: §5 · p. 20
- The authors report that ComBeautify's smaller models outperform current state-of-the-art LLMs, reducing hardware requirements for deployment.The deployment discussion describes a dedicated workstation service for a given project.Source: §5 · p. 20
6 Threats to Validity
The paper identifies limits involving evolving CCS adoption, the representativeness of its project sample, structural compliance detection, and LLM and prompt variability.
6 Threats to Validity
- Adoption trends may change as CCS evolves, and the project-specific evaluation uses twenty popular open-source projects that may not be representative.Projects were selected for popularity and diversity, with at least 30K GitHub stars each.Source: §6 · p. 20
- Structural matching can classify a formatted commit as compliant even if it does not satisfy the specification's intent.The authors manually inspected a sample to reduce this threat.Source: §6 · p. 20
- Results may be affected by LLM hallucinations, model-version updates, and prompt sensitivity; prompt crafting is identified as a major threat.The authors state that they preserved experimental findings to address future model updates.Source: §6 · p. 20
7 Related Work
Prior commit-message-generation research uses retrieval, in-context examples, or richer software context, but does not follow CCS. Earlier CCS work classified commits into predefined types; this paper emphasizes generation and project-specific conventions.
7 Related Work
- Earlier CMG approaches use retrieved examples or richer software context, but do not adhere to CCS; those techniques could potentially improve CCS descriptions.The paper also situates its work alongside history-aware commit-message completion.Source: §7 · p. 21
- Commit classification research spans word-frequency, NLP-feature, and generative methods, while Zeng et al. (2024) classified commits into ten manually curated CCS types.This paper instead focuses on generating CCS types and descriptions and reports that project-specific fine-tuning outperforms generalized models.Source: §7 · p. 21
- The paper extends CCS adoption analysis beyond Zeng et al. (2024), using over 20 million commits and adding case studies of CCS benefits.Source: §7 · pp. 21–22
8 Conclusion
ComBeautify generates structured commits using change summaries and patterns learned from historical commits. Its real-world evaluation reports gains in type prediction and description generation over specified baselines.
8 Conclusion
- ComBeautify summarizes code changes and learns from historical commits rather than manual annotations to generate CCS type and description.The conclusion identifies generating the body and adding scope alongside type as future directions.Source: §8 · p. 22
- On real-world projects, ComBeautify improves CCS type prediction over manual-definition approaches and outperforms two larger general-purpose models for description generation.Source: §8 · p. 22