CoCoRerank: Towards Conventional Commit Message Generation by Component and Candidate Consistency Reranking

Introduction

• Earlier CMG methods improved fluency and informativeness using code structure, edit representations, retrieval, pretraining, and hybrid designs, but generally generated messages as unstructured sequences (Jiang, Armaly, and McMillan 2017; Xu et al. 2019; Nie et al. 2021; Dong et al. 2022; Liu et al. 2018, 2020; Shi et al. 2022; He et al. 2023).CMG generates messages from code diffs, a compact representation of a commit’s changes.Source: Introduction · p. 1
• CCS formats messages as type (scope): subject, making them human- and machine-interpretable, but existing CMG methods do not explicitly ensure the three components agree with the same code change.A message can be fluent yet have an inconsistent type, irrelevant scope, or subject that mismatches its metadata.Source: Introduction · p. 1
• Existing CMG datasets generally contain unstructured messages or only partial structure, limiting training and evaluation of complete CCS components and their joint coherence.This leaves models’ type, scope, and subject accuracy difficult to assess separately.Source: Introduction · pp. 1–2
• The new dataset contains 86,688 high-quality commits normalized into complete type, scope, and subject fields, with structural normalization and semantic quality filtering.It enables supervision and fine-grained evaluation of each CCS component.Source: Introduction · p. 2
• CoCoRerank extracts consensus type and scope across candidates, scores subject compatibility with the diff and consensus metadata, then aggregates scores over semantically clustered subjects.It combines horizontal consistency among type, scope, and subject with vertical consensus across candidates to select structurally coherent, representative messages.Source: Introduction · p. 2
• Experiments show CoCoRerank improves direct LLM generation across conventional components, outperforms majority voting and minimum-Bayes-risk reranking, and benefits from both consistency signals.Ablation and cross-model analyses indicate that component consistency and candidate consensus both contribute to performance.Source: Introduction · p. 2

Commit Message Generation Methods

• Commit message generation methods span rule-based, retrieval-based, learning-based, and hybrid approaches, each using different strategies to produce or select messages.Rule-based systems use templates; retrieval-based systems reuse similar commits; learning-based systems map changes to text; hybrid methods combine retrieval and generation. Examples include NNGen, CODISUM, and RACE (Shi et al. 2022).Source: Commit Message Generation Methods · p. 2

Commit Message Generation Datasets

• Earlier datasets range from GitHub Java-repository collections to MCMD’s roughly 2.25 million commits, but MCMD’s lenient filtering can leave noisy, overlapping samples.CommitGenData and CoDiSumData draw from GitHub’s top 1,000 Java repositories, while PtrGNCMsgData aggregates commits from around 2,000 Java repositories.Source: Commit Message Generation Datasets · p. 2
• CommitBench applies extensive filtering to 1,664,590 commits, while CCS-compliant CommitSuite includes 63,533 commits but omits the scope field from its generation setup.CommitBench covers 71,676 projects; CommitSuite covers 243 repositories and includes AST-level diffs and LLM-aided annotations for commit what and why semantics.Source: Commit Message Generation Datasets · p. 3

Dataset Construction Overview

• Construction proceeds through data collection, structural normalization into type(scope): subject, and semantic quality filtering.The process collects commits from 206 GitHub repositories, applies repository, commit, diff, and subject filters, standardizes fields and labels, then screens semantic quality using embedding similarity and LLM assessment.Source: Dataset Construction Overview · p. 3

Final Dataset Statistics

• 86,688 high-quality commit–message pairs remain after structural and semantic filtering.The final dataset is the product of the paper’s filtering process.Source: Final Dataset Statistics · p. 3
• Diff lengths vary widely, commit subjects are usually concise, and commit types follow a long-tailed distribution that the dataset preserves.Using the Qwen3-0.6B tokenizer, 65.54% of diffs have fewer than 1,024 tokens and 12.00% have at least 4,096; 96.24% of subjects have fewer than 16 tokens, and fix is the most frequent type while ci, style, and update are much less frequent.Source: Final Dataset Statistics · p. 3
• The dataset is stratified by commit type into training, retrieval, validation, and test sets in a 6:2:1:1 ratio, with near-duplicates checked across splits.Source: Final Dataset Statistics · p. 3

Overview

• CoCoRerank retrieves similar commits, generates candidates, derives consensus type and scope, scores subjects, clusters weak matches, and selects the top-ranked subject for the final message.The selected subject is combined with the consensus structural fields.Source: Overview · p. 3

Retrieval-Augmented Candidate Generation

• For each change, five similar historical commits retrieved by diff similarity serve as in-context examples for generating 100 CCS-constrained candidates parsed into type, scope, and subject.Source: Retrieval-Augmented Candidate Generation · p. 3

Two-Dimensional Consistency Reranking

• Horizontal consistency checks agreement with the code change and structural fields, while vertical consistency measures support from other candidates for the same commit.Source: Two-Dimensional Consistency Reranking · p. 3

Vertical Structural Consensus

• The most frequent candidate type and scope form shared structural consensus, replacing each candidate’s potentially noisy fields when evaluating its subject.Source: Vertical Structural Consensus · p. 3

Horizontal Component Consistency

• A learned consistency scorer evaluates whether each subject describes the code change while remaining compatible with the consensus type and scope.Source: Horizontal Component Consistency · p. 4

Vertical Semantic Consensus

• Weakly matched subjects are clustered to aggregate support for interpretations despite lexical variation; clusters are ranked by aggregate consistency, and the strongest individual subject in the top cluster is selected.Weak matches require token-length difference at most three and similarity of at least 0.97 after case, whitespace, punctuation, and tokenization normalization.Source: Vertical Semantic Consensus · p. 4

Consistency Scorer

• A Qwen3-0.6B scorer evaluates semantic compatibility among the diff, consensus metadata, and subject, trained first for general alignment and then to distinguish plausible same-commit candidates.The second stage uses hard negatives generated from the same code change that may be inaccurate, incomplete, or misleading, and combines their loss with the in-batch objective.Source: Consistency Scorer · p. 5

Experimental Setup

• Experiments compare representative CMG baselines, reranking strategies, conventional generation, and individual CoCoRerank components through ablations.Source: Experimental Setup · p. 5

Baselines

• Evaluation includes retrieval-based, learning-based, and hybrid CMG baselines; three instruction-tuned LLMs; and MBR and Majority Voting rerankers.The LLMs are Qwen2.5-Coder-7B-Instruct (Hui et al. 2024), DeepSeek-Coder-V2-Lite-Instruct (DeepSeek-AI et al. 2024), and Meta-Llama-3.1-8B-Instruct (Grattafiori et al. 2024).Source: Baselines · p. 6

Evaluation Metrics

• Type prediction uses accuracy; scope uses BLEU; and subject uses BLEU, ROUGE-L, METEOR, and Pass@k.For scope BLEU, common filename and path separators are treated as token delimiters; Pass@k measures whether top-k reranked candidates contain a high-quality subject.Source: Evaluation Metrics · p. 6

Comparison of CMG Baselines

• METEOR relative gains are 13.99%, 17.96%, and 12.84% for Qwen2.5-Coder, DeepSeek-Coder-V2, and Meta-Llama-3.1, respectively.Source: Comparison of CMG Baselines · pp. 6–7
• Meta-Llama-3.1-8B-Instruct reaches 48.52% type accuracy, 43.18 scope BLEU, 25.91 subject BLEU, and 20.06 subject ROUGE-L, while FIRA retains the highest METEOR score.CoCoRerank improves type, scope, and subject generation across all three underlying LLMs.Source: Comparison of CMG Baselines · pp. 6–7
• Pass@1 improves by 17.10%, 28.72%, and 27.92% for Qwen2.5-Coder, DeepSeek-Coder-V2, and Meta-Llama-3.1, respectively.CoCoRerank consistently improves all Pass@k metrics across the three generators.Source: Comparison of CMG Baselines · p. 7

Comparison of Reranking Strategies

• CoCoRerank gains 4.75% BLEU, 12.79% ROUGE-L, 13.99% METEOR, and 17.10% Pass@1 over the base output by combining candidate consensus with commit-conditioned consistency.MBR improves over the base output across all metrics, while Majority Voting improves subject metrics and Pass@1 but lowers Pass@5 and Pass@2; CoCoRerank performs best across subject-generation and Pass@k metrics.Source: Comparison of Reranking Strategies · p. 7

Ablation Study

• Conventional prompting improves BLEU, ROUGE-L, and METEOR across all three generators; average ROUGE-L gains 10.08%, and DeepSeek-Coder-V2 gains 13.06% on average across the three metrics.The results suggest that jointly generating type and scope provides structural guidance for subject generation.Source: Ablation Study · p. 7
• Removing horizontal consistency reduces BLEU (1.63%), ROUGE-L (8.28%), METEOR (9.20%), and Pass@1 (9.82%), while removing vertical consistency more affects Pass@5 (12.31%) and Pass@2 (7.61%).Removing hard-negative training also consistently degrades all metrics; the components play complementary roles.Source: Ablation Study · p. 7

• The 86,688-commit dataset preserves the complete type (scope): subject structure and enables component-level evaluation.Source: Conclusion · pp. 7–8
• Experiments show that the framework improves both structural component prediction and subject generation quality.Source: Conclusion · p. 8
• The paper proposes a two-dimensional consistency-based reranking framework for LLM-based conventional commit message generation.Source: Conclusion · p. 8

Appendix Dataset Construction Pipeline

• The appendix provides step-by-step filtering procedures, precise data-volume changes, and label-normalization rules.Source: Appendix Dataset Construction Pipeline · p. 11

Detailed Data Collection and Preliminary Filtering

• The authors selected 206 repositories consistently following Conventional Commits, excluded those with fewer than 1,000 commits, and extracted diffs, messages, and repository metadata.Source: Detailed Data Collection and… · p. 11
• Filtering removes mechanically generated commits and retains action-oriented commit subjects using a V-DO subject filter.Mechanically generated examples include version updates, build synchronization, and automatic artifact maintenance commits.Source: Detailed Data Collection and… · p. 11
• Token filtering retains diffs of 64–32,767 tokens and subjects of 4–63 tokens, leaving 96,453 instances after preliminary filtering.The filters exclude overly trivial or ultra-long diffs and overly short or excessively verbose subjects.Source: Detailed Data Collection and… · p. 11

• Parsing messages into type, scope, and subject and removing samples missing type or scope reduced the dataset from 96,453 to 96,130 instances.Source: Detailed Structural Normalization Steps · p. 11
• Merging misspelled, abbreviated, and low-frequency type labels into standard categories left 95,959 instances while preserving message semantics.Uninterpretable arbitrary labels were eliminated to reduce label fragmentation and class imbalance.Source: Detailed Structural Normalization Steps · p. 11
• The final formatting pass corrected whitespace, delimiters, and casing, removed redundant appended metadata, and standardized samples into canonical CCS structure.Source: Detailed Structural Normalization Steps · p. 11

Detailed Semantic Filtering Process; LLM-Assisted Quality Assessment

• Semantic filtering first uses Qwen3-0.6B similarity screening to preserve samples scoring above 0.4; Qwen2.5-Coder-7B then assesses 26,185 ambiguous samples, retaining only those scoring above 6.Assessment covers accuracy, specificity, conciseness, and completeness on a 0–10 scale, followed by manual inspection of a random sample.Source: Detailed Semantic Filtering Process · p. 11

Case Study

• In the case study, CoCoRerank’s “ensure builder stops before checking logs” is more precise than the subject-only and CCS outputs, and consistency reranking resolves their type–subject conflict.Joint CCS generation supplies component and execution-order cues, while vertical consistency corrects the predicted type from refactor to test; the final subject connects the builder to log verification.Source: Case Study · pp. 11–13, Figure 4 · p. 12

Full Prompts

• The full-prompt section indicates that instance-specific inputs and retrieved examples are represented by angle-bracketed placeholders.The section presents the prompts used in the experiments.Source: Full Prompts · p. 13

Retrieval-Augmented Candidate Generation

• Retrieval-augmented candidate generation inserts five retrieved commit–message pairs as demonstrations and requests output in type(scope): subject format.The prompt defines the available change types and describes type, scope, and subject.Source: Retrieval-Augmented Candidate Generation · p. 13

Subject-Only Commit Message Generation

• The subject-only prompt asks for a short summary of the code change without requiring the generated message to follow CCS structure.Source: Subject-Only Commit Message Generation · p. 14

Hard-Negative Set Generation

• Hard-negative set generation uses the same prompt as retrieval-augmented candidate generation.Source: Hard-Negative Set Generation · p. 14