PaperlayerView exact revision on arXiv

Source-linked AI summary

CAMGen: A Core Change-Aware Multi-View Framework for Commit Message Generation

Bo Xiong, Weisheng Chen, Zongen Ren, Yifeng Zhu, Xutian Li, Runbang Yan, Chong Wang, Peng Liang

arXiv:2610.31552v1cs.SE

TL;DR

Existing CMG methods provide limited context for understanding how local edits affect functionality implemented across multiple entities. CAMGen anchors functional, structural, and historical context on core entities and gives the combined views with the diff to an LLM. It outperforms representative approaches on automatic metrics, and human evaluation finds more complete messages with clearer rationale and impact explanations.

  • Problem

    Existing CMG methods provide limited support for connecting local edits to functionality jointly implemented by multiple code entities, which matters for explaining a change and its impact.

    Source: §1 · p. 2, §2.3 · p. 7

  • Method

    CAMGen identifies core entities, selects semantic clusters containing them as functional context, adds structural dependencies and historical exemplars, and supplies all views with the diff to an LLM.

    Source: §1 · p. 4, §7 · p. 28

  • Results

    CAMGen consistently outperforms representative CMG approaches on automatic metrics, and human evaluation finds more complete messages with clearer rationale and more informative impact explanations.

    Source: §7 · p. 29

  • Takeaways & Limitations

    The results demonstrate the value of function-aware multi-view context for generating accurate, informative, and project-consistent commit messages.

    Source: §7 · p. 29

    CAMGen can still produce incorrect messages despite relevant diff and repository evidence, including messages that mischaracterize a fix or emphasize secondary edits.

    Source: §5.5.1 · p. 26

Abstract

from arXiv · show

Commit messages help developers understand code changes and support software maintenance. Recent approaches use repository context to improve commit message generation (CMG) with large language models (LLMs). However, existing CMG methods provide limited support for understanding how local edits affect functionality jointly implemented by multiple code entities. This functional understanding is essential for generating commit messages that explain the main change and its impact. To address this problem, we propose CAMGen, a core change-aware multi-view commit message generation framework that incorporates functional context into generation. CAMGen identifies core entities that represent the primary change intent. It then forms candidate functional units through repository-wide semantic clustering and uses those containing the core entities to construct a Functional Semantic View. This view supplies semantic context for interpreting the affected functionality. CAMGen also uses context provided by the Structural Dependency View and the Historical Exemplar View. The context from all three views is then provided to the LLM together with the code diff to guide CMG. The experimental results show that CAMGen outperforms representative CMG baselines across three programming languages. Human evaluation further shows that CAMGen generates more complete commit messages with clearer explanations of change rationale and impact. These results demonstrate the value of functional context and complementary repository information for CMG.

1 INTRODUCTION

Existing CMG methods often select repository context around a diff but do not systematically connect entities that jointly implement affected functionality. CAMGen anchors analysis on core entities and combines functional, structural, and historical context to guide commit-message generation.

1 INTRODUCTION

  • Existing context-selection methods do not systematically organize entities that jointly implement functionality, so local or fragment-level context may miss affected responsibilities.The challenge is to help LLMs describe the primary change and its functional impact.Source: §1 · p. 2
  • In the motivating Hudi example, structural and historical context foreground local edits, whereas functional clustering yields a message closer to the primary intent of adding Hive-style partitioning support.The example connects configuration declarations, parameter propagation, partition-path processing, and tests as one functional unit.Source: §1 · pp. 2–3
  • Without functional context, messages may describe local operations or generic changes rather than the affected functionality, leaving developers to inspect code to determine change impact.This weakens the support commit messages provide for code review, program comprehension, and maintenance.Source: §1 · p. 3
  • CAMGen identifies core entities and combines their structural dependencies and functionally related code with historical diff-message examples to guide an LLM.Its Functional Semantic View is built by clustering repository code and selecting units containing core entities.Source: §1 · p. 4
  • The paper evaluates CAMGen across three programming languages using automatic and human evaluation, ablations, and analysis of low-scoring cases.It also examines implications for integrating functional context into LLM-based CMG.Source: §1 · p. 4

2 RELATED WORK AND BACKGROUND

Prior CMG research spans retrieval, learned generation, hybrid methods, and LLM-based approaches, while code clustering has been used to organize repository functionality. The section motivates connecting that functional organization to diff interpretation.

2.1 Commit Message Generation

  • Existing CMG approaches range from retrieval and learned generation to hybrid and LLM-based methods, with repository context and historical examples used in several approaches.Earlier approaches also used rules and templates.Source: §2.1 · p. 5
  • CMG evaluation studies examine performance across commit types and assess how automatic metrics and LLM quality judgments correspond to human evaluations.Reported findings include weaker generation on implicit messages and stronger correlation between edit distance and expert editing effort than BLEU or METEOR.Source: §2.1 · p. 6

2.2 Code Clustering for Program Comprehension

  • Prior work uses code clustering to organize repository entities for program comprehension and functional documentation, including feature-oriented groupings of files and methods.Examples include clustering linguistic topics, recovering architectural components, and generating feature descriptions linked to code artifacts.Source: §2.2 · pp. 6–7

2.3 Conclusive Summary

  • Existing CMG methods retrieve similar examples or directly related code, but do not explicitly connect diffs to broader functionality jointly implemented by changed and related entities.The section identifies this broader functional context as important for understanding diff impact and its implications.Source: §2.3 · p. 7
  • CAMGen addresses this gap by anchoring semantic-clustered functional units and structural dependencies on core entities, while historical examples supply project terminology and writing conventions.The combined context accompanies the diff to help capture the primary change and its functional impact.Source: §2.3 · p. 7

3 METHODOLOGY

CAMGen identifies core entities in a diff, builds structural, functional, and historical views around them, and supplies the combined context with the diff to an LLM. The views respectively capture explicit dependencies, functionally related code, and project-specific message conventions.

3 METHODOLOGY

  • CAMGen follows three stages: identify core entities, construct three repository-context views, then generate a message from the diff and combined context.The views are ViewStruct, ViewFunc, and ViewExemplar.Source: §3 · p. 7

3.1 Overview

  • The overview distinguishes ViewFunc from structural expansion by retrieving semantically related entities beyond one-hop dependencies, while ViewExemplar supplies project-specific terminology and style.Together with ViewStruct, these views are combined with the diff for generation.Source: §3.1 · p. 8

3.2 Core Entity Identification

  • Core entity identification uses an LLM to locate classes or functions carrying primary logic changes, enhancements, or defect fixes, then resolves their names to repository entities.The resolved entities are shared inputs to the structural and functional views.Source: §3.2 · p. 8

3.3 Structural Dependency View (ViewStruct)

  • ViewStruct builds a repository-level graph of entities and call, inheritance/implementation, and instantiation relations, then prunes context to one-hop, role-relevant neighbors.For function cores, it keeps selected callee and parent-method skeletons and caller call-site blocks; for class cores, it retains inheritance contracts, instantiation snippets, and relevant type summaries.Source: §3.3 · p. 9

3.4 Functional Semantic View (ViewFunc)

  • ViewFunc clusters semantically similar repository entities into candidate functional units and selects clusters containing mapped core entities to provide code context for interpreting functionality.The pipeline encodes entities with CodeBERT, projects normalized vectors to 20 dimensions with UMAP, clusters with HDBSCAN, reassigns noise-labeled core entities to a nearest non-noise cluster, and limits oversized clusters to M=10 entities by default.Source: §3.4 · pp. 9–10

3.5 Historical Exemplar View (ViewExemplar)

  • ViewExemplar retrieves earlier commits from the same repository by semantic similarity and supplies their diff-message pairs as examples of project terminology and writing style.The current diff and eligible historical diffs are embedded, L2-normalized, and compared by cosine similarity.Source: §3.5 · pp. 10–11

3.6 Commit Message Generation

  • The LLM receives the raw diff with all three views: structural dependencies, selected functional-unit code, and historical diff-message examples.The functional context helps interpret changed-entity responsibilities and the behavioral impact of the diff.Source: §3.6 · p. 11

4 RESEARCH DESIGN

The research design evaluates CAMGen’s overall performance, component contributions, and perceived message quality across languages, while also examining core-entity identification and low-scoring cases. It combines benchmark comparisons with automatic and human evaluation.

4.1 Research Questions

  • Four research questions test overall effectiveness, component contributions, developer-perceived quality, and causes of low automatic scores.The ablations examine core-entity identification and the individual and combined contributions of the three context views.Source: §4.1 · pp. 11–12

4.2 Research Datasets

  • RQ1 uses 2,498 C++, 2,469 Java, and 2,555 Python commits; separate samples support entity identification, view ablation, and human evaluation.The RQ1 set totals 7,522 commits; the other samples comprise 100 commits for core-entity identification, 1,000 for ablation, and 50 for human evaluation.Source: §4.2 · pp. 12–13

4.3 Comparison Baselines

  • Five baselines are selected for reproducibility and adoption, retrained or reconstructed without evaluation-set commits, and compared under a shared evaluation pipeline.The historical retrieval corpus is also restricted to earlier commits from the same repository.Source: §4.3 · pp. 13–14

4.4 Evaluation Design

  • Automatic metrics measure lexical, sequential, and semantic similarity between generated messages and references; human evaluation assesses change coverage, rationale, impact, and language quality.The five reference-based metrics are BLEU, ROUGE-L, METEOR, CIDEr, and SBERT Cosine.Source: §4.4 · p. 14, §4.4.1 · pp. 14–15

4.4.1 Automatic Evaluation.

  • The five automatic metrics capture n-gram overlap, word-order overlap, lexical matches, distinctive phrases, and sentence-level semantic proximity.METEOR also supports morphological and synonym matches; CIDEr weights n-grams using TF-IDF.Source: §4.4.1 · pp. 14–15

4.4.2 Human Evaluation.

  • Three experienced evaluators blindly score messages on five dimensions, including Change Impact Explanation, using repository snapshots to inspect changes and context.Ratings use a 0–4 scale; the five dimensions are Rationality, Comprehensiveness, Conciseness, Expressiveness, and Change Impact Explanation.Source: §4.4.2 · pp. 15–16
  • The human-evaluation ratings are averaged across evaluators and commits, with ranking agreement and paired-difference confidence intervals also reported.Paired comparisons use 50,000 bootstrap resamples of the 50 commits.Source: §4.4.2 · p. 15

4.5 LLM Selection

  • Three models generate CMG outputs, while two DeepSeek models identify core entities; all models use temperature 0.0.DeepSeek-V4-Pro is used exclusively for core-entity identification.Source: §4.5 · p. 16, Table 2 · p. 16

5.1 Cross-Language Overall Performance of CAMGen (RQ1)

Across Java, C++, and Python, the best CAMGen configuration surpasses the strongest baseline on all five automatic metrics. The gains span three LLM backbones, with the leading model varying by language and metric.

5.1.2 Analysis of RQ1.

  • CAMGen’s best configuration surpasses CCT5, the strongest baseline, on all five metrics in Java, C++, and Python.The comparison uses a unified data and evaluation protocol and includes retrieval-based approaches and a code-change-pretrained baseline.Source: Table 3 · p. 17, §5.1.2 · p. 17
  • For Java, CAMGen with DeepSeek-V4-Flash scores 25.25 METEOR and 55.26 SBERT Cosine, improving over CCT5 by 3.18 and 7.81 points, respectively.Its gains over CCT5 on ROUGE-L and CIDEr are 2.31 and 0.70 points, respectively.Source: §5.1.2 · p. 17, §5.1.1 · p. 17
  • DeepSeek-V4-Flash leads all five metrics for Java and C++; on Python, DeepSeek-V3.2 leads BLEU, ROUGE-L, and CIDEr, while Flash leads METEOR and SBERT Cosine.GPT-5.4 remains competitive across all three languages, and the best CAMGen configuration exceeds the strongest baseline for every language.Source: §5.1.2 · p. 18

5.2 Reliability of Core Entity Identification in CAMGen (RQ2.1)

Both tested LLMs identify core entities with strong performance against manual annotations, and DeepSeek-V4-Pro scores higher than DeepSeek-V4-Flash at both entity and commit levels.

5.2.1 Results of RQ2.1.

  • 0.8301 Precision, 0.8141 Recall, 0.8220 F1, 0.7985 Accuracy, and 0.7700 CmtAccuracy are DeepSeek-V4-Pro’s scores.DeepSeek-V4-Pro leads DeepSeek-V4-Flash on all five reported metrics.Source: Table 4 · p. 18, §5.2.1 · pp. 18–19
  • DeepSeek-V4-Pro exactly recovers the annotated core-entity set for 77 of 100 commits, compared with 73 for DeepSeek-V4-Flash.Both models achieve entity-level Recall above 0.80, and their F1 and CmtAccuracy rankings are consistent.Source: §5.2.1 · p. 19

5.2.2 Analysis of RQ2.1.

  • Both models balance precision and coverage; CAMGen uses the identified core entities as shared anchors for ViewStruct and ViewFunc.The reported precision scores indicate that the models do not indiscriminately include every entity related to a modification.Source: §5.2.2 · p. 19

5.3 Complementarity of Multi-View Repository Context (RQ2.2)

The ablation compares three repository-context views, alone and in combination, while holding the generator and samples constant. The complete three-view configuration performs best across all five metrics, and the analyses describe distinct, complementary contributions from the views.

5.3.1 Results of RQ2.2.

  • ViewExemplar is the strongest single view, improving BLEU, ROUGE-L, METEOR, CIDEr, and SBERT Cosine by approximately 24.0%, 9.7%, 4.7%, 40.1%, and 0.3% over diff-only.ViewStruct and ViewFunc also improve the four text-overlap metrics over diff-only; ViewFunc improves all five metrics.Source: §5.3.1 · p. 20
  • Among two-view setups, ViewStruct + ViewExemplar leads BLEU, METEOR, CIDEr, and SBERT Cosine, while ViewFunc + ViewExemplar leads ROUGE-L.ViewFunc + ViewExemplar improves all five metrics over diff-only.Source: §5.3.1 · p. 20
  • The complete three-view configuration achieves the highest scores among all eight configurations on all five metrics.It scores 13.05 BLEU, 26.36 ROUGE-L, 23.35 METEOR, 11.73 CIDEr, and 57.10 SBERT Cosine.Source: §5.3.1 · p. 20, Table 5 · p. 20

5.3.2 Analysis of RQ2.2.

  • ViewExemplar provides project-specific terminology and writing styles, while ViewStruct or ViewFunc adds repository code context that improves semantic alignment with exemplars.ViewFunc supplies code from candidate functional units containing core entities, helping interpret changed-code responsibilities and edit implications.Source: §5.3.2 · p. 21
  • Removing any view degrades all five scores, indicating that the three views contribute complementary information rather than redundant context.Structural and functional views expose code relationships and functionality; exemplars supply similar diffs and human-written messages for project-specific terminology and style.Source: §5.3.2 · p. 21

5.4 Human Evaluation of Commit Message Quality (RQ3)

Human evaluators rated messages on five quality dimensions, including how well they explain change rationale and impact. CAMGen’s complete context configuration led on four dimensions, and the analyses identify functional context as especially valuable while structural and historical context add complementary gains.

5.4 Human Evaluation of Commit Message Quality (RQ3).

  • Three evaluators rated 50 commits; post-discussion Spearman correlations ranged from 0.6620 for Comprehensiveness to 0.6963 for Rationality.The highest correlations were for Rationality and Change Impact Explanation.Source: §5.4 · pp. 21–22, §5.4.1 · p. 23, Table 8 · p. 23

5.4.1 Results of RQ3.

  • CAMGen Full has the highest mean ratings for comprehensiveness, rationality, change impact explanation, and conciseness, and exceeds all five baselines across every dimension.Its respective scores on those four dimensions are 3.34, 3.05, 3.08, and 3.29.Source: §5.4.1 · p. 22, Table 6 · p. 22, §5.4.2 · p. 23
  • Compared with diff-only, CAMGen Full improves all five dimensions by approximately 0.69, 0.75, 0.60, 0.47, and 0.05 points, with every 95% confidence interval above zero.The dimensions are comprehensiveness, rationality, change impact explanation, conciseness, and expressiveness, respectively.Source: §5.4.1 · p. 22, Table 7 · p. 23

5.4.2 Analysis of RQ3.

  • Removing ViewFunc causes larger rating losses than removing either other view on the first four dimensions; both loss contrasts have confidence intervals above zero.ViewFunc also achieves the highest single-view ratings on these dimensions, connecting distributed edits to a common functional objective and behavioral impact.Source: §5.4.1 · p. 22, §5.4.2 · pp. 23–24
  • Structural dependencies and historical exemplars each further improve the first four dimensions when added to functional context, supplying code relationships and project-specific wording conventions.These complementary sources support more complete and focused explanations of rationale and impact.Source: §5.4.2 · p. 24
  • Richer context improves comprehensiveness and conciseness together, while expressiveness varies little across configurations generated by the same LLM.The authors attribute the paired improvement to retaining essential change information while filtering irrelevant or repetitive content.Source: §5.4.2 · p. 24

5.5 Qualitative Analysis of Low-Scoring Cases (RQ4)

The analysis groups low-scoring CAMGen outputs into core-entity misidentification, assessment mismatch, and incorrect generation. Assessment mismatch can arise even when a generated message accurately describes the change, because it differs from the reference.

5.5.1 Results of RQ4

  • Core-entity misidentification can send context construction off course, causing messages to miss the commit’s primary functional intent.In the Airflow example, CAMGen anchors on a helper function rather than the operator change removing deprecated pod-naming usage.Source: §5.5.1 · pp. 24–25
  • Assessment mismatch does not necessarily indicate a generation error: a semantically valid message may score poorly because it differs from the reference.The analysis identifies low-quality references and semantically similar messages as two subcategories.Source: §5.5.1 · pp. 25–26
  • When a reference commit message is too general or omits code-change information, generated messages can describe changes more fully yet receive low automatic metric scores because the reference lacks corresponding content.Source: §5.5.1 · p. 25
  • Semantically similar messages can receive low lexical scores when they describe the same intent with different wording or detail.The example contrasts two descriptions of URL-parsing support; the paper notes that a single reference may not capture every valid description.Source: §5.5.1 · p. 26
  • Incorrect generation occurs when the LLM mischaracterizes the change or emphasizes secondary edits despite relevant diff and repository evidence.The authors relate these errors to difficulty interpreting change semantics and distinguishing the main change from accompanying modifications.Source: §5.5.1 · p. 26

6 THREATS TO VALIDITY

The paper discusses threats to internal, external, construct, and conclusion validity, along with mitigation measures. The findings’ scope and reproducibility are bounded by evaluator composition, evaluated languages and models, core-entity identification errors, and possible vendor API updates.

6 THREATS TO VALIDITY

  • The validity discussion addresses factors that may affect findings and describes experimental mitigations or planned improvements.Source: §6 · pp. 26–27

6.1 Internal Validity

  • Internal validity may be affected by LLM versions, prompts, decoding settings, and human judgments; the experiments standardize configurations and scoring procedures to mitigate these risks.Human evaluation used three evaluators, concealed message sources, independent initial ratings, and discussion of substantial disagreements; future work calls for broader evaluator backgrounds.Source: §6.1 · pp. 27–28

6.2 External Validity

  • CAMGen was evaluated on Java, C++, and Python with two open-source LLMs and one closed-source LLM, so generalization to other languages and models remains to be evaluated.The authors propose extending evaluation to JavaScript, Go, Rust, and additional LLMs.Source: §6.2 · p. 28

6.3 Construct Validity

  • Core-entity identification can miss or misidentify entities, affecting the subsequent construction of structural and functional context.DeepSeek-V4-Pro achieved 0.8301 Precision, 0.8141 Recall, and 0.8220 F1 against manually annotated gold labels; the authors propose enlarging the annotated dataset and improving error detection and correction.Source: §6.3 · p. 28

6.4 Conclusion Validity

  • Vendor API updates may change model behavior and affect exact reproducibility even when inputs and configurations remain the same.The paper provides a replication package and fixes temperature at 0.0 to support reproduction and reduce generation randomness.Source: §6.4 · p. 28

7 CONCLUSIONS

CAMGen addresses the lack of functional context in commit message generation by retrieving functional and structural context around core entities and similar historical examples. The authors report stronger automatic-metric performance and more complete messages with clearer rationale and impact explanations; they outline future work and note that the replication package is available.

7 CONCLUSIONS

  • CAMGen identifies core entities and retrieves functional and structural context, alongside similar diff-message pairs, to address insufficient functional context in commit message generation.The combined context is provided to an LLM for message generation.Source: §7 · p. 28
  • CAMGen consistently outperforms representative CMG approaches on automatic metrics, and human evaluation finds more complete messages with clearer rationale and more informative impact explanations.The authors attribute these results to the value of function-aware multi-view context for generating accurate, informative, and project-consistent messages.Source: §7 · p. 29
  • Future work will examine context selection across bug fixes, feature additions, and refactorings, incremental context updates, and adding issue reports and code review discussions.The proposed evaluation of issue and review evidence will test whether it helps produce higher-quality messages.Source: §7 · p. 29

DATA AVAILABILITY

Loading 2610.31552v1…