PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Enhanced Commit Message Generation via Issue Information: An Exploratory Study

Zongen Ren, Wei Shi, Bo Xiong, Chong Wang, Peng Liang

arXiv:2608.22004v1cs.SE

TL;DR

Commit message generation has not systematically established how issue information contributes beyond code diffs. The paper introduces ISAC and ApacheCM-Issue to evaluate issue-augmented LLM inputs, finding consistent automatic-metric improvements while revealing trade-offs between raw issue text, summaries, and historical examples.

  • Problem

    The specific contribution of issue information to LLM-based commit message generation remains systematically underexplored, despite its task background and change motivation.

  • Method

    The paper proposes ISAC, combines code diffs with issue information, constructs the ApacheCM-Issue aligned dataset, and evaluates multiple context configurations with two LLMs.

  • Results

    Adding original issue information improves all five automatic metrics across evaluated model configurations and outperforms the reproduced SOTA baselines.

  • Takeaways & Limitations

    Issue information helps LLMs capture code-change intent, while the selection and presentation of context affect automatic and human-evaluated message quality.

  • Takeaways & Limitations

    Findings primarily apply to commits with identifiable, usable issue records and do not cover the full range of available models and inference settings.

Abstract

from arXiv · show

Commit messages help developers understand code changes, support collaboration, and improve long-term maintenance. However, the use of issue information alone as the external context for LLM-based CMG has not been systematically studied. We propose an ISsue-Augmented framework for Commit message generation (ISAC) by combining code diffs with issue information as LLM input. To support the evaluation, we construct ApacheCM-Issue, a commit-issue aligned dataset built upon ApacheCM by linking commits with issues from GitHub and Apache Jira. Using samples from Scala, Java, and C++ projects, we evaluate four input configurations using two representative LLMs, GPT-5.5 and DeepSeek-V4-Flash in different reasoning configurations. The results show that incorporating issue information consistently improves LLM-based CMG across all evaluated model configurations and metrics, with the largest gains observed for CIDEr. Incorporating a similar historical commit further improves automatic metric scores, while replacing full issue information with a structured issue summary decreases them. ISAC also outperforms the four reproduced state-of-the-art (SOTA) CMG baselines across all five automatic metrics on the experimental dataset. The human evaluation further shows that structured issue summaries may improve perceived completeness, although replacing the original issue information can sacrifice contextual details and lead to worse results on automatic metrics.

1 Introduction

The paper addresses the underexplored use of issue information as task-level context for LLM-based commit message generation. It introduces ISAC and ApacheCM-Issue, then evaluates issue augmentation and related context strategies.

  • Motivation: Many commit messages omit either what changed or why, motivating automatic generation of more detailed and informative messages.Prior evidence reports that around 44% of commit messages lack either “what” or “why” information.
  • Research gap: Existing CMG studies mainly augment code diffs with code context, repository history, or retrieved examples, leaving issue information insufficiently studied.Issue records can provide task background, reproduction steps, constraints, and change motivation that are difficult to infer from diffs alone.
  • Motivating example: For the same diff, adding issue information produces a more specific message that conveys both code operations and the motivation and impact of the change.The motivating example contrasts a generic diff-only description with an issue-augmented message containing task-level semantics.
  • Contributions: The paper constructs ApacheCM-Issue, a multi-language aligned dataset containing 47,664 issue-commit samples from 49 Apache repositories.The dataset links code diffs, commit messages, and issues collected from GitHub and Apache Jira.
  • Contributions: ISAC combines code diffs with issue information as LLM input to improve the model’s ability to capture the rationale behind code changes.The evaluation uses GPT-5.5 and DeepSeek-V4-Flash with BLEU, ROUGE-L, METEOR, CIDEr, and SBERT-Cos.
  • Contributions: Issue information improves generated-message quality, similar historical commits add project-specific expression patterns, and structured summaries are unsuitable as direct replacements for raw issue text.These conclusions are evaluated across the paper’s model configurations and five automatic metrics.

2 Related Work

Prior CMG research spans rule-based, retrieval-based, learning-based, and LLM-based methods, increasingly incorporating development context. However, the specific contribution of issue information remains systematically underexplored.

  • CMG approaches: CMG methods include rule-based, retrieval-based, learning-based, and LLM-based approaches that generate commit messages from code changes.The approaches differ in whether they use templates, retrieved examples, trained neural models, or prompted general-purpose LLMs.
  • Rule-based methods: Rule-based methods use predefined rules, templates, or heuristics to summarize diff types, impact scopes, and commit features.Some rule-based work explicitly targets both what changed and why the change was made.
  • Retrieval-based methods: Retrieval-based methods provide similar historical diffs or commit examples to capture project terminology, writing styles, and semantic relevance.These methods include context-aware retrieval, exemplars, and retrieval-augmented generation.
  • Learning-based methods: Learning-based methods formulate CMG as neural machine translation and improve diff understanding through structural modeling, copying, representation learning, and pretraining.The surveyed methods include CoDiSum, ATOM, CoreGen, FIRA, COME, and CCT5.
  • LLM-based methods: LLM-based methods frame CMG as prompting, enabling zero-shot or few-shot generation and benefiting from retrieval-based in-context learning.Prior comparisons report advantages for LLM-generated messages in human evaluation and effectiveness for CMG.
  • Research gap: Although prior work studies historical commits, repository context, retrieved examples, and issue-related information, the isolated effect of issue information remains underexplored.The paper identifies this as the specific gap addressed by its issue-augmented evaluation.

3 ApacheCM-Issue Dataset

ApacheCM-Issue aligns commits, code diffs, messages, and external issues through filtering and leakage control. The resulting dataset contains 47,664 pairs from 49 repositories, with substantial language and repository variation.

  • Dataset construction: ApacheCM-Issue extends ApacheCM with explicit alignments among issues, code diffs, and human-written commit messages for issue-aware CMG research.It combines GitHub commits with issues sourced from GitHub Issues and Apache Jira.
  • Dataset schema: The dataset captures commit, repository, linkage, and issue fields, with issue_title and issue_body supplying the textual augmentation.The nested issue object also stores metadata such as state, timestamps, user, and labels.
  • Quality assurance: Four filtering rules remove missing or ambiguous links, contaminated pull requests, invalid issue information, and target leakage to preserve alignment quality.The rules sequentially filter an initial pool of 249,830 ApacheCM commit records.
  • Quality assurance: 47,664 leakage-controlled commit-issue pairs remain after filtering.The final dataset retains the code diff, human-written commit message, and traceable issue information needed for issue-augmented CMG.
  • Demographics: 47,664 pairs come from 49 Apache repositories, including 37,587 Java pairs (78.9%) and 6,493 Scala pairs (13.6%).The remaining 3,584 pairs (7.5%) originate from repositories primarily written in C++, Rust, TypeScript, Python, Erlang, Go, and Lua.
  • Demographics: 19.08% of 249,830 original ApacheCM commits are retained, and repository-level retention varies with project-specific metadata quality and issue-tracking practices.The retention rate is the number of final ApacheCM-Issue pairs divided by original ApacheCM commits in the same repository.

4 Methodology

ISAC augments code diffs with associated issue information through a three-stage pipeline, then generates commit messages using structured prompts. Its alternatives add a similar historical commit or replace the original issue with a structured summary.

  • ISAC Framework: ISAC retrieves the issue associated with a target code diff, augments the diff–issue pair, and generates a commit message in three stages.The stages are Issue Retrieval, Issue-Aware Augmentation, and Commit Message Generation.
  • Issue-Aware Augmentation: Issue-only augmentation combines the target diff and issue title/body using separate <diff> and <issue> delimiters.The delimiters distinguish code-level modifications from task-level semantics such as motivation, constraints, and project terminology.
  • Issue-Aware Augmentation: Similar historical commit augmentation adds a same-project historical diff, commit message, and related issue information as a reference example.The example is selected from ApacheCM-Issue using a retrieval procedure.
  • Issue-Aware Augmentation: Structured issue augmentation replaces the original issue information with a compressed, structured issue summary.The summary organizes information around functionality, concepts, rationale, directives, and outcomes.
  • Commit Message Generation: The augmented prompt is sent directly to the LLM without additional fine-tuning, with output constrained to a JSON object containing one commit_message key.This design isolates the effect of injected issue information from additional fine-tuning factors.
  • Similar Historical Commit Retrieval: Historical candidates are restricted to earlier commits in the same repository and ranked using equally weighted normalized lexical and semantic similarity.Lexical and semantic scores are independently min-max normalized before computing Sim(ct,ch) = 0.5 · LexicalSim(ct,ch) + 0.5 · SemanticSim(ct,ch).
  • Similar Historical Commit Retrieval: The highest-scoring historical candidate is added to the target diff and issue information as an augmented LLM input.The added fields are the historical diff, issue information, and commit message.

5 Research Design

The research design tests issue augmentation against diff-only generation, historical examples, structured summaries, and reproduced baselines across multiple models, languages, metrics, and human judgments. It combines a 1,061-pair experimental dataset with automatic and human evaluation.

  • Research Questions: The study investigates whether issue information improves LLM-generated commit messages and formulates four research questions covering augmentation, baselines, history, and structured summaries.The RQs progress from the core diff-versus-issue comparison to broader benchmarking and two augmentation variants.
  • Experimental Dataset: The experimental dataset contains 1,061 commit-issue pairs sampled from Scala, Java, and C++ repositories.The sample includes 363 Spark pairs, 364 Camel pairs, and 334 pooled C++ pairs.
  • Input Configurations: Four input configurations compare Diff, Diff + Issue, Diff+Issue+History, and Diff+Structured Issue.The history setting adds a similar same-project commit’s diff, issue information, and commit message; the structured setting replaces the original issue information.
  • LLM Configurations: Experiments use GPT-5.5 and DeepSeek-V4-Flash in four reasoning configurations, with generation parameters held constant across input settings.The configurations include GPT-5.5 at none and high effort, plus DeepSeek chat and reasoner modes.
  • Evaluation: Generated messages are evaluated with BLEU, ROUGE-L, METEOR, CIDEr, and SBERT-Cos, covering lexical, structural, term-salience, and semantic perspectives.The study also conducts human evaluation on 50 stratified samples spanning the three language groups.
  • Human Evaluation: Human evaluators anonymously rank 17 candidate messages per sample using clarity, completeness, and correctness criteria.The candidates include outputs from the four LLM configurations and four settings, plus the original human-written message.

6 Results and Analysis

The evaluation examines issue augmentation across LLM configurations and metrics, finding consistent gains over code-diff-only inputs, especially for CIDEr.

  • Impact of issue augmentation: Issue augmentation improves all five metrics across every evaluated LLM configuration compared with code diffs alone.Average gains range from 8.70% in SBERT-Cos to 30.70% in CIDEr.
  • Impact of issue augmentation: 30.70% is the largest average gain, achieved by CIDEr after adding issue information.The other average improvements are 23.60% for BLEU, 20.48% for ROUGE-L, 25.21% for METEOR, and 8.70% for SBERT-Cos.
  • Impact of issue augmentation: Issue information adds task background, defects, requirements, and expected behavior that code diffs often omit.This additional context supports messages that better reflect change intent rather than only summarizing code operations.
  • Performance across LLM configurations: The benefit remains robust across model families and reasoning settings, although relative improvements vary by configuration and metric.DeepSeek-V4-Flash shows the largest relative improvement across all five metrics, while reasoning effort does not universally improve scores.
  • Performance across LLM configurations: GPT-5.5 with high effort achieves the highest Diff+Issue scores on BLEU, ROUGE-L, CIDEr, and SBERT-Cos, while GPT-5.5 with no effort leads METEOR.The corresponding high-effort scores are 18.68, 32.79, 14.06, and 63.89; the no-effort METEOR score is 34.30.

6.2 Effectiveness of ISAC Compared with SOTA CMG Baselines (RQ2)

ISAC is compared with reproduced SOTA commit-message-generation baselines and with historical-commit augmentation. It outperforms the baselines, while similar historical commits provide further gains.

  • Results of RQ2: Both the best- and worst-performing ISAC configurations outperform all four reproduced SOTA baselines across all five automatic metrics.The comparison uses issue-only augmentation and the replicated baselines CCT5, NNGen, RACE, and CoRec.
  • Analysis of RQ2 Results: ISAC’s gains over CCT5 indicate stronger lexical, phrase-level, and semantic alignment with reference commit messages.BLEU, ROUGE-L, METEOR, and CIDEr capture lexical or phrase-level alignment, while SBERT-Cos captures semantic relevance.
  • Benefits of Incorporating Similar Historical Commits for CMG: GPT-5.5 with high effort achieves the highest absolute scores with historical augmentation, while DeepSeek-V4-Flash reasoner has the largest CIDEr gain at 42.97%.The high-effort GPT-5.5 scores are 24.52, 38.01, 39.50, 19.85, and 66.29 for BLEU, ROUGE-L, METEOR, CIDEr, and SBERT-Cos.
  • Benefits of Incorporating Similar Historical Commits for CMG: Similar historical commits improve every metric across all four LLM configurations beyond issue augmentation alone.Average gains are 29.49% for BLEU, 14.84% for ROUGE-L, 13.67% for METEOR, 41.08% for CIDEr, and 2.66% for SBERT-Cos.

6.4 Effect of Structured Issue Augmentation (RQ4)

Replacing original issue information with structured summaries consistently reduces automatic performance, despite preserving broad semantic intent. The summaries may omit technical and project-specific details needed for specific commit messages.

  • Results of RQ4: Replacing original issue titles and bodies with structured summaries decreases all five metrics across all four LLM configurations.Every configuration scores lower with Diff+Structured Issue than with Diff+Issue.
  • Information loss introduced by structured issue summaries: The largest average decreases occur in CIDEr, BLEU, and METEOR, which are more sensitive to informative terms and content coverage.The structured summary condenses each issue into five predefined fields, increasing the risk of omitted concrete details.
  • Information loss introduced by structured issue summaries: Structured summaries preserve much of the original issue’s broad semantic intent, as reflected by the smaller SBERT-Cos decrease.Preserving general intent alone is insufficient to match the specificity of human-written commit messages.
  • Information loss introduced by structured issue summaries: Original issue information better balances intent-level semantics with concrete technical evidence required for commit-message generation.Structured summaries may omit technical details and project-specific expressions essential for high-quality messages.
  • Effect of the reasoning setting of LLMs: GPT-5.5 with high effort is more robust to issue compression, particularly on METEOR and SBERT-Cos, but remains below original-issue performance.The overall degradation persists across both ChatGPT and DeepSeek configurations.

6.5 Human Evaluation

Human evaluation indicates that original issue information improves clarity and generally improves completeness without reducing correctness, while historical examples can introduce irrelevant details. Structured issue summaries receive the highest completeness ratings but lose contextual detail and do not improve clarity or correctness over original issue text.

  • Original issue information improves clarity across all four LLM configurations and improves completeness in most cases compared with Diff.Correctness remains comparable to the Diff setting.
  • Diff+Issue+History consistently underperforms across all three human-evaluation scores.Retrieved examples may add irrelevant information, causing unrelated details to be included or important target changes to be overlooked.
  • Diff+Structured Issue achieves the highest average completeness but does not surpass original issue information in clarity or correctness.Compressing issues may remove contextual details needed to describe changes precisely.
  • Human and automatic evaluations can show inconsistent trends because reference-based metrics measure similarity to human-written messages rather than quality directly.Human-written messages may not always be ideal references for judging clarity and completeness.

7 Threats to Validity

The study identifies threats involving commit–issue alignment, generalizability, issue-field coverage, evaluation constructs, and conclusion validity. These limitations constrain how broadly the observed benefits of issue information should be interpreted.

  • Internal validity: Imperfect commit–issue alignments may misrepresent a diff’s background, purpose, or scope and affect estimates of issue information’s benefit.Issue informativeness also varies substantially across records.
  • External validity: The experimental data come from 47,664 samples across 49 Apache repositories, with 1,061 samples from six projects, limiting transfer to other ecosystems.Proprietary, small-scale, or differently governed projects may have different conventions and issue practices.
  • External validity: Findings primarily concern commits with identifiable, usable issue records and two LLM families across four configurations.Commits without useful issue context and broader model or inference settings are underrepresented.
  • Construct validity: Using only issue titles and bodies may omit contextual information from states, timestamps, labels, comments, and developer details.
  • Construct validity: Automatic metrics are approximate proxies, while human evaluation covers 50 samples and two evaluators whose ratings may not represent broader developer preferences.The two evaluation modes can capture different aspects of commit-message quality.

8 Conclusions & Future Work

The paper introduces ISAC and ApacheCM-Issue for issue-augmented commit message generation, finding that original issue information improves automatic and human-evaluation outcomes. It proposes refining context selection, structured representations, and human evaluation in future work.

  • Conclusions: ISAC combines code diffs with issue information, while ApacheCM-Issue provides aligned diffs, commit messages, and issue records.
  • Conclusions: Adding original issue information improves all five automatic metrics across evaluated configurations and outperforms reproduced SOTA baselines.
  • Conclusions: Similar historical commits improve reference-based metrics, whereas structured issue summaries perform worse than original issue text automatically.
  • Conclusions: Human evaluation finds that original issue information improves clarity and generally improves completeness without compromising correctness.Structured summaries achieve the highest completeness ratings, while historical examples may introduce unrelated information.
  • Future Work: Future work should select more relevant issue fragments and historical commits, preserve important details in structured representations, and expand human evaluation.

Data Availability

The study’s code and data are publicly available in a GitHub repository containing the resources needed to reproduce the analysis.

  • The repository provides the commit–issue aligned dataset, processing scripts, prompts, model outputs, evaluation scripts, and documentation.
Loading 2608.22004v1…