PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Tuning Large Language Models to Classify Pull Request-Issue Alignments: Going Beyond Prompting

Mustafa Yasir Altunhan, Hüseyin Özgür Kamalı, Eray Tüzün

arXiv:2609.01087v1cs.SE

TL;DR

PR–issue misalignment threatens traceability, defect localization, and maintainability, while existing approaches do not fully capture the taxonomy’s fine-grained distinctions. The study fine-tunes closed- and open-source LLMs and applies SHAP analysis, finding that fine-tuned models outperform baselines and that code diffs and textual bodies most influence predictions.

  • Problem

    PR–issue misalignment can undermine traceability, defect localization, and maintainability, while prior approaches and prompting do not fully support comprehensive fine-grained alignment classification.

  • Method

    The study extends and augments a PR–issue dataset, fine-tunes GPT-4o with instruction tuning and open-source LLMs with classification heads, and applies SHAP analysis to the best open-source model.

  • Results

    Fine-tuned LLMs outperform baseline models, with CodeLlama-7B performing best overall and code diffs, issue bodies, and PR bodies exerting the greatest influence on predictions.

  • Takeaways & Limitations

    Fine-tuning improves PR–issue alignment classification, while interpretability identifies the technical and textual fields shaping model decisions.

  • Takeaways & Limitations

    External validity is limited by the dataset’s size and single-repository scope, while implicit project requirements may not be represented in PR–issue fields.

Abstract

from arXiv · show

Context: Accurate alignment between pull requests (PRs) and corresponding issues is crucial for efficient software development and maintaining code quality, as misalignments can reduce traceability, hinder defect localization, and decrease maintainability. Objective: This study aims to improve automated PR-issue alignment classification by leveraging fine-tuned large language models (LLMs) across multiple alignment categories, and conducts interpretability analysis to investigate the effects of PR-issue fields on the predictions of fine-tuned LLMs. Method: Our methodology consists of dataset preparation, LLM fine-tuning, and interpretability analysis. We first extended an existing dataset and applied data augmentation to address class imbalance. GPT-4o was then fine-tuned via instruction tuning, and open-source LLMs including CodeLlama-7B, CodeQwen1.5-7B, StableCode-3B, CodeGemma-7B, and Deepseek-Coder-6.7B were fine-tuned using classification-specific heads. Interpretability analysis using Shapley Additive Explanations (SHAP) was conducted to examine the influence of PR-issue fields on predictions for the best-performing open-source LLM. Results: Fine-tuned LLMs outperformed baseline models, achieving average improvements of 6.15% in accuracy and F1-micro, 14.69% in F1-macro, and 6.15% in recall. CodeLlama-7B emerged as the best-performing fine-tuned LLM overall, while interpretability analysis revealed that code diffs together with issue body and PR body contents exert the greatest influence on predictions. Conclusions: Fine-tuning substantially enhances PR-issue alignment classification, improving both accuracy and efficiency. Interpretability analysis provides actionable insights into the dataset features driving alignment decisions, deepening understanding of how LLMs reason over software artifacts.

1 Introduction

PR–issue misalignment can damage traceability, defect localization, and maintainability, while prior work and prompting leave gaps in comprehensive, fine-grained alignment classification. This study addresses these gaps by fine-tuning LLMs and analyzing which PR–issue fields influence predictions.

  • Misalignment between PR commits and related issues can reduce traceability, hinder defect localization, and decrease maintainability.
  • Prompt-based LLM classification offers broad applicability but performs less effectively on domain-specific, fine-grained software artifacts.
  • Prior approaches focused mainly on detecting or untangling irrelevant changes rather than measuring whether issue requirements were fully satisfied.
  • The study fine-tunes closed-source and open-source LLMs for PR–issue alignment and compares their classification performance with previous approaches.
  • The study uses SHAP analysis to identify which PR–issue fields most influence fine-tuned model predictions.

2 Related Work

Related work establishes the prevalence and consequences of tangled commits, develops automated and human-in-the-loop untangling methods, and traces the evolution from conventional NLP to LLM-based software engineering analysis. The paper builds on this work by applying fine-tuning to comprehensive PR–issue alignment classification.

  • PR–Issue Alignment and Tangling: Tangled commits combine unrelated changes, complicating traceability, code review, debugging, and software engineering analyses.
  • PR–Issue Alignment and Tangling: The PR–issue alignment taxonomy distinguishes Exact, Missing, Tangling, and Missing and Tangling relationships.
  • Commit Untangling: Untangling research progressed from IDE-dependent clustering to graph-based, supervised, preventive, and human-in-the-loop approaches, each retaining distinct data or automation constraints.
  • NLP and LLMs: Text classification evolved from statistical and neural representations to transformer architectures and pretrained code-capable models.
  • NLP and LLMs: LLMs enable zero-shot and few-shot classification, but noisy, fine-grained software engineering tasks remain challenging for prompt-based methods.
  • Research Gap: This study extends prior work by fine-tuning closed-source and open-source LLMs rather than relying on ad-hoc prompting.

3 Methodology

The methodology combines dataset preparation, supervised fine-tuning of closed- and open-source LLMs, and SHAP-based interpretability analysis. Different fine-tuning strategies are used for GPT-4o and open-source models, with CodeLlama-7B selected for interpretability analysis based on accuracy.

  • Pipeline: The study extends an existing dataset and applies data augmentation to address class imbalance before model training.
  • Models: The evaluated models include GPT-4o, CodeLlama-7B, CodeQwen1.5-7B, StableCode-3B, CodeGemma-7B, and Deepseek-Coder-6.7B.
  • Fine-Tuning: GPT-4o is instruction-tuned to generate class labels, whereas open-source models receive newly added classification heads.
  • Interpretability: SHAP analysis is applied to CodeLlama-7B, the open-source model with the highest accuracy, to examine field contributions to classification decisions.
  • Pipeline: The methodology is organized into data collection and augmentation, model-specific fine-tuning, and SHAP-based interpretability analysis.

3.1 Data Collection and Processing

The dataset combines prior PR–issue pairs with manually labeled samples selected through stratified sampling, then uses stratified splitting and training-only text augmentation to address class imbalance. Annotation uses multiple raters and evaluates agreement before producing the final labels.

  • Dataset Construction: The initial dataset contained 194 PR–issue pairs across Exact, Tangling, Missing, and Missing and Tangling classes.
  • Splitting and Augmentation: Augmentation rephrased PR and issue text fields while preserving code diffs, and was applied to balance the training data.
  • Dataset Construction: Stratified random sampling and Cochran’s formula were used to select a statistically representative subset from the Transformers repository.
  • Manual Labeling: Two raters independently assigned four alignment labels, while a third experienced rater resolved disagreements and finalized annotations.
  • Manual Labeling: The annotation process achieved Cohen’s Kappa κ = 0.575 and IRR = 0.791, with 206 labeled instances across four classes.
  • Splitting and Augmentation: The extended dataset was split into training and test sets using 50%-50% stratified sampling based on merge status.

3.2 LLM-Based Analysis

The study fine-tunes GPT-4o and open-source LLMs to classify PR–issue alignment using structured artifact inputs and complementary classification designs. Open-source models use frozen backbones with lightweight trainable components, while predictions map to four alignment categories.

  • Fine-Tuning GPT-4o: GPT-4o is fine-tuned through instruction tuning using structured PR–issue prompts and supervised training.The workflow prepares chat-completion data containing system, user, and assistant components before fine-tuning GPT-4o.
  • Fine-Tuning Open Source LLMs: Open-source models use two binary classifiers to predict tangling and missing alignment, whose outputs form four alignment categories.The categories are Exact, Missing, Tangling, and Missing and Tangling.
  • Fine-Tuning Open Source LLMs: The open-source architecture replaces the language-modeling head with mask-aware mean pooling and a shared multilayer perceptron connected to classification heads.Pooling aggregates non-padding token embeddings into a fixed-length sequence representation.
  • Input and Evaluation: Inputs include issue and PR titles, descriptions, and code diffs, with context limited to 8,192 tokens for open-source models.Inputs exceeding the context window are truncated from the right side.
  • Fine-Tuning Open Source LLMs: Open-source training freezes the LLM backbones and updates only lightweight classification components, reducing computation and GPU-memory requirements.The setup also uses quantization, half-precision arithmetic, gradient accumulation when needed, and a fixed effective batch size of 16.
  • Experimental Setup: Evaluation queries fine-tuned GPT-4o repeatedly at zero temperature, whereas modified open-source models produce deterministic predictions and evaluate each sample once.The open-source models no longer include the language-modeling head used for stochastic decoding.

3.3 Interpretability Analysis

The interpretability analysis uses SHAP to quantify how PR–issue fields contribute to predictions from the best-performing open-source model. It separately analyzes the model’s missing and tangling classification heads and summarizes feature impact with mean absolute SHAP values.

  • Analysis Target: SHAP analysis is conducted on CodeLlama-7B, identified as the open-source model with the highest classification accuracy.The analysis examines how different PR–issue fields influence classification outcomes.
  • Rationale: SHAP evaluates feature contributions through trainable heads without requiring gradient flow through the frozen LLM backbone.Its model-agnostic framework also supports interactions between input features and output predictions.
  • Procedure: For the isMissing and isTangling heads, field-level SHAP values are computed independently and visualized with Beeswarm plots.Mean(|SHAP|) quantifies each field’s overall impact on model decisions.

4 Results

Fine-tuned LLMs generally improved PR–issue alignment classification across most metrics while reducing model size, although precision and some tangling-related specificity measures declined. CodeLlama-7B showed the strongest overall fine-tuned performance profile.

  • Performance Comparison with Previous Approach: 97.42% lower average storage size made fine-tuned models substantially more compact than base models.Fine-tuned models averaged 3.68 GB versus 143 GB for base models; proprietary GPT-4o was excluded from this comparison.
  • Performance Comparison with Previous Approach: 14.53% average improvement in F1-macro was reported for fine-tuned models, although the strongest base model remained higher.Fine-tuned models averaged 34.44%, while Llama-3.1-405B-Instruct reached 36.72%.
  • Performance Comparison with Previous Approach: Precision declined by 1.84 percentage points on average, and base models retained advantages in precision and some specificity measures.Base models also performed better in specificity for Missing and Tangling, while fine-tuned models generally led across other reported metrics.
  • Performance Comparison with Previous Approach: 6.15% average gains in accuracy and F1-micro, 14.69% in F1-macro, and 6.15% in recall were achieved by fine-tuned models.Fine-tuned models also averaged 97.43% less size than base models.
  • Performance Evaluation of Fine-Tuned LLMs: 66.00% accuracy, F1-micro, and recall made CodeLlama-7B the leading fine-tuned model on these metrics.CodeLlama-7B also achieved the highest fine-tuned weighted F1 at 62.66% and precision at 64.24%.
  • Performance Evaluation of Fine-Tuned LLMs: CodeLlama-7B combined leading core classification metrics with a compact 3.8 GB size.Its balanced performance profile positioned it as an effective solution for resource-constrained environments.

5 Threats to Validity

The study identifies threats to internal, construct, and external validity, including labeling and decision bias, issue atomicity assumptions, incomplete context, external resources, and limited dataset scope.

  • Overview: Validity threats span internal, construct, and external dimensions, including labeling errors, decision bias, issue atomicity, missing requirements, external resources, and dataset limitations.The authors specifically identify risks affecting both the study’s measurements and the broader generalizability of its conclusions.
  • Internal Validity: Manual labeling may introduce human error and subjective interpretation despite independent annotation, arbitration, and a replication package.Annotators may lack complete architectural context when evaluating the broader implications of code changes.
  • Internal Validity: PR and issue discussions can bias annotators toward classifications through contextual cues, although independent judgments and reconciliation reduce this risk.The authors note that confirmation bias may persist in rare cases involving especially strong or ambiguous contextual information.
  • Construct Validity: The analysis assumes issues are atomic and assignable to one of four alignment categories, but rare non-atomic issues may violate this assumption.The taxonomy’s applicability is therefore constrained when a single issue contains multiple unrelated requirements.
  • Construct Validity: Models may miss implicit requirements and external context because project guidelines, linked documents, and other resources are not included in the analyzed PR–issue fields.Examples include unobserved testing requirements and features specified in external documents.
  • External Validity: The dataset’s small, single-repository, primarily Python sample limits generalizability to other repositories, programming languages, and domains.Manual labeling constrained the sample size and repository coverage.

6 Discussion

The discussion describes practical uses for fine-tuned PR–issue alignment models, including local deployment, workflow integration, and code-review support. It also proposes future work on fine-tuning, augmentation, richer context, label calibration, and cross-project evaluation.

  • Practical Implications: Fine-tuned PR–issue alignment models could support tool development and integration into software-development workflows.The authors also propose cross-project and multilingual evaluation, richer context, and refined label mapping as research extensions.
  • Future Tool Development: Reduced storage requirements and improved performance make local deployment on edge devices more feasible without large server infrastructure.The discussion connects this deployment model with more flexible and decentralized usage scenarios.
  • Future Tool Development: Alignment classifiers could evaluate PR characteristics, inform decisions, support project management, and complement code-review bots with additional traceability context.The proposed integration targets workflow efficiency within integrated development environments.
  • Workflow Integration: The four-class taxonomy and fine-tuned models could prioritize PR reviews by alignment quality and support more structured software-maintenance workflows.The discussion links automated alignment assessment with traceability and consistency across development tasks.
  • Code Review Tool Integration: Embedding alignment classification in code-review tools would add PR–issue context as a complementary feature to existing review bots.The authors frame this as a practical transformation of code-review workflows.
  • Reflection of Coding Practices: Repository-level metrics could expose recurring PR–issue misalignment and code-tangling patterns beyond traditional productivity indicators.The proposed feedback could prompt teams to evaluate coding, collaboration, and maintenance practices.
  • Future Research: Future studies should compare full fine-tuning, LoRA, QLoRA, and Adapter-Tuning for accuracy, efficiency, and generalizability.The aim is to understand how alternative fine-tuning processes affect domain-specific classification.
  • Future Research: Future research can evaluate back-translation, synonym replacement, insertion, deletion, and LLM-based augmentation for alignment classification.These strategies are proposed to assess effects on model generalization and accuracy.

7 Conclusion

The study advances PR–issue alignment classification by fine-tuning closed-source and open-source LLMs, identifying influential input fields, and introducing a two-classification-head architecture. Fine-tuned models improve classification reliability, while interpretability results emphasize the importance of technical and textual context and indicate potential workflow applications despite generalizability constraints.

  • Fine-tuned LLMs outperform baseline and proprietary models, offering more reliable PR–issue alignment classification.
  • CodeLlama-7B proved most consistent overall, while other models excelled in specialized tasks or resource-limited contexts.Deepseek-Coder-6.7B, ChatGPT-4o, and CodeGemma-7B excelled in specialized tasks, while StableCode-3B showed promise for resource-limited contexts.
  • Interpretability analysis highlighted code diffs and issue descriptions as key factors shaping classification quality.The findings emphasize the combined role of technical and textual contexts in model predictions.
  • Fine-tuned models can support automated alignment checks, code review prioritization, code review tools, and repository-level metrics.The study also suggests that structured feedback from these applications can enhance code review quality and awareness of coding practices.
  • The dataset’s single-repository scope and issue atomicity assumptions constrain generalizability.The authors identify refined label mappings and evaluation frameworks as directions for strengthening methodological rigor.
Loading 2609.01087v1…