LabelMate: An LLM-Driven Framework for Refined Issue Report Labeling

1 Introduction

• Only 46.07% of over 10 million issue reports [29] and 54.59% of more than 13 million reports were labeled in prior studies.The introduction presents these figures as evidence of a gap between labeling's recognized value and its limited adoption.Source: §1 · p. 2
• Adoption is hindered by the effort to derive a project-suitable label list and the manual, labor-intensive, time-consuming work of assigning labels to incoming reports.Deriving a suitable taxonomy requires understanding project structure and workflows and anticipating issue types [29].Source: §1 · p. 2
• ETM requires specifying label counts a priori and relies heavily on manual inspection and interpretation; generic approaches limit labels to two to four categories, versus a median of 124 labels across 30 GitHub repositories.Source: §1 · p. 2
• A granular-labeling approach trained on project reports depends on a consistent, sufficiently large labeled dataset for each project.The paper argues this requirement does not effectively mitigate adoption challenges.Source: §1 · p. 2
• 8,020 unique labels were generated for 13,210 reports by Llama-3.1-8B-Instruct without label-space constraints, and 61.77% came from a single report.The introduction warns that such an inflated label set would hinder navigation and retrieval of related issues.Source: §1 · p. 2
• LabelMate consolidates LLM-generated labels from historical reports into a project-specific taxonomy, producing 275 labels and evaluating the framework on 16,500 reports from 30 repositories.The introduction describes clustering synonymous labels by embedding similarity as one way to reduce redundancy.Source: §1 · p. 3, RQ1: How can we derive a coherent… · p. 3
• 0.178 average cosine similarity: Qwen2.5-7B-Instruct assigning from the derived list scored highest across tested combinations of assigners and label lists.The comparison measures semantic alignment between assigned labels and issue-report contents.Source: RQ2: How do LLMs assign labels to… · p. 3

RQ3: Can we enhance our labeling pipeline using retrieval-augmented generation?

• 89.84% overall label accuracy: RAG retrieves a context-specific label subset, outperforming the full 275-label prompt while reducing prompt token usage and runtime.The full label list can increase inference cost and include labels irrelevant to a given report.Source: RQ3: Can we enhance our labeling… · p. 4
• Up to 100% coverage at embedding-similarity thresholds as high as 0.4 shows that LabelMate’s labels reflect widely used issue concepts and real-world labeling practices.The coverage is measured against labels in existing taxonomies.Source: RQ4: How does our coherent label list… · p. 4

• LabelMate is an end-to-end, domain-adaptive LLM framework that derives customized labels from historical reports and automatically assigns them to incoming reports.Source: RQ4: How does our coherent label list… · p. 4
• The replication package includes the implementation, prompts, usage instructions, experimental results and datasets, derived labels, assigned labels, and a real-world workflow example.Source: RQ4: How does our coherent label list… · p. 4

2 Experimental Setup

• The experimental setup covers LabelMate’s approach, model choices, dataset curation, and preprocessing pipeline.The dataset is partitioned into historical reports for taxonomy and knowledge-base construction and newer reports for label assignment.Source: §2 · p. 4, §2.3 · p. 6

2.1 Overview of Approach

• Historical reports yield candidate labels that are semantically refined into a coherent taxonomy, then context-specific retrieval selects labels for new reports.The assignment stage uses retrieval-augmented generation.Source: §2.1 · pp. 4–5

2.2 Model Selection

• The framework combines label-assigner LLMs, a label-evaluator LLM, and a text-embedding model for label generation and assignment, evaluation, clustering, and retrieval.The embeddings encode both labels and issue reports.Source: §2.2 · p. 5

2.2.1 Label Assigner LLMs.

• Gemma, Llama, and Qwen are the open-source, instruction-tuned 7–9B label assigners selected for prompt adherence and computational efficiency.Their moderate parameter size is described as a trade-off between performance and computational cost.Source: §2.2.1 · pp. 5–6

2.2.2 Label Evaluator LLM.

• The 70B evaluator assesses label assignments while building the RAG knowledge base, so its deeper reasoning need not be deployed repeatedly.The evaluator is substantially larger than the 7–9B assigner models.Source: §2.2.2 · p. 6

2.2.3 Text Embedding Model.

• all-mpnet-base-v2 provides embeddings for semantic clustering and retrieval, using a model fine-tuned on 1B sentence pairs for semantic similarity.The paper selects it to capture deep semantic relationships.Source: §2.2.3 · p. 6

2.3 Dataset Curation

• The dataset contains 16,500 reports from 30 repositories, selected using popularity, activity, and relevance criteria, with the newest 20% held out for testing.Repositories ranked among GitHub’s 500 most-starred were required to have at least 100 recent closed issues linked to resolution patches, and reports focused on software engineering.Source: §2.3 · p. 6

2.4 Data Pre-processing

• Preprocessing removes null characters, standardizes URLs, deletes hidden HTML comments, and replaces code or logs with concise natural-language summaries.The summaries are intended to reduce input noise while preserving semantic information needed for label assignment.Source: §2.4 · p. 7

• The section reports the motivation, approach, and findings of two motivational studies.Source: §3 · p. 7

3.1.1 Motivation.

• The study addresses the gap between earlier LLM evaluations using two to four broad categories and real repositories’ more fine-grained labels by testing under realistic labeling conditions.The authors describe this as the first investigation of LLM performance under these conditions.Source: §3.1.1 · pp. 7–8

3.1.2 Approach.

• The study uses repository label taxonomies, removes labels reflecting changing resolution status, and compares a Full label list prompt with two RAG prompts across k=1 to k=19.The RAG prompts use retrieved labels alone or retrieved labels together with their associated issue reports.Source: §3.1.2 · p. 9

3.1.3 Findings. The labels assigned by LLMs provide a more technically detailed description of the issue

• Qwen using the Labels only prompt at k=19 achieved the greatest alignment to the original labels, with precision 0.61, recall 0.59, and F1 0.60.Source: §3.1.3 · p. 12
• In 24 of 67 manually reviewed reports, at least one LLM configuration assigned labels that described the reported concern more technically than the original labels.For issue #44915, the original type:task label was less specific than the LLM-assigned type: enhancement and theme: containers labels for a request to run Docker tests in parallel.Source: §3.1.3 · p. 13
• Original labels may provide less detailed characterizations than LLM labels, so comparison only against originals may underestimate system performance.Source: §3.1.3 · p. 13
• Labels only was most efficient: runtime and prompt length remained relatively stable as retrieved examples increased, and it used fewer tokens and shorter runtime than Full Label List.Source: §3.1.3 · pp. 13–14, Figure 7 · p. 14
• LLM-assigned labels often differed from originals yet could describe issues more accurately, making original labels problematic ground truth for evaluating automated labeling.Source: §3.1.3 · p. 14

3.2.1 Motivation.

• MS2 investigates LLMs’ unconstrained labeling behavior because prior studies restricted labeling to broad categories such as bug, feature, question, or documentation.The authors state that, to their knowledge, this behavior had not previously been explored.Source: §3.2.1 · p. 14

3.2.2 Approach.

• The prompt frames the LLM as the repository maintainer, supplies issue titles and bodies, and uses guidelines with negative examples to steer label generation.Positive examples are omitted to avoid constraining creativity and to retain broad thematic coverage.Source: §3.2.2 · p. 15

3.2.3 Findings.

• 5,925 to 8,020 unique labels were generated per model from 13,210 issue reports, creating large, fragmented label spaces.Table 3 summarizes the generated labels across the three label-assigner LLMs.Source: §3.2.3 · p. 15, Table 3 · p. 16
• At least 59% of unique labels across all models came from a single issue report, indicating many labels were overly specific.Such labels may capture incidental details rather than generalizable issue categories.Source: §3.2.3 · p. 16, Table 3 · p. 16
• Surface normalization handles formatting variants, but not distinct text labels with nearly identical meanings, such as “ui” and “user interface.”The authors identify semantic synonymy as a source of redundant labels that can fragment categorization.Source: §3.2.3 · p. 16
• Across the three models, reports received an average of 8.14 labels, but only 2.49 labels per report came from more than one model.An average of 5.65 labels per report were generated by only one model, showing disagreement even under the same prompt.Source: §3.2.3 · p. 16

4 Results

• The section organizes the motivation, approaches, and findings for each of the four research questions.Source: §4 · p. 16

4.1.1 Motivation.

• The authors frame a fixed label list as promoting consistency, reusability, and interpretability, while manual taxonomy design is labor-intensive and context-dependent.Source: §4.1.1 · p. 17

4.1.2 Approach.

• The process starts with 14,960 unique labels and retains labels generated across multiple reports or by multiple LLMs, leaving 6,172 candidates.This filtering is intended to favor categorical stability and reusability.Source: §4.1.2 · p. 17
• A distance threshold of 0.3 provides the best trade-off between cohesion and differentiation, producing 529 clusters with 11.67 labels per cluster on average and a median of 8.Only 22 of the 529 clusters contain a single label.Source: §4.1.2 · pp. 17–18
• For each cluster, the most frequently generated label represents the concept, and technology-specific tool names are generalized to categories such as compiler or database.Frequency is used as an indicator of prominence in developer discourse.Source: §4.1.2 · pp. 17–18
• The authors identify substantial synonymy in unconstrained labels: at threshold 0.3, most labels group into multi-label clusters, indicating semantic overlap.The clustering-density finding supports the authors’ conclusion that LLMs generate multiple variations of underlying concepts when unconstrained.Source: §4.1.2 · p. 18
• After discussing and reaching consensus on inclusion, the authors excluded 254 labels as non-descriptive, yielding a final coherent list of 275 descriptive labels.Source: §4.1.2 · p. 19

4.2.1 Motivation.

• Unlike prior LLM-labeling studies focused on broad categories such as bug, feature, question, or documentation, this question examines assignments constrained to a project-specific list.The section frames this constraint as a way to test LLMs on the greater thematic diversity of real-world software projects.Source: §4.2.1 · p. 19

4.2.2 Approach.

• The evaluation embeds each report’s title and body and its assigned labels with all-mpnet-base-v2, then averages cosine similarity across labels.The models are evaluated on 13,210 training reports using the 275-label list; a report with no assigned labels receives a similarity of 0.Source: §4.2.2 · pp. 19–20

4.2.3 Findings. Labels assigned from our derived list are more semantically aligned than those assigned

• 0.143 average cosine similarity for the strongest baseline (Llama with Catolino labels) trails 0.147 for the weakest derived-list configuration (Llama).All configurations using the derived list outperform those using any of the three baseline lists.Source: Table 5 · p. 21, §4.2.3 · pp. 21–22
• For 13,210 training-set issue reports, Qwen without ensembling achieves the highest overall alignment, averaging cosine similarity 0.178 in the comparison of developer- and LLM-assigned labels.Source: Table 5 · p. 21, §4.2.3 · p. 22
• 0.171 average cosine similarity for developer-assigned labels is exceeded by five of the eight derived-list configurations.The authors interpret this as evidence that structured, LLM-generated labels can match or surpass original labels in semantic relevance.Source: Table 5 · p. 21, §4.2.3 · p. 22
• Agreement between LLMs ranges from Moderate to Almost Perfect; Colavito-list configurations range from Substantial to Almost Perfect, while other lists range from Moderate to Substantial.The paper notes that the Colavito list offers only four choices, compared with at least nine for the other lists.Source: §4.2.3 · p. 22

4.2.3 Findings

• Figure 9 presents the label evaluation prompt.Source: Figure 9 · p. 23

4.3.1 Motivation.

• RAG dynamically retrieves a context-specific subset of labels to reduce the costs and positional bias of prompting with 275 labels and avoid irrelevant-label drift or hallucinations.The approach draws labels from historically labeled issue reports, based on the premise that semantically similar issues often share labels.Source: §4.3.1 · p. 23

4.3.2 Approach.

• The evaluator accepted 26,575 of 30,554 Qwen-assigned labels, or 87%, as accurately reflecting their corresponding issue reports, and these labelings formed the knowledge base.The labels came from Qwen assignments on the derived label list for training-set reports, without ensembling other LLMs.Source: §4.3.2 · pp. 23–24
• Human reviewers agreed with the evaluator on 72 of 80 sampled labels (90%), and a second author concurred with the evaluator in seven of the eight disagreements.The leading author reviewed a random, statistically representative sample of 80 assignments; the authors’ agreement supported the evaluator’s reliability for assessing label accuracy.Source: §4.3.2 · p. 24
• For each new report, the pipeline embeds its title and body, retrieves the k most similar reports with evaluator-validated labels, and prompts the LLM using Labels Only or Labels and Issue Reports.The evaluation compared these RAG prompts with the full derived label list, three prior label-list baselines, and developer-assigned labels; cosine similarity measured semantic alignment, while the evaluator assessed label correctness.Source: §4.3.2 · p. 24
• At k=3, the Labels Only prompt reached 0.178 average cosine similarity and the Labels and Issue Reports prompt reached 0.169.Both prompts achieved their highest average cosine similarity at k=3; Figure 10 also reports average labels per issue across retrieval settings.Source: §4.3.2 · p. 24, Figure 10 · p. 25
• The final evaluation used evaluator-based accuracy measures alongside cosine similarity, and Mann–Whitney U-tests assessed whether configuration differences were statistically significant.The evaluator directly judged whether labels accurately reflected reports, whereas cosine similarity measured semantic relatedness; the test was selected because the per-report values were not normally distributed.Source: §4.3.2 · p. 26
• The RAG-based Labels only prompt achieves the highest label accuracy among the evaluated prompting configurations.Source: §4.3.3 · p. 26

4.3.3 Findings.

• At k=3, RAG Labels only reduces average runtime per issue and prompt tokens versus the full 275-label prompt while maintaining comparable semantic alignment.Source: §4.3.3 · pp. 27–28, Table 8 · p. 27
• Higher cosine similarity tends to accompany higher evaluator-based labeling accuracy, making it a scalable quality indicator when direct LLM correctness evaluation is impractical.Source: §4.3.3 · p. 27
• 89.84% overall label accuracy, 89.18% label accuracy per issue report, and 81.34% of reports with 100% label accuracy make RAG Labels only at k=3 the highest-performing configuration.It also has the second-highest average cosine similarity, 0.178.Source: §4.3.3 · pp. 26–28
• 66.47% overall label accuracy, 66.47% label accuracy per issue report, and 66.47% of reports with 100% label accuracy were the lowest scores, achieved by the Catolino et al. label-list prompt.This configuration also had the second-lowest average cosine similarity, 0.135.Source: §4.3.3 · p. 27
• At k=3, 86% of retrieved issue reports come from the same domain as their corresponding unseen reports, while cross-domain retrieval can still find comparable underlying problems.Source: §4.3.3 · pp. 27–28

4.4.1 Motivation.

• The research question asks whether the 275-label list can serve as a meaningful baseline for constructing taxonomies in other collaborative software repositories.The comparison focuses on four repositories not used in the label-generation dataset.Source: §4.4.1 · p. 28

4.4.2 Approach.

• The comparison uses four repositories spanning popularity levels, filters out non-issue and project-specific labels, then measures semantic coverage using cosine similarity across label embeddings.Coverage is the share of generic labels in an existing list with at least one sufficiently similar counterpart among the 275 derived labels.Source: §4.4.2 · pp. 28–29

4.4.3 Findings. Our derived label list provides substantial coverage of label taxonomies used by existing

• Six derived labels exactly match labels used by at least one repository, while other labels have near matches or differ mainly by categorical prefixes.Examples include “test” and “testing,” and repository prefixes such as “category:” and “comp:”.Source: §4.4.3 · p. 30
• Up to 100% of labels in the four repositories’ lists matched a derived label at similarity thresholds as high as 0.4, with at least 65% covered at thresholds of 0.4 or below.The authors describe 0.4 as a conservative threshold and interpret the overlap as evidence that the derived labels capture related concepts.Source: §4.4.3 · pp. 30–31

Real-world Implementation Recommendations

• Automated labeling can be integrated without pre-labeled training data or model fine-tuning, and the paper provides an example implementation for issue-triage workflows.The example uses a locally loaded LLM and a Flask application that processes incoming issue reports.Source: §5 · p. 31

Supporting Project-Specific Label Evolution and Insight

• Projects can adopt the curated 275-label list or generate a project-specific taxonomy, then reapply the generation pipeline as new issue types emerge.The paper also describes periodically rerunning the pipeline to keep the taxonomy representative of an evolving issue space.Source: §5 · p. 31

Supporting Automated Issue Resolution

• Issue labels can also route reports to automated coding agents suited to their resolution, alongside their traditional role in human-centered triage.The paper frames this as a potential use in workflows incorporating autonomous coding agents.Source: §5 · p. 31

Enabling Practical Adoption of Automated Issue Report Labeling

• LabelMate can reduce manual labeling effort by shifting work from choosing labels across a taxonomy to validating a smaller set of pre-assigned labels.The labels may be assigned directly or sent to a contributor for validation first.Source: §5 · p. 32

Supporting Project-Specific Label Evolution and Insight

• Project-tailored labels can reflect changing issue themes and priorities, providing an analytical lens for planning and understanding project evolution.The paper names recurring needs for fixes, performance improvements, and refactoring as examples that can inform planning.Source: §5 · p. 32

6.1 External Validity

• The evaluation uses issue reports from 30 of the 500 most-starred GitHub repositories that met strict inclusion criteria, so performance may differ on other repositories, including proprietary industrial ones.Reports needed at least one linked resolution patch; the dataset nevertheless includes large-scale, industry-developed open-source projects.Source: §6.1 · p. 32
• The label assigners are open-source, relatively lightweight LLMs, and filtering removes labels generated for only one report by one model or referring to specific tools or infrastructure.These choices support a coherent taxonomy of recurring categories applicable across repositories, but may exclude labels describing specific issue details.Source: §6.1 · pp. 32–33

6.2 Internal Validity

• Prompt sensitivity can cause performance variability independent of experimental design, despite testing prompt variations and publishing template visualizations and justifications.The authors identify changes in wording or structure as factors that can influence LLM outputs.Source: §6.2 · p. 33

6.3 Construct Validity

• Cosine similarity and LLM-based accuracy judgments may not fully characterize label accuracy, while the authors report that existing assigned labels are unreliable ground truth.They cite strong agreement between evaluator-LLM and human assessments as support for treating the measures as practical indicators of relevance.Source: §6.3 · p. 33

7.1 Three-to-Four Label Approaches

• LabelMate derives detailed label sets that can be assigned automatically, unlike approaches restricted to three or four categories.The paper also describes its use of free open-source LLMs as more accessible than approaches requiring purchased tokens for proprietary models.Source: §7.1 · p. 34

7.2 Assigning Labels used in Practice

• IssueBERT assigns labels from the repository’s top 10% most frequent labels but requires project-specific fine-tuning, limiting use where a consistent labeling scheme is absent.LabelMate derives and applies tailored labels without pre-labeled data.Source: §7.2 · p. 34

7.3 Deriving a Label List

• LLM-generated candidate labels can be refined through clustering, grouping semantically similar labels into a curated taxonomy with minimal effort.The paper presents this as a scalable alternative to manually deriving labels.Source: §7.3 · p. 34
• ETM requires choosing the number of categories in advance and manually inferring label names from generated keywords.Unlike word-presence topic modeling, the paper describes its LLM-based approach as context-aware and capable of interpreting nuanced language.Source: §7.3 · p. 34

7.4 LLMs Labeling, Annotating, or Categorizing Software Engineering Content

• For software-engineering content beyond issue reports, prior LLM studies assign one label from predefined or manually derived options, whereas this method supports any number of labels.Its automated pipeline curates labels from historical issue reports and assigns them to reports.Source: §7.4 · p. 35

8 Conclusion

• LabelMate derives a coherent, context-specific taxonomy from historical issue reports, then uses it to label new reports.The conclusion describes the framework as end-to-end and domain-adaptive.Source: §8 · p. 35
• Semantic clustering and a fixed, coherent label list are reported to overcome the fragmentation and redundancy of unconstrained LLM-generated labels.The conclusion identifies fragmentation and redundancy as limitations of unconstrained generation.Source: §8 · p. 35
• Up to 89.8% label accuracy accompanies substantial gains in inference efficiency in time and token usage.The conclusion attributes the narrowed candidate label space to a RAG-based mechanism.Source: §8 · p. 35
• Future work could integrate LabelMate into issue-triaging workflows, including duplicate detection and developer assignment.The conclusion describes this as a step toward a holistic issue-report management assistant.Source: §8 · p. 35
• LabelMate is presented as supporting scalable, interpretable, resource-efficient workflows powered by open-source LLMs.The conclusion frames adoption across repositories of varying sizes and computational capacities as practical.Source: §8 · p. 35