PaperlayerView exact revision on arXiv

Source-linked AI summary

TianoForge: An Automated Bug Triage Approach for the TianoCore UEFI Firmware Development Community

Nazanin Siavash, Terrance E. Boult, Armin Moin

arXiv:2608.23259v1cs.SE

TL;DR

Manual bug triage in large open-source projects is resource-intensive and difficult to scale, so TianoForge automates four TianoCore EDK II triage tasks with LLMs and RAG. It reduces average triage time from 260.57 hours to 7.08 minutes, a 99.95% reduction.

  • Problem

    Manual bug triage is resource-intensive and difficult to scale in large open-source projects, where efficiently processing incoming reports remains challenging.

  • Method

    TianoForge integrates LLMs, retrieval-augmented generation, and domain-specific prompting to automate invalid detection, duplicate detection, prioritization, and developer assignment.

  • Results

    99.95% reduction: integrated triage averages 7.08 minutes versus 260.57 hours for manual triage across 37 GitHub-native EDK II issues.

  • Takeaways & Limitations

    The experiments demonstrate that LLMs can effectively support several bug triage activities in the TianoCore EDK II ecosystem.

  • Takeaways & Limitations

    The evaluation is limited to the TianoCore EDK II ecosystem, so the approach’s generalizability to additional open-source repositories remains to be assessed.

Abstract

from arXiv · show

We propose a novel approach to bug triage in the TianoCore open-source UEFI firmware development ecosystem. This integrated approach, called TianoForge, deploys the state of the art in artificial intelligence, specifically machine learning, to enable automated bug triage. This includes invalid bug report detection, duplicate bug report detection, bug report prioritization, and bug report assignment. We use various Generative Pretrained Transformer (GPT) Large Language Models (LLMs) with and without Retrieval Augmented Generation (RAG) to automate these tasks. Given the crucial role of bug triage in software maintenance and the huge number of untriaged issues in the TianoCore community, in particular, their primary project, EDK II, we expect a significant impact on the efficiency of TianoCore software maintenance processes, primarily bug triage and resolution. Our experimental study shows that TianoForge reduces the average bug triage time from around 11 days to approximately 7 minutes, which is a 99.95% reduction.

1 Introduction

TianoForge addresses the challenges of manual bug triage in the TianoCore EDK II ecosystem with an integrated automated workflow. The study evaluates its ability to support multiple triage tasks rather than treating them in isolation.

  • Motivation: Bug tracking systems collect and organize defect reports, but efficiently processing those reports remains challenging.These reports provide information needed to reproduce and resolve issues.
  • Motivation: Manual bug triage is resource-intensive and impractical for large-scale open-source projects with high report volumes.Delays in assignment and frequent reassignment increase resolution time and reduce efficiency.
  • Research gap: Prior automated approaches generally address one or two triage tasks in isolation and have largely studied general-purpose projects rather than TianoCore EDK II.The passage identifies the lack of prior automated-triage study focused on the TianoCore EDK II ecosystem.
  • Contribution: TianoForge is an integrated automated bug triage script designed for the TianoCore EDK II ecosystem.It performs invalid detection, duplicate detection, priority prediction, and developer assignment within a unified workflow.
  • Evaluation: The experimental study examines whether automation reduces resolution time and evaluates effectiveness and efficiency across duplicate detection, prioritization, invalid-report detection, and assignment.The study also investigates factors involved in triage decisions.

2 Background

This section establishes bug triage and bug report enhancement as foundations for managing software defects, then introduces LLMs, prompt engineering, and RAG as technologies supporting automated triage. It emphasizes that project-specific retrieval can supplement LLM knowledge for bug-report analysis.

  • Bug triage foundations: Bug tracking systems structure defect reporting, tracking, and resolution through summaries, descriptions, metadata, comments, attachments, and status updates.
  • Bug triage foundations: Bug triage evaluates reported issues for invalidity, duplication, prioritization, and developer assignment to support efficient resource allocation.
  • Bug report enhancement: Incomplete, ambiguous, missing, or redundant report information can reduce triage accuracy, motivating enhancement through refined titles, added context, and submission filtering.Improving report quality supports more accurate and consistent triage decisions.
  • LLMs and prompting: LLMs use semantic relationships in unstructured text for software-engineering tasks, while zero-shot and few-shot prompting specialize their behavior without additional training.Prompts can also incorporate domain knowledge and decision criteria.
  • Retrieval-augmented generation: RAG retrieves external documents and adds them to prompts, grounding bug-triage predictions in historical reports, documentation, discussions, and resolved issues.These project-specific sources can provide context for duplicate detection, prioritization, and developer assignment.

3 Related Work

Prior bug-triage research framed the task as supervised classification or optimization using report content and metadata, with traditional models targeting developer expertise and report characteristics. Recent work extends this foundation through enriched linguistic features and hybrid LLM–structural architectures for complex triage scenarios.

  • Foundational approaches: Bug triage research commonly used supervised text classification or optimization over bug-report content and metadata, alongside duplicate detection, prioritization, and validity assessment.Traditional techniques included Naïve Bayes classifiers, topic models, and bug tossing graphs.
  • Feature-enriched prioritization: Term graphs enriched through neighborhood relationships improved bug prioritization over traditional TF-IDF and baseline deep-learning methods.This approach treated words as nodes and modeled their relationships.
  • Feature-enriched prioritization: A CNN-based prioritization framework combining syntactic, semantic, and emotional features produced substantial cross-project F1-score improvements.The findings highlight the value of combining multiple linguistic feature types for prioritization.
  • Hybrid architectures: Hybrid models combining LLM-derived representations with multi-scale feature fusion and GNNs consistently improved multiple evaluation metrics for triage scenarios requiring relational information.The architecture captures global semantic context and local structural relationships.

4 Proposed Approach

TianoForge is a unified LLM-based workflow that automates invalid-report detection, duplicate detection, priority prediction, and developer assignment for TianoCore EDK II. It combines project bug reports, historical Bugzilla context, hybrid retrieval, and task-specific prompting into a sequential end-to-end triage script.

  • Integrated workflow: TianoForge integrates invalid detection, duplicate detection, bug prioritization, and developer assignment within one automated workflow.The sequential script produces validity status, duplicate status, priority level, and a recommended developer.
  • Data and systems: The approach evaluates 75 GitHub-native EDK II bugs, including 37 bugs with a known first assignee, using Systems A and B for in-context LLM prompting.System B uses the historical Bugzilla XML corpus as a retrieval source, whereas System A uses fixed or task-specific in-context examples.
  • Retrieval layer: RAG configurations combine dense BGE-base-env1.5 retrieval with sparse BM25 retrieval through Reciprocal Rank Fusion.The shared retrieval layer supports retrieval from the GitHub bug corpus and historical Bugzilla records.
  • Invalid bug detection: Invalid detection classifies reports as genuine defects or invalid cases such as spam, usage questions, designed behavior, or insufficient reproduction information.System A uses 3 fixed few-shot examples per class, while System B retrieves the top 3 similar Bugzilla bugs using RRF.
  • Priority and assignment: Priority prediction assigns low, medium, or high urgency, while developer assignment predicts an assignee from the closed set observed among 37 labelled bugs.Both tasks use domain-specific heuristics and in-context examples; assignment predictions outside the closed set are corrected or defaulted to the first assignee.
  • Duplicate bug detection: Duplicate detection retrieves candidate reports with the hybrid ranker and uses LLMs to decide whether the issue shares the same root cause and codebase component with an existing report.System B additionally retrieves up to 3 confirmed duplicate pairs from Bugzilla as demonstrations.

5 Experimental Study · 5.1 Experimental Dataset

The experimental dataset is a curated collection of closed, bug-labelled EDK II issues retrieved from GitHub and organized for automated triage evaluation. It contains 2,610 issues, including historical Bugzilla-transferred reports and a labelled GitHub-native subset for supervised assignment evaluation.

  • 5.1 Experimental Dataset: The dataset is constructed from the publicly available TianoCore EDK II GitHub issue tracker and was curated on April 3, 2026.The repository state was fixed at the collection date, and the dataset had not previously been used for this research problem.
  • 5.1 Experimental Dataset: Issues are retrieved programmatically through the GitHub REST API using a custom Python script with pagination.The collection includes only issues, excluding pull requests, and applies the remaining filters described below.
  • 5.1 Experimental Dataset: Only closed issues labelled type:bug are included in the dataset.These constraints define the issue population used for the experimental study.
  • 5.1 Experimental Dataset: 2,610 issues comprise the final EDK II dataset, represented in CSV format with issue metadata, triage information, and categorical features.Fields include issue number, title, state, first assignee, assignment timestamp, triage hours, creation and closure timestamps, URL, milestone, comment count, package, priority, and state.
  • 5.1 Experimental Dataset: 2,535 issues are Bugzilla-transferred reports, while 75 are GitHub-native issues filed directly on GitHub.The two subsets distinguish historical reports migrated from the legacy Bugzilla tracker from reports created natively in GitHub.
  • 5.1 Experimental Dataset: The 75 GitHub-native issues were created between December 2024 and February 2026, with priority labels distributed as 32 medium, 28 low, and 15 high.These labels characterize the GitHub-native subset by reported priority.
  • 5.1 Experimental Dataset: 37 of the 75 GitHub-native issues have a known first assignee and constitute the labelled subset G_L for supervised bug-assignment evaluation.The labelled subset is formed from GitHub-native issues with available first-assignee information.

5.2 Evaluation Metrics

TianoForge evaluates all four tasks with standard classification metrics, using macro and weighted aggregation across classes. It also measures practical efficiency through bug resolution and triage times, including comparison with the current manual process.

  • Classification metrics: All four tasks use standard classification metrics based on ground-truth and predicted labels, with TP, FP, FN, and TN defined per class.These quantities denote true positives, false positives, false negatives, and true negatives for each class.
  • Classification metrics: Accuracy measures the fraction of correctly classified bugs, while precision measures the fraction of predicted positives that are truly positive.Recall measures the fraction of true positives correctly identified, and F1 is the harmonic mean of precision and recall.
  • Aggregation strategies: Macro averaging computes the unweighted mean across classes, whereas weighted averaging weights each class by its support.Class support is the number of true instances belonging to that class.
  • Efficiency metric: Total bug resolution time combines triage time, from issue creation to first assignment, with fix time, from first assignment to issue closure.Because fix time depends on developer effort and is unaffected by the triage pipeline, reducing triage time is the relevant efficiency objective.
  • Efficiency metric: 260.57 hours (10.86 days) is the mean triage time for 37 labeled GitHub-native issues, serving as the manual-process baseline.The integrated script’s setup, inference, and total elapsed times are measured with Python’s time.time() and averaged over three runs.

5.3 Experimental Setup

The experimental setup evaluates ten LLMs from OpenAI and Anthropic in a Python pipeline running on Google Colab. Retrieval combines dense and sparse methods through Reciprocal Rank Fusion, while OpenAI inference uses deterministic settings.

  • Implementation: The pipeline is implemented in Python and runs on Google Colab using libraries including openai, anthropic, chromadb, and scikit-learn.Additional listed libraries are sentence-transformers, rank_bm25, and tqdm.
  • Models and inference: Ten LLMs from OpenAI and Anthropic are evaluated across all tasks.OpenAI models use temperature 0.0 for deterministic outputs, while Claude models use default settings.
  • Retrieval: Dense retrieval uses BAAI/bge-base-en-v1.5 embeddings indexed in ChromaDB with cosine similarity.Sparse retrieval uses BM25Okapi, and the two methods are combined through Reciprocal Rank Fusion with k0 = 60.

5.4 Experimental Analysis

TianoForge’s experiments evaluate invalid detection, duplicate detection, priority classification, assignment, and an integrated pipeline across multiple models and prompting systems. The integrated pipeline averages 7.08 minutes, reducing historical triage time by 99.95%.

  • Duplicate and invalid detection: Duplicate detection under pair-based ground truth finds one of two confirmed pairs on average, while System A’s three zero-false-positive models achieve P+ = 1.000, Mac-F1 = 0.826, and Acc = 0.960.System A generally outperforms System B because Bugzilla RAG increases false positives and transfers poorly to GitHub issues.
  • Duplicate and invalid detection: Only claude-sonnet-4-6 System A detects confirmed invalid bugs, identifying one of four per run on average with R+ = 0.250, F1+ = 0.333, Mac-F1 = 0.653, Acc = 0.947, and Wgt-F1 = 0.938.RAG benefits are model-dependent: System A helps Claude models, whereas System B improves GPT models’ false-positive behavior.
  • Priority classification: Priority classification has no universal winner: gpt-5.4-mini System A leads accuracy and weighted metrics with Acc = 0.671 and Wgt-F1 = 0.663, while claudesonnet-4-6 System A leads Mac-R = 0.616 and Mac-F1 = 0.620.System A substantially outperforms System B across all ten models; for example, gpt-5.4-mini achieves Acc = 0.671 versus 0.542.
  • Bug assignment: gpt-5.5 System A achieves the best assignment performance on 37 labeled issues, with Acc = 0.973, Mac-F1 = 0.977, and Wgt-F1 = 0.973.The second-best model, gpt-5.4 System A, achieves Acc = 0.883 and Mac-F1 = 0.851.
  • Integrated pipeline: 7.08 minutes is the integrated pipeline’s average runtime, including 419.20 seconds of LLM inference and 5.40 seconds of setup; assignment takes 145.39 seconds.The pipeline uses claude-sonnet-4-6 for Tasks 1–3 and gpt-5.5 for Task 4 under System A.
  • Integrated pipeline: 99.95% is the reduction from the historical 10.86-day average triage time to 7.08 minutes, corresponding to approximately 2,208× speedup.The historical average is calculated from the dataset’s Triage Hours field for the same 37 labeled issues.

6 Discussion

TianoForge substantially reduces automated bug-triage latency while showing strongest reported performance on bug assignment. The discussion also indicates that LLM-based prioritization depends on prompt-engineered criteria that are necessary but not sufficient, and that manual verification can improve issue-tracking data quality.

  • RQ1: Triage latency: 7.08 minutes versus 260.57 hours (10.86 days) reduces average triage time by 99.95%, corresponding to approximately a 2,208× speedup.The integrated script processes all four triage tasks for 37 labeled GitHub-native issues.
  • RQ2: Task effectiveness: 0.973 accuracy and 0.977 macro-F1 make bug assignment the strongest-performing triage task for gpt-5.5 System A.The result combines leave-one-out in-context examples with the described system configuration.
  • RQ2: Task effectiveness: Manual verification of LLM-flagged reports found three previously unlabeled invalid issues and two duplicate pairs.These findings show that the framework can support issue-tracking data-quality improvement in addition to automated triage.
  • RQ3: Priority criteria: Iterative prompt engineering identifies priority heuristics needed for accurate LLM-based prioritization.The discussion frames these heuristics as necessary but not sufficient for priority assignment.
  • RQ3: Priority criteria: 0.620 suggests that available priority factors are necessary but not sufficient, because implicit criteria may include reporter identity, milestone urgency, and cross-package dependencies.Degradation under System B after retrieving Bugzilla EDK II examples indicates legacy Bugzilla priority conventions do not directly transfer.

7 Conclusion and Future Work

TianoForge provides end-to-end automated bug triage for the TianoCore EDK II ecosystem by integrating four triage tasks with LLMs, RAG, and domain-specific prompting. The evaluation also demonstrates practical dataset-quality benefits and reduced triage latency, while motivating broader repository testing, improved retrieval, and confidence-aware triage.

  • Contributions: TianoForge integrates invalid issue detection, duplicate detection, issue prioritization, and developer assignment into an automated EDK II triage script.The framework combines LLMs, RAG, and domain-specific prompting for end-to-end triage automation.
  • Findings and Future Work: Retrieval augmentation does not consistently improve performance and can introduce additional noise into triage decisions.This finding motivates investigation of more advanced retrieval strategies with better contextual relevance.
  • Findings and Future Work: The framework substantially reduces triage latency compared with the current manual process, potentially accelerating issue handling and improving developer productivity.The result supports applying automated triage to large-scale Open Source Software projects.
  • Practical Value: Three additional invalid issues and two additional duplicate issue pairs were identified through manual review of model-flagged reports.These findings were not originally labeled in the GitHub issue tracker, demonstrating practical value beyond automation.
  • Future Work: Future work will evaluate additional OSS repositories, investigate advanced retrieval strategies, and calibrate LLM reflective confidence scores for confidence-aware triage.The intended goal is to distinguish reliable predictions from cases requiring human intervention.
Loading 2608.23259v1…