Source-linked AI summary
Automatic Bug Triage using Semi-Supervised Text Classification
Jifeng Xuan, He Jiang, Zhilei Ren, Jun Yan, Zhongxuan Luo
TL;DR
Bug triage lacks sufficient reliable labeled reports, limiting supervised approaches. The paper combines naive Bayes with expectation-maximization and a weighted recommendation list to use labeled and unlabeled reports. On Eclipse bug reports, the approach improves classification accuracy by up to 6%, although the authors report that results remain insufficient for real-world applications.
Problem
Supervised bug triage requires labeled reports, but labels are insufficient and nearly half may be mistaken after initial assignment.
Method
The approach combines a naive Bayes classifier with expectation-maximization and a weighted recommendation list using labeled and unlabeled bug reports.
Results
The semi-supervised approach improves Eclipse bug-triage classification accuracy by up to 6% over supervised naive Bayes.
Takeaways & Limitations
Weighted probabilistic recommendations for multiple developers augment the semi-supervised approach's use of unlabeled reports.
Takeaways & Limitations
The authors state that results remain insufficient for real-world applications because report quality, labeling reliability, and EM's mixture-model assumption can be problematic.
Abstract
from arXiv · showhide
In this paper, we propose a semi-supervised text classification approach for bug triage to avoid the deficiency of labeled bug reports in existing supervised approaches. This new approach combines naive Bayes classifier and expectation-maximization to take advantage of both labeled and unlabeled bug reports. This approach trains a classifier with a fraction of labeled bug reports. Then the approach iteratively labels numerous unlabeled bug reports and trains a new classifier with labels of all the bug reports. We also employ a weighted recommendation list to boost the performance by imposing the weights of multiple developers in training the classifier. Experimental results on bug reports of Eclipse show that our new approach outperforms existing supervised approaches in terms of classification accuracy.
I. INTRODUCTION
Bug triage assigns reports to developers, but manual assignment is costly and labels are often insufficient or incorrect. The paper proposes a semi-supervised classifier with a weighted recommendation list to improve automated triage accuracy.
- Manual bug triage is labor-intensive; in Eclipse, assigning 37 daily reports takes about 3 person-hours.
- Supervised triage approaches require labeled reports, yet labels are scarce and nearly half may be mistaken after initial assignment.
- The paper combines naive Bayes and expectation-maximization to iteratively use labeled and unlabeled bug reports.
- On Eclipse data, the semi-supervised approach improves classification accuracy by up to 6% over supervised naive Bayes, whose original accuracy ranges from 11% to 43%.
- A weighted recommendation list assigns probabilistic weights to multiple developers when incorporating unlabeled reports.
II. RELATED WORK
Prior bug-triage research used supervised or unsupervised learning, while this paper identifies semi-supervised learning as a way to address limited labeled bug reports.
- Existing bug-triage approaches were based on supervised or unsupervised learning, with no semi-supervised approach identified in the literature.
- Duplicate bug reports were addressed through both supervised natural-language processing and unsupervised clustering approaches.
- Earlier work applied unsupervised clustering to group bug reports with similar errors and analyze program-failure causes from stack traces.
- Semi-supervised learning uses unlabeled bug reports to assist an existing supervised classifier, addressing the deficiency of labeled reports.
A. Semi-supervised framework of bug triage
The proposed framework combines a supervised classifier with expectation-maximization to exploit labeled and unlabeled bug reports, while weighted recommendations represent multiple relevant developers.
- The framework first trains a classifier on labeled reports, then uses expectation-maximization with labeled and unlabeled reports.EM alternates between evaluating unlabeled reports and rebuilding the classifier.
- In the E-step, the classifier evaluates and labels reports in the unlabeled subset; in the M-step, it rebuilds a classifier using labels from all reports.
- The training framework takes labeled and unlabeled report subsets as input and produces a semi-supervised bug-triage classifier while iterating as the classifier improves.
- Naive Bayes serves as the basic classifier because it suits text classification, supports probability-ranked recommendations, and integrates readily with probability-weighted EM.
- A weighted recommendation list incorporates weights for multiple relevant developers during classifier training and supplies probabilistic labels for unlabeled reports.
B. NB classifier
The naive Bayes classifier assigns an incoming bug report to the developer with the highest posterior probability, estimated from word and developer statistics.
- Naive Bayes predicts the relevant developer by choosing the developer with maximum posterior probability for a bug report.
- The classifier models bug-report text using a word vocabulary and assumes words are independently and identically distributed when estimating likelihoods.
- For a report, the likelihood is computed as a product of word probabilities conditioned on the candidate developer.
- Training estimates the model probabilities from the training set, while prediction calculates posterior probabilities for each candidate developer.
- Laplace smoothing is applied in practice to avoid zero probabilities in the estimated model.
C. EM in semi-supervised bug triage
The semi-supervised triage classifier adapts naive Bayes with expectation-maximization to incorporate unlabeled bug reports under a mixture-model assumption.
- EM assumes that data arise from a mixture model whose components correspond to classes.
- The E-step estimates label probabilities, while the M-step modifies classifier parameters using labeled and unlabeled subsets.
- The weight factor λ ranges from 0 to 1, interpolating between naive Bayes, basic EM, and weighted EM.
- Cross-validation estimates λ, and EM stops when classifier parameters no longer improve.
D. Weighted recommendation list
The weighted recommendation list extends EM by assigning training weights to multiple probable developers in unlabeled reports rather than only the most probable developer.
- The weighted recommendation list guides EM’s M-step by adding weights for multiple developers in unlabeled bug reports.
- The E-step selects the top-n posterior-probability developers for each unlabeled report and distributes their probabilities across the list.
- For labeled reports, the recommendation-list size is one and the original relevant developer remains the label.
- For unlabeled reports, higher posterior probability gives a developer a larger training weight during each M-step.
- The algorithm initializes λ by cross-validation, builds naive Bayes from labeled reports, then iterates E- and M-steps until improvement stops.
A. Data preparation
The experiments use resolved Eclipse Bugzilla reports, heuristically assigned developer labels, and word-vector representations derived from bug descriptions.
- The evaluation uses Eclipse Bugzilla reports with identifiers 150001–170000, retaining 10,747 reports after filtering.
- Developer labels follow prior heuristics rather than relying directly on the assigned-to field, whose value may differ from the actual solver.
- Reports without resolved status, verified status, fixed resolution, or duplicate resolution are removed before analysis.
- Each report is represented by word counts from its short description and first long description after tokenization.
- Preprocessing removes stopwords, non-alphabetic tokens, low-frequency words, and low-frequency developers, without stemming.
B. Experimental results
On Eclipse data, semi-supervised NB with EM improves classification accuracy over NB, while the weighted recommendation list generally adds further gains but can hurt very small lists.
- The evaluation partitions each developer’s reports into 5% labeled training, 20% testing, and 75% unlabeled training data.
- Accuracy is measured by whether the relevant developer appears in a recommendation list of size n.
- Accuracy improves 2%–5% for list size 3 and 3%–6% for list size 5 over NB, while NBEM+WRL adds 1%–3% over NBEM.
- WRL can reduce performance for a small recommendation list because it covers ignored developers while over-weighting relevant developers.
- Varying λ changes accuracy by up to 3% for a fixed list size; NBEM favors values near 0.1, whereas NBEM+WRL favors values near 1.0.
V. DISCUSSION
The discussion identifies data, labeling, and modeling constraints that limit semi-supervised bug triage, while describing differences that make generic text-classification methods unsuitable. It also notes labeling heuristics and potential extensions for improving practical accuracy.
- Classification accuracy improves by up to 6%, but the results remain insufficient for real-world bug-triage applications.The reported NB accuracy ranges from 11% to 43%.
- Three factors limit performance: poor bug-report quality, difficult developer labeling, and an often-unsatisfied EM mixture-model assumption.
- Bug triage datasets are smaller and use more specialized vocabulary than typical text-classification datasets, limiting direct transfer of generic methods.
- Heuristic labeling uses bug-lifecycle knowledge but is difficult to implement in traditional bug repositories.
VI. CONCLUSION AND FUTURE WORK
The paper concludes that NB with EM and a weighted recommendation list improves bug-triage classification using labeled and unlabeled reports. Future work targets model extensions, co-training, and integration with bug repositories.
- The proposed semi-supervised NB-with-EM approach improves bug-triage classification accuracy by up to 6% using labeled and unlabeled reports.A weighted recommendation list adds multiple-developer weights during classifier training.
- Future work will modify EM through many-to-one mixture-component correspondence and combine it with the weighted recommendation list.
- The authors plan to evaluate co-training for bug triage because it can use labeled and unlabeled data with two feature views.
- A planned bug-repository plug-in would evaluate automatic triage in real-world applications and collect additional research information.