Source-linked AI summary
DeepTriage: Exploring the Effectiveness of Deep Learning for Bug Triaging
Senthil Mani, Anush Sankaran, Rahul Aralikatte
TL;DR
Automated bug triaging must assign noisy, sequential bug-report text to appropriate developers, while bag-of-words features omit syntactic and sequential information. The paper proposes an attention-based DBRNN-A representation learned from long text sequences and compares it with BOW-based classifiers, finding higher rank-10 average accuracy across three datasets.
Problem
Bug triaging assigns noisy bug titles and descriptions to developer classes, but bag-of-words features do not capture syntactic and sequential information.
Method
The paper learns unsupervised bug-report representations with an attention-based deep bidirectional recurrent neural network and uses them for classifier training.
Results
DBRNN-A with a softmax classifier outperforms bag-of-words models in rank-10 average accuracy across Chromium, Mozilla Core, and Mozilla Firefox.
Takeaways & Limitations
The released datasets, benchmarking protocol, and source code make the study reproducible, while the results support using deep learned representations for this triaging task.
Takeaways & Limitations
The implemented comparison algorithms may differ slightly from their original implementations, and the classifier assumes a particular training and testing setup.
Abstract
from arXiv · showhide
For a given software bug report, identifying an appropriate developer who could potentially fix the bug is the primary task of a bug triaging process. A bug title (summary) and a detailed description is present in most of the bug tracking systems. Automatic bug triaging algorithm can be formulated as a classification problem, with the bug title and description as the input, mapping it to one of the available developers (classes). The major challenge is that the bug description usually contains a combination of free unstructured text, code snippets, and stack trace making the input data noisy. The existing bag-of-words (BOW) feature models do not consider the syntactical and sequential word information available in the unstructured text. We propose a novel bug report representation algorithm using an attention based deep bidirectional recurrent neural network (DBRNN-A) model that learns a syntactic and semantic feature from long word sequences in an unsupervised manner. Instead of BOW features, the DBRNN-A based bug representation is then used for training the classifier. Using an attention mechanism enables the model to learn the context representation over a long word sequence, as in a bug report. To provide a large amount of data to learn the feature learning model, the unfixed bug reports (~70% bugs in an open source bug tracking system) are leveraged, which were completely ignored in the previous studies. Another contribution is to make this research reproducible by making the source code available and creating a public benchmark dataset of bug reports from three open source bug tracking system: Google Chromium (383,104 bug reports), Mozilla Core (314,388 bug reports), and Mozilla Firefox (162,307 bug reports). Experimentally we compare our approach with BOW model and machine learning approaches and observe that DBRNN-A provides a higher rank-10 average accuracy.
1 INTRODUCTION
Automated bug triaging assigns bug reports to developers, but report content is noisy and difficult for classifiers to process. The paper frames this assignment as developer classification using bug summaries and descriptions.
- Bug reports typically contain a brief summary and detailed description, including steps to reproduce.
- Automated bug triaging maps a bug title and description to a developer who could potentially solve the bug.Manual triaging is laborious in large-scale systems with many incoming bugs.
- Bug reports combine unstructured text with code snippets and stack traces, creating a major challenge for classifier learning.
2 RESEARCH CONTRIBUTIONS
The paper proposes DBRNN-A to learn ordered, semantic representations of long bug-report text while leveraging previously ignored untriaged reports. It studies classifier performance, training-data effects, title-only input, and transfer learning, while releasing data and code for reproducibility.
- Research Questions: The study evaluates whether unsupervised feature engineering, training samples per class, title-only input, and cross-dataset transfer learning affect automated triaging.
- Research Contributions: Figure 3 illustrates how bag-of-words similarity can favor the wrong bug report even when another report matches the ground-truth developer.
- Research Contributions: DBRNN-A uses an attention-based deep bidirectional recurrent network with LSTM units to preserve long-range word context in bug-report representations.The representation is learned from data without manual feature engineering.
- Research Contributions: About 70% of open-source repository bugs are untriaged or unsolved, and the paper uses them to learn bug representations in an unsupervised manner.
- Research Contributions: The paper releases bug reports from Chromium, Mozilla Core, and Mozilla Firefox together with exact train-test splits and source code to support reproducibility.
3 PROPOSED APPROACH
The proposed pipeline learns dense bug-report representations with DBRNN-A, then uses them in a supervised classifier for developer assignment. It combines sequential, bidirectional, attention-based modeling with a compact alternative to BOW features.
- Pipeline: DBRNN-A learns bug-report features from combined titles and descriptions, and a supervised classifier uses those features to assign potential developers.The pipeline trains representations with the deep recurrent model before predicting developer probabilities with the classifier.
- Pipeline: The preprocessing removes URLs, stack traces, hexadecimal code, and code snippets before vocabulary construction.
- DBRNN-A: Attention weights selectively emphasize important words, while bidirectional recurrence combines preceding and following context for long bug-report sequences.
- Representation: DBRNN-A produces a dense compact representation smaller than the vocabulary-sized BOW representation.For 10,000 bugs and 250,000 unique words, BOW yields a 10,000 × 250,000 matrix, whereas DBRNN-A yields 10,000 × 1,200.
- Contribution: The authors present the deep sequence model as a first application of learned bug representations to supervised automated software bug triaging.
- Classification: The classifier maps bug features to developer classes using a softmax model that outputs class scores normalized as probabilities.
4 LARGE SCALE PUBLIC BUG TRIAGE DATASET
The study assembles a reproducible large-scale benchmark from Chromium, Mozilla Core, and Mozilla Firefox. Triaged bugs support classifier training and testing, while untriaged bugs provide additional data for representation learning.
- Dataset: The benchmark combines bug reports from Chromium, Mozilla Core, and Mozilla Firefox, with the dataset, train-test protocol, and source code made publicly available.
- Dataset: Chromium contributes 383,104 bug reports, while Mozilla Core and Mozilla Firefox contribute 314,388 and 162,307 reports, respectively.
- Data split: Classifier data use fixed or verified reports with valid developer owners, while other reports are used to learn the deep representation.
- Data split: For Chromium, 263,936 bug reports, or 68.9%, train the deep model, and 118,643, or 31%, support classifier training and testing.
- Evaluation protocol: Chronological splitting preserves substantial developer overlap between training and testing, and a minimum number of examples per developer supports classifier training.
5 EXPERIMENTAL EVALUATION
The evaluation compares DBRNN-A bug representations with BOW-based classifiers across three datasets and examines training-data size and bug-report text inputs. DBRNN-A achieves higher rank-10 accuracy, while more training samples and including descriptions improve performance.
- 5 EXPERIMENTAL EVALUATION: The evaluation reports rank-k accuracy because the softmax classifier assigns each developer a probability and the true developer may appear among the top-k results.Cross-validation sets use different developer classes, so per-set top-k accuracy is also needed to expose performance variance.
- 5.3 Result Analysis: Using only the bug title reduces deep-learning triaging performance significantly compared with using the title together with the description.Figure 8 reports this comparison across all three datasets.
- 5.3 Result Analysis: DBRNN-A achieves rank-10 average accuracy of 37–43% on Chromium, improving over BOW-based approaches by 12–15%.BOW + Softmax reaches about 9–12%, while the best BOW classifier reaches 26–28%; DBRNN-A also improves accuracy by 3–5% on Mozilla Core and 7–17% on Mozilla Firefox.
- 5.3 Result Analysis: Increasing the minimum training samples per class improves classification across all three repositories, with Chromium accuracy rising from 37.9% at threshold zero to 42.7% at threshold 20.Thresholds above 20 did not improve accuracy, and Figure 7 summarizes the trend across datasets.
6 THREATS TO VALIDITY
The study identifies threats related to repository scope, input features, baseline reimplementation, and the assumption that each bug has one rightful developer owner.
- The results use three open-source repositories with different characteristics, but may not extend directly to commercial bug-tracking systems.Commercial systems may follow different patterns.
- The approach considers only bug-report titles and descriptions, although additional information may be needed to sufficiently triage some bugs.The experiments show these two unstructured text fields are necessary, but do not establish that they are sufficient.
- Some comparison algorithms were reimplemented because their original implementations were unavailable, allowing minor deviations from the originals.
- The evaluation assumes each bug has one rightful owner, although multiple active developers might potentially address a bug.
7 OTHER APPLICATIONS
The learned bug representation is task-independent, allowing it to support supervised or semi-supervised applications beyond developer assignment.
- The unsupervised, task-independent representation can be reused to train classifiers for other tasks or applications.
- The proposed features could support classifying fixed versus non-fixed bugs, predicting bug-fix time, and analyzing reopened bugs.
8 RELATED WORK
Prior bug-triaging work mainly uses titles and descriptions with varied engineered feature models, while this study learns representations directly from data with a deep bidirectional RNN.
- Most prior approaches use bug summaries or descriptions because these fields are available at submission and remain unchanged during a ticket’s lifecycle.
- Related studies also incorporate developer activity, component information, code information, or external expertise signals to shortlist developers.
- Prior work employs tf, normalized tf, tf-idf, and n-gram features, making feature-model selection an engineering design choice.
- DBRNN-A addresses feature-selection challenges by learning a representation directly from data in an unsupervised fashion.
9 CONCLUSION
The paper proposes DBRNN-A for bug-report representation and evaluates it across three large open-source repositories. DBRNN-A with a softmax classifier improves rank-10 average accuracy over BOW in all three datasets and supports transfer across Mozilla and Chromium data.
- The study collects bug reports from Google Chromium, Mozilla Core, and Mozilla Firefox for experimental analysis.
- DBRNN-A represents bug-report titles and descriptions while preserving word ordering over longer contexts and semantic relationships.
- DBRNN-A with a softmax classifier improves rank-10 average accuracy over BOW across all three evaluated datasets.The datasets are Google Chromium, Mozilla Core, and Mozilla Firefox.
- A model learned on Chromium competitively triaged bugs in the Mozilla dataset, demonstrating experimentally supported transfer-learning ability.
- The dataset, benchmarking protocol, and implemented source code are publicly available to improve reproducibility.