Source-linked AI summary
Improving IR-Based Bug Localization with Context-Aware Query Reformulation
Mohammad Masudur Rahman, Chanchal K. Roy
TL;DR
IR-based bug localization may falter when reports lack relevant program entities or contain excessive structured information. BLIZZARD classifies report quality and reformulates queries before retrieval. Across the reported evaluations, it outperforms the baseline and improves on the state-of-the-art in key metrics.
Problem
IR-based localization may perform poorly when bug reports lack relevant program entities or contain excessive structured information such as stack traces.
Source: §1 · p. 1Method
BLIZZARD classifies bug reports as noisy, rich, or poor, then refines or complements their queries before information retrieval.
Source: §1 · p. 1Results
Against the baseline, BLIZZARD achieves 7%–56% higher Hit@10, 6%–62% higher MAP@10, and 6%–62% higher MRR@10, and improves MAP@10 by 19% and MRR@10 by 20% over the state-of-the-art.
Source: §1 · p. 2Takeaways & Limitations
The reported results show that quality-aware query reformulation can improve IR-based bug localization across noisy and poor reports.
Source: §1 · pp. 1–2The report-classification regular expressions are limited for some contexts, such as ill-structured stack traces, and currently require limited manual analysis.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Recent findings suggest that Information Retrieval (IR)-based bug localization techniques do not perform well if the bug report lacks rich structured information (eg relevant program entity names). Conversely, excessive structured information (eg stack traces) in the bug report might not always help the automated localization either. In this paper, we propose a novel technique--BLIZZARD-- that automatically localizes buggy entities from project source using appropriate query reformulation and effective information retrieval. In particular, our technique determines whether there are excessive program entities or not in a bug report (query), and then applies appropriate reformulations to the query for bug localization. Experiments using 5,139 bug reports show that our technique can localize the buggy source documents with 7%--56% higher Hit@10, 6%--62% higher MAP@10 and 6%--62% higher MRR@10 than the baseline technique. Comparison with the state-of-the-art techniques and their variants report that our technique can improve 19% in MAP@10 and 20% in MRR@10 over the state-of-the-art, and can improve 59% of the noisy queries and 39% of the poor queries.
1 INTRODUCTION
IR-based bug localization is challenged by reports with either too little or too much structured information. BLIZZARD classifies report quality, reformulates queries accordingly, and improves localization across the reported evaluations.
1 INTRODUCTION
- Software debugging remains largely manual and consumes significant development time and effort.Bug localization is identified as one of debugging's three steps.Source: §1 · p. 1
- IR-based localization may struggle with reports lacking relevant program entities or containing excessive structured information such as stack traces.The paper notes that many existing techniques use nearly verbatim report text as the query.Source: §1 · p. 1
- BLIZZARD classifies reports as noisy, rich, or poor, then refines noisy queries or complements poor queries before IR-based localization.Unlike earlier approaches, it applies query reformulation based on report quality.Source: §1 · p. 1
- For the noisy and poor examples, BLIZZARD returns the first correct result, while the baseline ranks them 53rd and 30th, respectively.BugLocator ranks those correct results 19th and 26th, respectively.Source: §1 · pp. 1–2
- 7%–56% higher Hit@10, 6%–62% higher MAP@10, and 6%–62% higher MRR@10 than the baseline; BLIZZARD also improves MAP@10 by 19% and MRR@10 by 20% over the state-of-the-art.The evaluation uses 5,139 bug reports from six Java-based systems.Source: §1 · p. 2
- The paper reports a comprehensive evaluation against seven techniques, alongside a working prototype and experimental data for replication and reuse.The broader evaluation includes comparisons with state-of-the-art techniques and query-reformulation approaches.Source: §1 · p. 2
2 GRAPH-BASED TERM WEIGHTING
The section introduces term weighting as a way to estimate the relative importance of terms. It describes graph-based weighting for identifying important terms in bug reports that may mix structured elements and regular text.
2 GRAPH-BASED TERM WEIGHTING
- Term weighting estimates the relative importance of a term within a body of text.TF-IDF is described as a widely used proxy for term importance, while TextRank was proposed as another proxy that considers relationships among terms.Source: §2 · p. 2
- TextRank encodes a document as a graph of unique-word nodes connected by meaningful relations, including statistical, syntactic, or semantic links.The paper identifies important terms using graph-based term weighting.Source: §2 · p. 2
- The paper applies graph-based term weighting to bug reports that may contain both structured elements, such as stack traces, and regular text.Source: §2 · p. 2
3 BLIZZARD: PROPOSED TECHNIQUE
BLIZZARD classifies bug reports by their structured content, reformulates queries accordingly, and uses information retrieval to rank potentially buggy source files. In working examples, its reformulated queries place the first correct result first for noisy, rich, and poor reports.
3 BLIZZARD: PROPOSED TECHNIQUE
- BLIZZARD classifies reports by structured content, reformulates each query accordingly, then submits it to an information-retrieval system for source-code localization.The pipeline uses report quality, query reformulation, and information retrieval to localize bugs from reports of any quality.Source: §3 · p. 2, §3.1 · p. 2, §3.3 · p. 5
3.1 Bug Report Classification
- Reports with stack traces are noisy; those with program elements but no traces are rich; reports with neither are poor natural-language queries.Classification uses regular expressions, with manual analysis when ill-defined report structures prevent automatic classification.Source: §3.1 · pp. 2–3
3.2 Query Reformulation
- For noisy reports, BLIZZARD builds a trace graph and uses PageRank to select keywords; for rich reports, it builds text graphs from word co-occurrences and syntactic dependencies.The text-graph approach treats words as nodes, encoding nearby co-occurrences and POS-based syntactic dependencies as edges.Source: §3.2 · pp. 3–4
- For poor reports, BLIZZARD retrieves top-K source documents, extracts and splits method and field signatures, then uses their term graph to supplement the query.The source-term graph captures term co-occurrences within a fixed window of two words from phrases derived from signatures.Source: §3.2 · p. 4
- BLIZZARD ranks graph terms by PageRank, selects the top 8–30, and constructs different queries: replace trace entries, combine poor-report text with source terms, or use rich-report terms.The reformulated query is submitted to Lucene using Okapi BM25, and results are returned as a ranked list for developer analysis.Source: §3.2 · pp. 4–5, §3.3 · p. 5
3.3 Bug Localization
- All three showcased reformulated queries return their first correct result at rank one, versus baseline ranks 53 for noisy, 27 for rich, and 30 for poor reports.BLIZZARD refines noisy queries, selects important keywords from rich reports, and adds complementary source-code terms to poor reports.Source: §3.3 · p. 5
4 EXPERIMENT
The experiments evaluate BLIZZARD against baseline and existing localization and query-reformulation techniques across report-quality groups, using 5,139 bug reports from six systems. They also examine evaluation metrics, parameter settings, and the effect of reformulating queries.
4.1 Experimental Dataset
- The dataset comprises 5,139 resolved bug reports from six open-source systems, with gold sets formed from files changed by bug-fixing commits.Reports without changed source files or relevant files in the collected snapshot were excluded.Source: §4.1 · p. 5, Table 4 · p. 6
4.2 Performance Metrics
- Evaluation uses Hit@K (a buggy file in the top K), MAP@K (rank-sensitive precision across buggy files), MRR@K (the first correct result’s reciprocal rank), and Effectiveness (the first buggy file’s rank).The authors use these measures to compare localization performance and query effectiveness.Source: §4.2 · p. 6
4.3 Experimental Results
- BLIZZARD localizes 74.52% of 5,139 bugs, with 47% MAP@10 and 0.50 MRR@10—12%, 13%, and 14% above baseline, respectively.For noisy reports, BLIZZARD achieves 75% Hit@10 and 46% precision, compared with 48% and 28% for baseline.Source: Table 5 · p. 6, §4.3 · p. 6
- The baseline uses preprocessed title-and-description text, split complex tokens, and Lucene; selected reformulation lengths are 11 for noisy, 30 for entity-rich, and 8 for poor reports.Stemming reduced BLIZZARD’s localization accuracy, and Lucene performed better than Indri on this dataset.Source: §4.3 · pp. 6–7
- 59% of noisy and 39%–56% of poor queries improve (approximately 25% worsen); noisy-query ranks improve by up to 122 positions, and poor-query results arrive 16 positions earlier.The poor-query improvement is statistically significant (p-value=0.004, Cliff’s ∆=0.94).Source: Table 6 · p. 7, §4.3 · p. 7
4.4 Comparison with Existing Techniques
- BLIZZARD+ outperforms AmaLgam+, while BLIZZARDBRO beats AmaLgam+BRO across report groups; BLIZZARD delivers approximately 75%–80% Hit@10 irrespective of report quality.BLIZZARDBRO has 20% higher MRR@10 than AmaLgam+BRO using the same bug-report-only resources.Source: Table 7 · p. 8, §4.4 · p. 8
- BLIZZARD improves 22% more noisy queries and 20% more poor queries than the state-of-the-art query-reformulation techniques.For noisy queries improved by BLIZZARD, 75% return the first correct result within the top nine positions.Source: Table 9 · p. 9, §4.4 · p. 9
- Across six systems, BLIZZARD+ and BLIZZARDBRO outperform their respective state-of-the-art comparators; standalone BLIZZARD wins Hit@10 on four systems and trails on two.BLIZZARD+ has higher MRR@10 and MAP@10 than existing techniques across systems.Source: Figure 7 · p. 9, Figure 8 · p. 9, §4.4 · p. 9
5 THREATS TO VALIDITY
The experiments face possible implementation errors and report-classification errors, while generalizability is bounded by the Java-only evaluation. The authors note that adaptation to other OOP systems is possible when reports contain structured items.
5 THREATS TO VALIDITY
- Regular-expression classification is limited for ill-structured stack traces, so these cases currently require limited manual analysis.The authors suggest more sophisticated classification approaches as future work.Source: §5 · p. 10
- Implementation errors cannot be ruled out; the team re-implemented four techniques using original authors’ input and reported settings and parameters.They used the authors’ implementations for three other techniques.Source: §5 · p. 10
- The evaluation uses Java systems, although the technique may be adapted to other OOP systems with structured bug-report items.Examples of such items include stack traces and program entities.Source: §5 · p. 10
6 RELATED WORK
IR-based bug localization uses shared vocabulary between bug reports and source code, but performance is sensitive to report quality. Prior approaches combine IR with other analyses or reformulation, while BLIZZARD instead adapts reformulation to report quality using the available report and source code.
6 RELATED WORK
- IR-based localization uses shared vocabulary between bug reports and source code, but its performance is inherently subject to bug-report quality.Prior work also combines IR with spectra analysis, machine learning, or repository mining.Source: §6 · p. 10
- Combining IR with multiple analyses or external repositories can improve localization, but repository mining may be costly and external-source dependencies less scalable.Wang and Lo combine bug reports with three external repositories and outperform five earlier IR-based techniques.Source: §6 · p. 10
- Prior stack-trace methods boost document rankings without discarding superfluous trace entries, while learning-to-rank and deep-learning methods may use classic IR without query reformulation.Query reformulation can complement distributional-semantic approaches when noisy terms are present.Source: §6 · p. 10
- Traditional query-reformulation methods mostly target unstructured text, while bug-report approaches have not addressed both structured and unstructured contents [42].Sisman and Kak introduced reformulation for IR-based bug localization but cannot remove query noise.Source: §6 · p. 10
- BLIZZARD combines report-quality-aware reformulation with IR, exploiting available resources and delivering equal or higher state-of-the-art performance at relatively lower cost.The authors describe this as a comprehensive solution not previously provided by existing studies.Source: §6 · p. 10
7 CONCLUSION AND FUTURE WORK
BLIZZARD addresses the sensitivity of IR-based bug localization to bug-report quality through quality-aware query reformulation. Across 5,139 reports from six open-source systems, it reports higher precision than the best baseline and the state of the art, and the authors plan to apply the approach to automatic bug fixing.
7 CONCLUSION AND FUTURE WORK
- Up to 62% and 20% higher precision than the best baseline technique and the state-of-the-art, respectively, was reported across 5,139 reports from six open-source systems.BLIZZARD also improved 22% more noisy queries and 20% more poor queries than the state of the art.Source: §7 · p. 10
- BLIZZARD incorporates bug-report quality into IR-based localization through context-aware query reformulation to address the method’s sensitivity to report quality.The authors characterize traditional IR-based localization as inherently subject to the quality of submitted reports.Source: §7 · p. 10
- The authors plan to apply their learned insights and technique to more complex debugging activities, including automatic bug fixing.Source: §7 · p. 10