Source-linked AI summary
On the Performance of Hybrid Search Strategies for Systematic Literature Reviews in Software Engineering
Erica Mourão, João Felipe Pimentel, Leonardo Murta, Marcos Kalinowski, Emilia Mendes, Claes Wohlin
TL;DR
SLR researchers need search strategies that balance evidence quality with review effort, while database searches or snowballing alone have limitations. This paper proposes four hybrid strategies combining Scopus database searches with variations of backward and forward snowballing, and evaluates them over three existing SLRs using precision, recall, and F-measure. For the analyzed SLRs, parallel or sequential snowballing with Scopus provided an appropriate balance of precision and recall, although the findings have limited generalizability and do not measure relative exclusion effort.
Problem
Evidence on how SLR search strategies balance evidence quality and review effort is scarce, despite limitations in database searching and snowballing.
Method
The paper proposes four hybrid strategies combining Scopus database searches with iterative, parallel, or sequential backward and forward snowballing, evaluated over three existing SLRs.
Results
Scopus combined with parallel or sequential snowballing was reported as an efficient hybrid alternative, with a single backward-and-forward snowballing iteration providing 90% to 100% recall.
Takeaways & Limitations
Depending on SLR goals and available resources, a representative digital library combined with parallel or sequential snowballing may be an appropriate evidence-search alternative.
Takeaways & Limitations
The evaluation used only three SLRs from software processes, so results may not generalize across software-engineering fields, and relative exclusion effort was not considered.
Abstract
from arXiv · showhide
Context: When conducting a Systematic Literature Review (SLR), researchers usually face the challenge of designing a search strategy that appropriately balances result quality and review effort. Using digital library (or database) searches or snowballing alone may not be enough to achieve high-quality results. On the other hand, using both digital library searches and snowballing together may increase the overall review effort. Objective: The goal of this research is to propose and evaluate hybrid search strategies that selectively combine database searches with snowballing. Method: We propose four hybrid search strategies combining database searches in digital libraries with iterative, parallel, or sequential backward and forward snowballing. We simulated the strategies over three existing SLRs in SE that adopted both database searches and snowballing. We compared the outcome of digital library searches, snowballing, and hybrid strategies using precision, recall, and F-measure to investigate the performance of each strategy. Results: Our results show that, for the analyzed SLRs, combining database searches from the Scopus digital library with parallel or sequential snowballing achieved the most appropriate balance of precision and recall. Conclusion: We put forward that, depending on the goals of the SLR and the available resources, using a hybrid search strategy involving a representative digital library and parallel or sequential snowballing tends to represent an appropriate alternative to be used when searching for evidence in SLRs.
1. Introduction
The paper examines how database searches and snowballing can be combined to balance evidence quality and review effort in software-engineering SLRs. It proposes and evaluates four hybrid strategies using simulations over three existing SLRs.
- Database searches are common but face challenges involving library selection, search-string design, interface differences, operators, synonyms, overlap, and execution inconsistencies.
- Snowballing offers an alternative or complement to database searching, but depends on an appropriate seed set and can introduce selection difficulties and author-related bias.
- Combining database searches across several libraries with iterative backward and forward snowballing can improve coverage but increases review effort and retains drawbacks of both approaches.
- Because evidence on balancing search quality and effort remains scarce, the paper evaluates database, snowballing, and hybrid strategies using precision, recall, and F-measure.
- The study proposes four hybrid variations and reports that hybrid strategies may be an appropriate alternative to database search or snowballing alone.
2. Background
The background positions hybrid searching as a response to weaknesses in database-only and snowballing-only approaches. It extends earlier work by defining and evaluating multiple combinations of Scopus searching with backward and forward snowballing.
- SLR guidance recommends database searches across digital libraries alongside complementary searches to support complete evidence identification.
- Database searches can miss relevant studies because of search-string, keyword, interface, search-engine, and digital-library limitations.
- Snowballing provides an alternative search strategy but requires an appropriate seed set of papers.
- Prior SLRs combined searches across several digital libraries with iterative backward and forward snowballing to reduce the risk of missing relevant evidence, at additional effort.
- Earlier hybrid work combined Scopus with parallel backward and forward snowballing and found similar results to searches across several digital libraries, but lacked broader baseline comparisons.
- This paper evaluates hybrid strategies against database search, snowballing, and exhaustive combined database-and-snowballing searches using three existing SLRs.
3. Hybrid Search Strategies
The paper defines baseline and hybrid search strategies that combine database searching with different backward- and forward-snowballing arrangements. The hybrid strategies use Scopus to create a seed set before applying snowballing.
- DB Search queries multiple digital libraries, while SB Search starts from a Google Scholar seed set and applies iterative backward and forward snowballing.
- DB Search + BS*FS combines searches across all digital libraries with full iterative backward and forward snowballing.
- Scopus + BS*FS uses a Scopus seed set followed by iterative backward and forward snowballing.
- Scopus + BS||FS runs backward and forward snowballing in parallel over the same Scopus seed set, without cross-processing newly found papers.
- Scopus + BS+FS performs all backward-snowballing iterations before forward snowballing, whereas Scopus + FS+BS reverses that order.
- The study simulates each strategy in a supporting tool, using reference lists for backward snowballing and Google Scholar citations for forward snowballing.
4. Research Questions
The research questions assess database searches, snowballing, and hybrid strategies in published software-engineering SLRs. They compare precision, recall, and F-measure while examining complementarity, overlap, and review effort.
- RQ1: Database search: RQ1 evaluates database-search performance, including digital-library precision, recall, F-measure, indexing, complementarity, and overlap.
- RQ2: Snowballing: RQ2 evaluates snowballing performance, including how backward and forward iterations complement database searches.
- RQ2: Snowballing: RQ2 also investigates overlap between backward and forward snowballing by simulating each independently and examining the intersection of retrieved sets.
- RQ3: Hybrid strategies: RQ3 measures precision, recall, and F-measure for each hybrid strategy and contrasts them with database, snowballing, and exhaustive-combination baselines.
5. Search Strategy Evaluation
The evaluation selected three high-quality SLRs and simulated hybrid search strategies using extracted study-selection data. It compared database searches, snowballing, and hybrid approaches using precision, recall, and F-measure.
- SLR corpus selection: The evaluation screened seven candidate SLRs against quality criteria and selected three that combined database search with iterative backward and forward snowballing.The additional QA5 criterion required this specific combination for simulating the hybrid strategies.
- Selected SLRs: The three selected SLRs used different digital-library sets, with each searching at least seven libraries and sharing several libraries.The corpus covered SLRs on strategic alignment of software process improvement, Definition of Done criteria, and ontologies in software process assessment.
- Study selection: Across the three SLRs, database searches returned 2,803 papers, including 891 duplicates and 1,912 unique papers; 49 were selected from database searches and 36 from snowballing.The study used the selection step, before quality assessment, as the basis for evaluating precision and recall.
- Evaluation procedure: The evaluation extracted search data, simulated the strategies, and analyzed their performance using precision, recall, and F-measure.When source data were incomplete, the researchers contacted authors or reran digital-library queries; they also tracked duplicates and paper origins.
6. Results and Discussion
Across the analyzed SLRs, Scopus was the most consistent database option, while hybrid strategies combined database coverage with snowballing to balance precision, recall, and effort. Parallel or sequential snowballing with Scopus stood out, although exhaustive snowballing achieved higher recall at greater effort and lower precision.
- Digital library performance: Scopus consistently ranked highly for precision, recall, and F-measure across the three SLRs, clearly ahead when precision and recall were considered together.It had the highest F-measure for all three SLRs and the highest recall for SLR1 and SLR2.
- Digital library performance: 13% to 35% of relevant papers were found by Scopus alone, motivating supplementation with other digital libraries or snowballing.Scopus remained consistent, but its standalone recall was limited.
- Database search plus snowballing: One snowballing iteration raised recall from 43%–80% after database searching to 90.2%–100% of the published SLR papers.For SLR1, recall increased from 43.14% to 90.2%, while precision decreased from 4.73% to 3.72%.
- Backward and forward snowballing: Backward snowballing was prominent for recall, forward snowballing for precision, and the two approaches were complementary in SLR1.In SLR2 and SLR3, backward snowballing included all papers retrieved by forward snowballing.
- Hybrid strategies: Scopus + BS||FS and Scopus + BS+FS stood out on F-measure, while DB Search + BS*FS alone consistently achieved 100% recall but with low precision and substantial effort.Scopus + BS||FS consistently provided the highest precision, and the two Scopus hybrids were prominent overall.
7. Threats to Validity and Limitations
The study identifies validity threats involving measurement choices, reproducibility, corpus construction, exclusion effort, statistical testing, and limited generalizability. These constraints qualify how the reported search-strategy performance should be interpreted.
- Construct validity: The study used precision, recall, and F-measure to assess search-strategy performance, following measures commonly used in related research.
- Construct validity: Recall was computed against each SLR’s total retrieved-article set, although the authors could not guarantee that every possible article had been obtained.
- Construct validity: The analysis did not model the differing effort required to exclude irrelevant or duplicate papers, because the source SLRs lacked exclusion-process information.
- Internal validity: Reproducibility was constrained by subjective Google Scholar searching, mitigated by consistently examining the top 60 results.
- Internal validity: The researchers reconstructed missing duplicate information and analyzed all 51 SLR1-selected articles rather than only its final 30-article set.
- Conclusion validity: The authors did not use statistical tests because only three SLRs were analyzed, so the observed strategy performance is not conclusive.
- External validity: The sample comprised three SLRs in software processes, limiting generalizability to other software-engineering areas.
8. Related Work
Related work compares database searching, snowballing, manual searching, query design, and other approaches for identifying or updating SLR evidence. This study extends those comparisons by examining digital-library complementarity, snowballing direction and iteration, and hybrid strategies.
- Comparisons of search strategies: Prior studies generally found database searching and snowballing comparable in identifying relevant papers, while snowballing was sometimes reported as more efficient.
- SLR updates: For SLR updates, database search achieved higher recall, whereas forward snowballing achieved significantly higher precision and could reduce updating effort.
- SLR updates: Generic databases such as Scopus or Google Scholar were considered sufficient for forward-snowballing updates, whereas relying on one specific database was not recommended.
- Alternative search approaches: Other work found that broad automated searches can retrieve more studies than manual searches but with poorer quality, while query optimization is not straightforward.
- This study’s contribution: The present study extends prior contrasts by evaluating digital-library performance and complementarity, snowballing direction and iterations, and four hybrid strategies.
9. Conclusions and future work
The study evaluates hybrid database-and-snowballing strategies for SLR searches and finds that Scopus combined with selected snowballing schemes can balance coverage and efficiency. The authors qualify this conclusion because effort was not directly measured and the evidence comes from only three SLRs.
- Conclusions: Scopus alone found 13% to 35% of relevant papers, while Scopus plus ACM Digital Library found 23% to 60%.
- Conclusions: Scopus + BS||FS, Scopus + BS+FS, and Scopus + FS+BS were efficient hybrid strategies compared with plain database search and DB Search + BS*FS.
- Limitations: The authors did not measure time, so reduced paper counts do not directly establish reduced review effort.
- Future work: Future work should replicate the study across more SLRs and software-engineering areas, evaluate additional search strategies, and incorporate effort into comparisons.