Source-linked AI summary
Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studies
Claes Wohlin, Marcos Kalinowski, Katia Romero Felizardo, Emilia Mendes
TL;DR
The paper addresses whether combining database searching and snowballing can improve identification of primary studies in systematic literature reviews. It evaluates hybrid strategies through a replication of an industry–academia collaboration SLR and finds the full-fledged hybrid strategy superior, while introducing wild cards and borderline articles to accommodate reviewer differences.
Problem
Database searches have traditionally been complemented by snowballing, motivating evaluation of whether combining them can identify relevant primary studies more effectively and cost-efficiently.
Method
The study replicates an industry–academia collaboration SLR using a Scopus start set followed by backward and forward snowballing, while comparing four hybrid strategies.
Results
The full-fledged hybrid search strategy identified 30% more articles than the original SLR and performed even better when focusing on peer-reviewed articles.
Takeaways & Limitations
Hybrid searching is a strong contender for systematic literature studies, while alternative strategies may be viable when selected in relation to the snowballing start set.
Takeaways & Limitations
The evaluation may be affected by selection bias from the specific SLR and evaluator bias from researchers’ vested interest in hybrid searching.
Abstract
from arXiv · showhide
Background: A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which has later been complemented with snowball searches. Our conjecture is that we can perform even better searches if combining the two search approaches, referred to as a hybrid search strategy. Objective: Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared some alternative hybrid search strategies to assess whether it was possible to identify more cost-efficient ways of searching for relevant primary studies. Method: To compare and evaluate the hybrid search strategy, we replicated an SLR on industry-academia collaboration in software engineering. The SLR used a more traditional approach to searching for relevant articles for an SLR, while the replication was conducted using a hybrid search strategy. Results: In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30 percent more primary studies and even more when focusing only on peer-reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when conducting systematic literature studies. Conclusions: The hybrid search strategy is a strong contender for being used when conducting systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.
1. Introduction
Systematic literature reviews require search strategies that move toward finding all relevant primary studies. This study defines hybrid searching as integrating database search with snowballing, evaluates it through an industry–academia collaboration replication, and introduces concepts to accommodate reviewer differences.
- Hybrid search strategy: Hybrid searching integrates at least two pre-planned systematic approaches, such as database searching followed by snowballing from relevant articles.It excludes complementary venue searches and snowballing from only a sample of database-found articles.
- Study objective: The study compares an original SLR using a traditional search approach with a replication using a hybrid search strategy.The replication concerns industry–academia collaboration in software engineering and also compares alternative hybrid strategies.
- Novel concepts: Wild cards and borderline articles were introduced to accommodate different reviewer judgements and reduce the risk of missing relevant primary studies.Wild cards allow full-text assessment of articles that fail the usual threshold, while borderline articles remain separately tracked for later judgement.
- Study objective: The replication identified approximately 30% more relevant articles than the original SLR.The comparison evaluates differences and similarities between the original review and replication.
- Hybrid search strategy: The hybrid strategy uses a Scopus search to create a start set, followed by backward and forward snowballing from included articles.Three alternative snowballing methods were also considered to assess potentially more cost-efficient approaches.
2. Related work
Earlier software-engineering SLR research compared database searching, snowballing, and related approaches, with findings often showing comparable performance or strategy-dependent results. Hybrid strategies subsequently emerged as a distinct research direction, but only a small number of studies had evaluated them.
- Search strategies: Database searching and snowballing were the leading approaches used to identify primary studies in software-engineering SLRs.The related-work overview traces these approaches chronologically.
- Database search: Optimising search strings for retrieving relevant software-engineering experiments was not straightforward across 29 investigated strategies.Dieste et al. analysed different terms and term combinations for SLR searches.
- Database search versus snowballing: Earlier comparisons found similar included articles and conclusions for database searching and backward snowballing, while snowballing could be more efficient with general keywords.The reported efficiency advantage was associated with reducing database-search noise.
- Database search versus snowballing: Snowballing effectiveness was comparable to database searching, but its reliability depended strongly on creating a suitable start set.Other work also reported that forward snowballing can find relevant articles when updating SLRs.
- Hybrid strategies: Only two software-engineering studies had directly addressed hybrid search strategies before this evaluation.Those studies proposed and compared combinations of database searching and snowballing, motivating further evaluation of hybrid approaches.
3. Research design
The research design replicates an industry–academia collaboration SLR and evaluates hybrid-search alternatives through preparation, conduct, and comparison of included articles and topic coverage.
- Research questions: The study asks how many articles the hybrid-search replication finds relative to the original SLR and how four hybrid strategies compare in identified-article counts.These questions define the evaluation targets.
- Hybrid strategies: All four hybrid strategies begin with a Scopus start set and differ in how backward and forward snowballing are sequenced or iterated.The full-fledged strategy applies both directions to all articles meeting inclusion criteria.
- Evaluation approach: The evaluation simulates the alternative strategies by conducting the first strategy while tracking when articles are found.This enables comparison without independently repeating every search process.
- Research process: The research approach consists of prepare, conduct, and evaluate steps.Preparation selects the SLR, conduct identifies a Scopus start set and snowballs it, and evaluation compares articles and topic coverage.
4. Prepare
The preparation selected an industry–academia collaboration SLR and reconstructed its search and inclusion procedures before evaluating a more inclusive replication with wild-card and borderline-article handling.
- Selected SLR: The selected SLR investigates challenges and best practices in industry–academia collaboration in software engineering.The replication therefore focused on articles presenting relevant collaboration findings.
- Original search process: The original SLR searched IEEE Xplore, ACM Digital Library, and Google Scholar using 54 search strings.The strings combined four keyword listings across the three sources.
- Original search process: The original search was conducted in January and February 2015, supplemented by snowballing from five randomly selected articles and searches of additional venues.The venues used for those additional searches were not reported.
- Reviewing process: Wild cards allowed reviewers to nominate articles for full-text assessment despite failing the usual screening threshold.Each reviewer could nominate a wild card, while three researchers independently assessed the articles.
- Reviewing process: Borderline articles received a full-text score of three and were tracked separately for readers to assess.The process distinguishes included, excluded, and excluded-but-borderline articles.
5. Conduct
The replication used a Scopus-derived start set followed by systematic backward and forward snowballing, with reviewer assessment and tracking of all identified articles. It ultimately included nine articles and identified 43 publications, while also recording borderline articles and differing reviewer judgments.
- Start-set construction: The hybrid search began by using Scopus to form a snowballing start set for the replicated SLR.The search targeted industry–academia collaboration in software engineering during 2010–2014, yielding 40 articles.
- Article assessment: Fifteen articles entered full-text reading: twelve selected by reviewer scores and three designated as wild cards.All 15 were then assessed using scores from three reviewers.
- Article assessment: Nine articles were included by unanimous reviewer consensus, while six were excluded despite differing opinions among reviewers.The replication retained records of excluded articles because they had nevertheless been identified by the search strategies.
- Snowballing procedure: Backward snowballing proceeded iteratively through reference lists, adding articles included from earlier backward and forward rounds to later rounds.Reviewers screened reference lists and excluded articles definitively outside the review scope.
- Snowballing procedure: Forward snowballing used Google Scholar citations, restricted to articles published in 2014 or earlier and assessed through peer-reviewed citing-article links.The first author organized citing articles in an Excel sheet to support reviewer assessment.
- Search yield: The replication identified 43 articles published in 2014 or earlier, compared with 33 in the original SLR, and recorded ten borderline articles.Figure 2 presents articles identified across snowballing iterations from the nine-article start set.
6. Evaluate
The evaluation found that the hybrid search identified more relevant studies, while reviewer judgement and scope differences complicated direct comparison. Wild cards and borderline articles supported inclusiveness, and alternative hybrid strategies depended on their starting set.
- Evaluation findings: One wild card article was included after full-text assessment, and four nominated wild cards became borderline articles.The included wild card was absent from the original SLR.
- Evaluation constraints: Comparison was affected by differences in inclusion criteria, search strategy, inclusiveness, and subjective full-text judgement.Some differences between the original SLR and replication were intentional, while others were unintentional.
- Evaluation constraints: The replication used a more inclusive scope, reducing the risk of missing poorly signposted articles but requiring extra effort whose value remains for future research.The focused original scope was more efficient but risked missing essential collaboration experiences.
- Evaluation findings: 30% more relevant publications were found by the replication than by the original SLR, with 20 articles overlapping.The comparison concerns industry–academia collaboration in software engineering.
- Evaluation findings: 22 peer-reviewed articles were included in the original SLR and 43 in the replication, yielding a 45-article peer-reviewed superset.The superset includes the 20 overlapping articles and findings published in 2014 or earlier.
- Evaluation findings: The hybrid strategy outperformed the original database and complementary searches in effectiveness, although search effort and cost-efficiency could not be compared.The replication used one Scopus search plus snowballing, whereas the original used 54 search strings across databases and an indexing service.
- Alternative strategies: Alternative hybrid-strategy performance depended strongly on the start set and publication patterns.In this evaluation, many articles were found through first-round backward and second-round forward snowballing.
- Alternative strategies: An adaptive alternating strategy may be preferable when the start set is relatively small, but selecting search order from reference and citation counts remains future work.The authors identify going backward first as most likely best in this case.
7. Threats to validity
The evaluation faces threats from selection and evaluator bias, implicit agreements, and unassessed publication bias. The study design and reviewer procedures were judged to mitigate some, but not all, of these threats.
- Selection and evaluator bias were identified as the primary threats to conclusion validity.
- The selected SLR and predefined selection criteria were intended to reduce selection bias, although the study design could still favor the hybrid strategy.
- Three independent reviewers, wild cards, and borderline articles helped mitigate evaluator bias and make inclusion boundaries more transparent.
- The study did not assess whether publication bias favored a particular search strategy.
8. Conclusions
The study compares a traditional database-and-indexing-service search with hybrid strategies in a replicated SLR. The full-fledged hybrid strategy found more relevant articles, while alternative strategies traded effort against possible omissions.
- The replication compared the original SLR’s database and indexing-service searches with four alternative hybrid search strategies.
- 30% more articles were found by the full-fledged hybrid strategy than by the original SLR, even though the replication included fewer publication types.
- The full-fledged hybrid strategy was superior to the alternatives but required more effort to assess all identified articles.
- Alternative hybrid strategies depend on the start set; with relatively new starting articles, backward snowballing found more inclusions initially than forward snowballing.
- Wild cards allowed reviewers to advance disputed articles to full-text assessment, while borderline articles were retained for readers’ independent judgements.
- The full-fledged hybrid strategy identified substantially more articles on industry-academia collaboration than the original SLR.
Appendix A. Supplement – Selecting an SLR for replication
Appendix A describes how the study selected an SLR for replication.
- Appendix A describes the procedure used to select an SLR for replication.
Appendix B. Supplement – Included articles and borderline articles
Appendix B lists the included and borderline articles and records where each was identified during the hybrid search.
- Appendix B lists included and borderline articles together with the hybrid-search step that identified each article.