Search Strategy Formulation for Systematic Reviews: issues, challenges and opportunities

Definition of terms and Review Scope

• Search strategies for systematic reviews should support reproducibility and replicability, transparency and explainability, user trust and control, and high recall to reduce consequential omissions.The review notes that omissions may leave evidence incomplete or biased, potentially affecting outcomes such as clinical guidance.Source: Definition of terms and Review Scope · p. 2
• Trust is user-focused: searchers need to trust the system and feel in control, a concern also raised in discussions of “Humans in the Loop”.Source: Definition of terms and Review Scope · p. 3
• Control over strategy design can include switching auto term mapping or query expansion on or off and being informed of each change’s impact.The review links such control to developing a transparent and explainable search strategy.Source: Definition of terms and Review Scope · p. 3
• Transparency explains how a system works, while explainability addresses why it acted; searchers should be able to understand strategy construction and why it produced a particular result set.The review says current systems are not explainable in this sense and that explainability can support trust, reproducibility, and replicability.Source: Definition of terms and Review Scope · p. 3

Reproducibility and Replicability

• Using ACM definitions, reproducibility means another team uses the same test collection for directly comparable results, whereas replicability uses a different collection.The different collection may also differ in topics, documents, relevance assessments, and results.Source: Reproducibility and Replicability · p. 3
• The review groups impediments to successful search-strategy formulation into formalism, platform, and community barriers.Formalism concerns the representation scheme, platform concerns tooling and technologies, and community concerns adopted processes, practices, and conventions.Source: Reproducibility and Replicability · p. 4
• The review uses these barrier categories throughout its literature analysis to identify problems affecting the concepts it examines.Source: Reproducibility and Replicability · p. 4

Search Strategy Development Methods: Current Practice

• The typical workflow moves from defining concepts and facets through term collection, Boolean combination, testing, team feedback, database translation, and documentation.PICO can help structure concepts, and published search templates or filters may be considered during testing.Source: Search Strategy Development Methods:… · p. 4

Boolean Search strategies for Systematic Reviews

• Searchers combine synonyms within facets using OR and facets using AND, with operators such as AND NOT, adjacency, truncation, wildcards, and field restrictions.Single-line, block-by-block, and line-by-line formats are possible; the choice affects transparency and explainability, as the complex example illustrates.Source: Boolean Search strategies for… · pp. 4–5, Figure 1 · p. 5
• Boolean strategies can obscure how facets and terms combine and scale poorly, limiting transparency, reproducibility, replicability, and explainability.Information needs may involve many facets and terms spread over multiple pages, creating a formalism barrier.Source: Boolean Search strategies for… · p. 5

Search strategy development

• Although Boolean basics can be learned quickly, no design standards exist, and effective professional skill takes years to develop through workplace learning (Yoo & Mosa, 2015).Searchers may lack knowledge of effective keyword and operator tactics, contributing to exploratory searching.Source: Search strategy development · pp. 5–6
• Exploratory iterations can increase review-development costs, and searchers may struggle to know when enough studies have been found (Hemmingway and Brereton, 2009; Borah et al, 2017; Booth, 2010).PRESS 2015 can audit a strategy, but its open questions and subjectivity can make revision difficult or expensive (McGowan et al, 2016; Shokraneh, 2018).Source: Search strategy development · p. 6
• Strategies are not always published, development iterations are often undocumented, and the lack of a central repository can compromise reproducibility and duplicate work (Koffel and Rethlefsen, 2016; Biocic et al, 2019; Shokraneh, 2018).The documentation gap is also identified as a formalism and platform barrier (Russell-Rose and Shokraneh. 2020).Source: Search strategy development · p. 6
• Conventional Boolean search systems do not fully meet searcher needs, particularly for merging queries or results and publishing strategies to support reproducibility and replicability.The search strategy output itself can be time-consuming and difficult to replicate.Source: Search Strategy Development Methods:… · p. 4, Search strategy development · p. 6

Errors in search strategies

• 90% of published MEDLINE strategies had at least one error, 80% of those errors limited search effectiveness, and Cochrane reviews had 73% faulty designs, 53% accuracy-limiting errors, and a 92.7% error rate in a later study.The figures come from Sampson and McGowan (2006), Franco et al (2018), and Salvador-Oliván et al (2019), respectively.Source: Errors in search strategies · p. 6
• Search strategy errors are strategic, tactical, or logical, including incorrect line references, missing synonyms, and using OR when AND was required.Strategic errors can also involve redundant overlapping elements or incorrect translation to a database; tactical errors include spelling, truncation, and subject-heading mistakes.Source: Errors in search strategies · p. 6
• These errors occur even among experienced Boolean searchers, highlighting limited transparency and explainability as a formalism barrier.Source: Errors in search strategies · p. 6

Heterogeneity in systems

• Database operators can differ in syntax and semantics, so translating strategies across systems risks errors in NOT, proximity, truncation, and wildcard use.For example, NEAR/5 allows a maximum of five intervening words in Web of Science but four in Embase; truncation symbols can also mean different character counts across interfaces.Source: Heterogeneity in systems · p. 7
• Different databases can use different terms for the same concept, requiring searchers to learn varying operators and knowledge-organization schemes and complicating reproducibility and replicability.The example uses ‘AND’ between fields: Breast Neoplasms/ OR (Breast/ AND Neoplasms/).Source: Heterogeneity in systems · p. 7

Summary of problems with Boolean Strategies

• Boolean strategies are difficult to conceptualize, maintain, and explain, and their complexity and system heterogeneity limit transparency, reproducibility, and replicability.They can also be time consuming and error-prone, even for searchers with significant prior knowledge and experience.Source: Summary of problems with Boolean… · p. 7
• Despite these shortcomings, Boolean strategies remain trusted foundations for systematic reviews and other domains because exact-match logic offers control and transparency that best-match ranking schemes do not.The passage names healthcare, law, patents, and recruitment as domains where Boolean strategies continue to be used.Source: Summary of problems with Boolean… · p. 7

Tools and Approaches to Augment Search Strategy Development

• Proposed tools and approaches extend beyond Boolean methods, but remain short of widespread use unless they demonstrate significant benefits and address transparency concerns.The review also considers hybrid schemes, ranking, machine learning, and potential data-modelling and representation methods.Source: Tools and Approaches to Augment Search… · p. 8

Tools 1: Text and Data Extraction

• Text and data extraction can support scoping searches and identify search terms from gold studies, protocols, cited documents, or keyword co-occurrence networks.Examples include PICO facet analysis, keyword extraction from protocols, and filters based on cited documents.Source: Tools 1: Text and Data Extraction · p. 8
• Text-mining approaches can help identify terms and address spelling or synonym errors, but their best applications remain unknown and they do not resolve search-strategy complexity.O’Mara-Eves et al (2015) found almost no replication between studies, and Olorisade et al (2017) demonstrated severe limits in independently matching published study outcomes; NLP transparency and explainability may also impede acceptance.Source: Tools 1: Text and Data Extraction · p. 8

Tools 2: Automatic Query Expansion (AQE)

• Decision support systems can suggest keywords, operators, and combinations, but healthcare systems focus on end users rather than systematic-review production and are not particularly explainable.Extraction methods could populate such systems using PICO or other frameworks; related healthcare technologies are described by O’Sullivan et al (2010; 2013), while Tsafnat et al (2014) discuss DSS tactics.Source: Tools 2: Automatic Query Expansion (AQE) · pp. 8–9
• Automatic query expansion methods transform queries through syntactic or semantic techniques, but the reviewed methods retain Boolean logic and associated limitations, including lack of transparency.Scells et al (2018a) examined operator replacement, MeSH explosion, field restriction, and adjacency replacement; Scells et al (2019;2020c) added semantic methods, while Scells et al (2020a) used gold studies to optimize queries.Source: Tools 2: Automatic Query Expansion (AQE) · p. 9

Tools 3: Tool Automation

• Automated tools face workflow incompatibility, poor support, steep learning curves, and usability and functionality problems, creating substantial adoption barriers.The review argues new systems must be seamless and easier to use, logically equivalent to Boolean approaches at minimum, and transparent enough to support trust (van Altena et al, 2019; O’Connor, 2019; Yoo & Mosa, 2015; O’Connor et al, 2019).Source: Tools 3: Tool Automation · p. 9

• Professional-search adoption of ranking algorithms is limited partly by transparency concerns, while BM25 has been augmented with learning-to-rank schemes (Russell-Rose et al, 2018; Li et al, 2018).Source: Tools 4: Ranking algorithms, Machine… · p. 9
• Machine learning has filtered review articles using NLP-identified terms and protocol terms.Source: Tools 4: Ranking algorithms, Machine… · pp. 9–10
• Neural networks, Boolean-query learning-to-rank, and dynamic approaches have also been proposed; living reviews use periodic machine-learning classification and information extraction (Marshall and Wallace, 2019; Scells et al, 2020b; Cormack and Grossman, 2018; Thomas et al, 2017).Source: Tools 4: Ranking algorithms, Machine… · p. 10
• Despite utility, limited explainability inhibits adoption, and models often target specific study designs or topics rather than multifaceted searches.Source: Tools 4: Ranking algorithms, Machine… · p. 10
• They can still serve as query filters or reformulation tools after query development (Del Fiol et al, 2018; Russell-Rose et al, 2021).Source: Tools 4: Ranking algorithms, Machine… · p. 10
• Some models, including BioBERT, are beginning to have significant impact (Lee et al, 2020).Source: Tools 4: Ranking algorithms, Machine… · p. 10

Tools 5: Hybrid Schemes

• Karimi et al (2010) found evidence that the hybrid scheme pushes more relevant documents higher up the ranked list.The experiments combined Boolean and ranking schemes.Source: Tools 5: Hybrid Schemes · p. 10
• Despite its focus on reproducibility and Boolean-strategy problems, the hybrid approach leaves the conceptual challenge of building the search strategy unresolved.The authors also acknowledge the downsides of ranking.Source: Tools 5: Hybrid Schemes · p. 10

Tools 6: Structures/Representation

• Declarative search specifies relevant documents by what is wanted rather than how to retrieve them, with Verberne (2018) proposing knowledge graphs for transparent professional search.Russell-Rose et al (2019) and Russell-Rose (2019) describe a 2D sketchpad that represents concepts as blocks and establishes relationships, mapping more directly to intended semantics.Source: Tools 6: Structures/Representation · p. 10
• Venn diagrams and “dust and magnet” representations retain Boolean logic while offering transparency and better concept representation.These approaches partially address the formalism barrier by allowing strategies to be recorded and shared separately from the search platform.Source: Tools 6: Structures/Representation · pp. 10–11

Approaches 1: Data Modelling/Theory

• Boolean logic underlies systematic-review search strategies, with proximity, truncation/wildcards, and search fields treated as extensions.Proximity is a special case of AND, truncation/wildcards a special case of OR, and search fields are based on metadata.Source: Approaches 1: Data Modelling/Theory · p. 11
• Ranking is not accepted by the community, while fuzzy logic has gained little headway in information retrieval and may face similar implementation resistance.Source: Approaches 1: Data Modelling/Theory · p. 11
• To the authors’ knowledge, no serious attempt has used other models or theories beyond those reviewed.Source: Approaches 1: Data Modelling/Theory · p. 11

Approaches 2: Explainability

• There is little work on explanations for systematic-review search strategies, and little primary research specifically on professional search.Source: Approaches 2: Explainability · p. 11
• Mi & Jiang (2019) and Thomas et al (2019) focus on screening-result interpretability, not the earlier professional-search problem.Source: Approaches 2: Explainability · p. 11
• Russell-Rose and MacFarlane (2020) and Verberne (2018) identify gaps and issues in professional search.Source: Approaches 2: Explainability · p. 11

Summary of Tools and Approaches for Search Strategy Development

• Some methods supplement Boolean search but only help reformulate queries after relevant or ‘gold standard’ studies have been identified.Unsilo (2021) and ResearchRabbit (2021) can help develop initial queries for non-complex topics, but applying them to complex topics remains a long way off, if possible at all, given the tacit knowledge involved.Source: Summary of Tools and Approaches for… · p. 11
• Approaches use Boolean methods or add machine learning and information extraction to standard retrieval models such as BM25, retaining many conventional limitations.Some structured approaches use frameworks such as PICO, SPIDER, or SPICE, but little work explains the underlying problem or helps searchers understand their information needs.Source: Summary of Tools and Approaches for… · p. 11
• Resistance to automated tools reflects workflow incompatibility and steep learning curves, while limited transparency and explainability can erode trust (van Altena et al, 2019; O’Connor, 2019; Yoo & Mosa, 2015).The review calls for new solutions that make search strategies transparent and trustworthy and facilitate explainability, reproducibility, and replicability.Source: Summary of Tools and Approaches for… · p. 11

Discussion: A Framework for Search Strategy Formulation

• The framework proposes five principles: transparent visual mappings, scalable declarative formalisms, automated syntax handling, real-time query feedback, and support for collaboration and team working.The principles are intended to make search strategies easier to understand, manage, test, and share.Source: Discussion: A Framework for Search… · p. 12
• Visual mappings should connect logical and physical structure, while scalable formalisms support abstraction, encapsulation, and movement between overview and detail.Users should also be able to manipulate individual query elements independently.Source: Discussion: A Framework for Search… · pp. 12–13
• Automating lower-level syntax and providing real-time feedback can reduce error-prone string manipulation and help users understand each query element’s contribution to effectiveness.The framework maps abstractions and feedback to the prevalence of errors in published strategies and to limited query-management support.Source: Discussion: A Framework for Search… · pp. 12, 14
• Repositories of best-practice examples can support versioning, sharing, and peer review, addressing barriers in collaboration, search documentation, and strategy sharing.The framework also links team-working support to separation of search strategies from platforms during reporting and sharing.Source: Discussion: A Framework for Search… · pp. 12, 14
• The framework maps principles to barriers including difficulty articulating strategy structure, limited scalability, steep learning curves, development costs, and inadequate standards or guidelines.It also proposes abstractions, feedback, and standard models to address errors, system heterogeneity, limited automation transparency, and workflow compatibility.Source: Discussion: A Framework for Search… · pp. 13–14
• Applying the principles requires accommodating needs from relatively simple, automatable scoping reviews to complex systematic reviews requiring substantial human input.The paper identifies transparency and explainability as key to trust, which in turn supports replicability and reproducibility.Source: Discussion: A Framework for Search… · p. 15

Summary

• The paper uses reproducibility, replicability, transparency, explainability and trust to examine problems professional searchers face when developing effective evidence-synthesis search strategies.The analysis focuses on healthcare.Source: Summary · p. 15
• Its main contribution is a framework for the information retrieval and systematic review communities to address the identified problems.Source: Summary · p. 15
• The authors aim to bring the information retrieval and systematic review communities together to collaborate on solutions, while noting that other domains such as patents, law and recruitment may also benefit.Source: Summary · p. 15