PaperlayerView exact revision on arXiv

Source-linked AI summary

Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?

Shuai Wang, Harrisen Scells, Bevan Koopman, Guido Zuccon

arXiv:2302.03495v3cs.IRcs.AI

TL;DR

The paper investigates whether ChatGPT can generate and refine Boolean queries for systematic review literature searches. Across extensive experiments, ChatGPT generally improves precision at the expense of recall, with refinement particularly effective for queries produced by the objective formulation method.

  • Problem

    Constructing effective Boolean queries for systematic reviews is time-consuming, while poor queries can retrieve too few or too many relevant studies.

  • Method

    The study evaluates unguided and guided prompts, example-query prompting, and ChatGPT-based refinement for Boolean query formulation and refinement.

  • Results

    ChatGPT compares favourably with state-of-the-art automated methods in precision, while refinement effectively improves existing queries, especially those generated by the objective method.

  • Takeaways & Limitations

    ChatGPT shows potential for systematic review search, particularly when higher precision is preferred and some recall trade-off is acceptable.

  • Takeaways & Limitations

    ChatGPT’s non-deterministic outputs can produce substantially different query effectiveness across repeated generations with the same prompt.

Abstract

from arXiv · show

Systematic reviews are comprehensive reviews of the literature for a highly focused research question. These reviews are often treated as the highest form of evidence in evidence-based medicine, and are the key strategy to answer research questions in the medical field. To create a high-quality systematic review, complex Boolean queries are often constructed to retrieve studies for the review topic. However, it often takes a long time for systematic review researchers to construct a high quality systematic review Boolean query, and often the resulting queries are far from effective. Poor queries may lead to biased or invalid reviews, because they missed to retrieve key evidence, or to extensive increase in review costs, because they retrieved too many irrelevant studies. Recent advances in Transformer-based generative models have shown great potential to effectively follow instructions from users and generate answers based on the instructions being made. In this paper, we investigate the effectiveness of the latest of such models, ChatGPT, in generating effective Boolean queries for systematic review literature search. Through a number of extensive experiments on standard test collections for the task, we find that ChatGPT is capable of generating queries that lead to high search precision, although trading-off this for recall. Overall, our study demonstrates the potential of ChatGPT in generating effective Boolean queries for systematic review literature search. The ability of ChatGPT to follow complex instructions and generate queries with high precision makes it a valuable tool for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and often trading-off higher precision for lower recall is acceptable.

1 INTRODUCTION

The paper investigates ChatGPT for constructing systematic-review Boolean queries, addressing the time and effectiveness challenges of existing query development. It evaluates prompting strategies and reports promising precision-oriented performance alongside practical caveats.

  • 1 INTRODUCTION: Systematic reviews require Boolean queries to retrieve evidence relevant to a focused research topic.These queries support reproducibility, explainability, and filtering of irrelevant articles.
  • 1 INTRODUCTION: Automated Boolean query formulation and refinement methods have been developed, but existing techniques remain unable to consistently produce high-quality queries.The paper positions ChatGPT as a generative alternative for this task.
  • 1 INTRODUCTION: The study evaluates ChatGPT against state-of-the-art methods and examines how prompts and multiple interactions affect generated-query effectiveness.It also investigates caveats and challenges associated with ChatGPT-based query creation.
  • 1 INTRODUCTION: The evaluation is presented as the first comprehensive investigation of ChatGPT for creating Boolean queries for systematic-review literature search.The study uses engineered prompts and standardised systematic-review test collections.
  • 1 INTRODUCTION: ChatGPT compares favourably with current query-generation methods, achieving high precision while trading off recall.The findings support potential use in settings where precision is prioritised, while the authors also identify limitations and caveats.

2 RELATED WORK

Related work frames systematic-review query construction as a time-consuming, expertise-dependent process and surveys automation across review phases. It motivates ChatGPT as an underexplored approach to Boolean query formulation and refinement.

  • 2 RELATED WORK: Systematic-review automation spans question definition, query authoring, screening, data extraction, and evidence synthesis.The paper focuses on query generation because query quality affects the results returned and later review phases.
  • 2 RELATED WORK: Query formulation derives a Boolean query from a research question using established guidelines.The conceptual method identifies concepts and synonyms iteratively, while the objective method extracts and validates terms from seed studies.
  • 2 RELATED WORK: Existing formulation procedures require substantial information-specialist time and remain prone to human error.Automating them has not matched human effectiveness, although manual refinement of automatically generated queries can help.
  • 2 RELATED WORK: Automatic query-refinement methods transform an initial human-authored query by adding terms or clauses to improve effectiveness.These methods were developed from observed benefits of manual query refinement.
  • 2 RELATED WORK: The paper proposes unguided and guided ChatGPT formulation, alongside refinement of existing queries.Guided prompting asks ChatGPT to follow conceptual or objective formulation procedures.
  • 2 RELATED WORK: Prompt engineering guides generative language models to perform tasks through natural-language instructions.Its use for Boolean query creation in systematic-review search had not previously been examined in the cited work.

3 ENGINEERING PROMPTS FOR SYSTEMATIC REVIEWS

The paper designs progressively complex prompts for ChatGPT to formulate or refine systematic-review Boolean queries. The designs include examples, PICO guidance, related examples, and multi-step prompting based on the objective method.

  • 3 ENGINEERING PROMPTS FOR SYSTEMATIC REVIEWS: ChatGPT receives a text prompt, processes it with a Transformer model, and generates a response token by token.The paper studies how prompt design controls Boolean-query generation.
  • 3 ENGINEERING PROMPTS FOR SYSTEMATIC REVIEWS: The study develops prompts of increasing complexity for both Boolean-query formulation and refinement.It compares example-based prompts, multi-interaction guided prompts, and prompts informed by automated formulation methods.
  • 3.1 Single Prompts for Query Formulation: Single formulation prompts use a review title and are grouped as simple, detailed, or example-based.Five formulation prompts are designed to represent these categories.
  • 3.1 Single Prompts for Query Formulation: Example formulation prompts may describe a Boolean-query strategy for identifying PICO elements.PICO denotes Patient/Problem, Intervention, Comparison, and Outcome.
  • 3.1 Single Prompts for Query Formulation: Example-based prompts substitute the target review title, example review title, and example Boolean query into designed templates.The paper illustrates this process with a trauma-related thromboelastography review example.
  • 3.2 Single Prompts for Query Refinement: Single refinement prompts provide a review title and corresponding Boolean query, then request a modification intended to improve search effectiveness.They can support manual query improvement or integration into an automated pipeline.
  • 3.2 Single Prompts for Query Refinement: Refinement prompts are categorised as simple or example-based, with examples showing ChatGPT what successful refinement means.The study examines both high-quality systematic-review examples and semantically related examples selected with a neural ranking model.
  • 3.3 Guided Prompts for Query Formulation: A guided formulation prompt follows the objective method through a four-step process beginning with term identification from a seed study.The identified terms are then classified into categories before query generation continues.

4 EXPERIMENTAL SETTINGS

The experiments use CLEF TAR and Seed Collection systematic-review datasets to test ChatGPT-generated Boolean queries. Queries are generated from designed prompts, reformulated when incorrectly formatted, executed through PubMed’s Entrez API, and evaluated with set-based retrieval measures.

  • Datasets: The experiments use CLEF TAR datasets from 2017–2018 and a Seed Collection containing 40 systematic-review topics with seed studies.Both collections contain review titles, Boolean queries, and relevant assessments.
  • Prompt construction: Prompts replace designed-template fields with review-topic information from the collections.For example-based prompts, topic CD010438 supplies a high-quality example query for both collections.
  • Experimental scope: Query refinement is omitted for the Seed Collection, while guided prompting is omitted for CLEF TAR because the required seed studies are unavailable.Guided prompting relies on at least one seed study, which the CLEF TAR collection does not contain.
  • Query generation: Incorrectly formatted Boolean queries are removed and regenerated for the relevant review topic.This formatting filter is applied after ChatGPT generates the queries.
  • Evaluation: Generated queries are executed through PubMed’s Entrez API and assessed using precision, f-measure, and recall against abstract-level relevance judgments.The evaluation uses retrieved PubMed IDs and set-based measures.

5 MAIN RESULTS

ChatGPT-generated queries generally improve precision and can improve f-measure or refinement effectiveness, but often trade recall for these gains. Prompt examples, query refinement, and guided prompting affect performance, while results remain variable across runs and seed studies.

  • 5.1 Single Prompt Query Formulation: ChatGPT-generated formulation queries generally achieve higher precision but lower recall than existing methods; f-measure is higher on CLEF and lower on the Seed Collection.The paper compares ChatGPT with state-of-the-art and original authored queries.
  • 5.1 Single Prompt Query Formulation: Including a high-quality query example consistently improves F_1, F_3, and recall, whereas prompt detail has only a minor effect.Generating PICO elements first produces higher precision but lower recall than the alternative example prompt.
  • 5.1 Single Prompt Query Formulation: Relevant-topic examples yield higher precision but lower recall than high-quality examples, which partially mitigates ChatGPT’s low-recall pattern.This comparison is reported for both CLEF and the Seed Collection, with generalisability limited by the Seed Collection’s number of topics.
  • 5.1 Single Prompt Query Formulation: Recall varies most across repeated formulation runs, with variance equal to 12% of its mean, compared with 7.1% for precision, 6.6% for F-1, and 7.2% for F-3.The repeated runs use the same prompt, q4.
  • 5.2 Single Prompt Query Refinement: ChatGPT refinement increases precision and f-measure while lowering recall, making a high-recall seed query important before refinement.The paper reports that refinement improves existing Boolean queries, with the seed query providing the initial recall.
  • 5.2 Single Prompt Query Refinement: Objective-method seed queries produce the greatest refinement improvement and the highest initial recall; refinement then causes an 11% recall drop alongside gains in precision, F-1, and F-3.Query refinement is less variable than formulation because the seed-query structure is already supplied.
  • 5.3 Guided Prompt Query Formulation: Well-chosen seed studies make guided-prompt queries more effective than single-prompt queries, but effectiveness varies across runs and depends on the seed study.Variability is examined both across different seed studies and across repeated runs using the same seed study.
  • 5.3 Guided Prompt Query Formulation: Guided-prompt generation is unstable: some topics show precision and recall ranges from 0 to 1 across runs, especially when average effectiveness is high.The instability occurs both across seed studies and when the same seed study is reused.

6 QUERY FAILURE ANALYSIS

The failure analysis identifies excessive retrieval, incorrect MeSH terms, and possible uncertainty in relevance judgments as characteristics or explanations of poorly performing queries.

  • Seven successful and six failed topics were identified for query formulation, while refinement had three successful and eleven failed topics.
  • For formulation, failing queries had a median retrieved-to-relevant ratio of 10.28, compared with 0.30 for successful queries.
  • For refinement, the corresponding median ratios were 2.74 for failing queries and 1.39 for successful queries.
  • The best formulation prompt generated three MeSH terms on average, with 55% absent from the MeSH vocabulary.
  • Relevance judgments relied on the original query, which may bias evaluation toward that query.
  • For some failing queries, unjudged documents exceeded 94.8% of retrieved documents, versus 63.7% for original queries, though this pattern differed across topics.

7 SUMMARY OF FINDINGS

The study finds that ChatGPT can produce competitive Boolean queries, with prompt design and iterative guidance substantially affecting effectiveness. Its gains in precision are often accompanied by lower recall, while incorrect MeSH terms and nondeterministic outputs remain important caveats.

  • RQ1: ChatGPT compared favourably with state-of-the-art formulation methods in precision, but at the expense of lower recall.
  • RQ1: ChatGPT effectively refined existing queries, with the largest improvements obtained when refining queries generated by the objective formulation method.
  • RQ2: Prompt type had considerable effects on the effectiveness of generated queries.
  • RQ2: Example Boolean queries generally improved results; semantically close examples favoured precision, whereas high-quality unrelated examples favoured recall.
  • RQ2: Instructions to generate PICO elements caused considerable recall losses and improved precision only on the CLEF collection.
  • RQ3: Guided multi-prompt strategies produced higher effectiveness than single prompts, improving both precision and recall.
  • RQ4: The main caveats are incorrect MeSH terms and high variability in query effectiveness across repeated requests.

8 LIMITATIONS IN OUR USE OF CHATGPT

The study’s black-box use of ChatGPT leaves important uncertainties about model internals, training-data exposure, retained user information, and undocumented model changes.

  • ChatGPT’s undisclosed architecture and training data leave open whether it may have seen test-collection queries during training.
  • Experiments ran over two weeks in early January 2023, and undocumented engineering or model updates could not be controlled.

9 CONCLUSION

The paper evaluates ChatGPT for systematic-review Boolean query formulation and refinement across more than 100 topics. It reports higher precision but lower recall than existing automatic formulation methods, while guided prompts improve effectiveness; variability and reproducibility remain unresolved concerns.

  • The study evaluated ChatGPT for formulating and refining Boolean queries across more than 100 systematic review topics.
  • ChatGPT-generated queries achieved higher precision but lower recall than queries from current state-of-the-art automatic formulation methods.
  • Guided prompting produced Boolean queries with higher effectiveness than unguided prompting.
  • Different queries can be generated from the same prompt, and their effectiveness can vary.
  • The paper contributes prompt strategies, broad evaluation, and failure analysis while identifying variability, robustness, and reproducibility as unresolved issues.
Loading 2302.03495v3…