PaperlayerView exact revision on arXiv

Source-linked AI summary

The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review

Dmitry Scherbakov, Nina Hubig, Vinita Jansari, Alexander Bakumenko, Leslie A. Lenert

arXiv:2409.04600v1cs.DLcs.AI

TL;DR

Scientific information can be overwhelming, and completing reviews requires substantial, sometimes unpredictable resources. This study reviews research projects using LLMs to automate review work; it reports that few papers address the full review cycle, while concluding that LLMs are likely to reduce review time and produce similar or higher-quality data in greater quantities than manual reviews.

  • Problem

    Scientific information can be overwhelming, and completing reviews requires substantial resources with unpredictable time demands.

    Source: Introduction · p. 4

  • Method

    The study reviews research projects using LLMs to automate systematic and other review stages.

    Source: Short description · p. 2, Discussion · p. 15

  • Results

    Few papers focused on the full review-automation cycle, while the work demonstrated LLM use in abstract and full-text screening, data extraction, and knowledge synthesis.

    Source: Discussion · p. 15

  • Takeaways & Limitations

    The authors conclude that LLMs are likely to reduce review time while producing similar or higher-quality data in greater quantities than manual reviews.

    Source: Conclusion · p. 16

    The review's assessment of research projects is limited by its simplified scoring system.

    Source: Limitations · p. 16

Abstract

from arXiv · show

Objective: This study aims to summarize the usage of Large Language Models (LLMs) in the process of creating a scientific review. We look at the range of stages in a review that can be automated and assess the current state-of-the-art research projects in the field. Materials and Methods: The search was conducted in June 2024 in PubMed, Scopus, Dimensions, and Google Scholar databases by human reviewers. Screening and extraction process took place in Covidence with the help of LLM add-on which uses OpenAI gpt-4o model. ChatGPT was used to clean extracted data and generate code for figures in this manuscript, ChatGPT and Scite.ai were used in drafting all components of the manuscript, except the methods and discussion sections. Results: 3,788 articles were retrieved, and 172 studies were deemed eligible for the final review. ChatGPT and GPT-based LLM emerged as the most dominant architecture for review automation (n=126, 73.2%). A significant number of review automation projects were found, but only a limited number of papers (n=26, 15.1%) were actual reviews that used LLM during their creation. Most citations focused on automation of a particular stage of review, such as Searching for publications (n=60, 34.9%), and Data extraction (n=54, 31.4%). When comparing pooled performance of GPT-based and BERT-based models, the former were better in data extraction with mean precision 83.0% (SD=10.4), and recall 86.0% (SD=9.8), while being slightly less accurate in title and abstract screening stage (Maccuracy=77.3%, SD=13.0). Discussion/Conclusion: Our LLM-assisted systematic review revealed a significant number of research projects related to review automation using LLMs. The results looked promising, and we anticipate that LLMs will change in the near future the way the scientific reviews are conducted.

Short description

The study examines how large language models are used to create scientific reviews. It turns the tool “on itself” by reviewing research projects that use LLMs for systematic and other types of reviews.

Short description

  • The study aims to summarize LLM use in creating scientific reviews.It focuses on the range of review-process stages that can be automated.Source: Short description · p. 2
  • The authors describe the work as turning the tool “on itself” by reviewing projects that use LLMs for reviews.Source: Short description · p. 2
  • The reviewed projects concern systematic and other types of reviews.Source: Short description · p. 2

Introduction

Scientific information can be overwhelming, while reviews require substantial, sometimes unpredictable resources and may take months or years. The authors examine LLM automation projects to map their scope and assess project quality and performance.

Introduction

  • Reviews require substantial resources, and their completion time can be unpredictable and extend to months or years.Source: Introduction · p. 4
  • Existing review platforms organize manual work, including screening and extraction, which typically require two independent analysts and may involve an expert resolving disagreements.Source: Introduction · p. 4
  • Even double-checking reviewers may miss up to 3% of relevant citations, while a single reviewer may miss as many as 13%.Source: Introduction · p. 4
  • LLMs have emerged as powerful NLP tools across different tasks, motivating examination of their use to guide and direct the review process.Source: Introduction · p. 5
  • The review aims to summarize LLM automation projects, map review types and stages, and assess project quality and LLM performance.Source: Introduction · p. 5
  • The authors frame the work as using LLMs to review research on LLM use in systematic reviews.Source: Introduction · p. 5

Methods

The authors developed a review plan using systematic-review guidance and defined inclusion and exclusion criteria. A human reviewer searched four databases in June 2024 for eligible English-language publications on LLM automation of review stages.

Methods

  • The review plan followed PRISMA-P 2015 and JBI guidance, and the protocol was registered with the Open Science Framework.Source: Methods · p. 5
  • Eligible publications focused on LLM use to automate review stages and were English-language publications, including conference abstracts and reviews created with LLMs.Source: Methods · p. 5
  • Studies were excluded if they did not use an LLM, describe review-process automation, or provide retrievable English full text; review articles that did not use LLMs were also excluded.Source: Methods · p. 5
  • A human reviewer searched PubMed, Scopus, Dimensions, and Google Scholar in June 2024.Source: Methods · p. 5

Source: Authors’ own work

The team used a Covidence add-on to automate screening and extraction with LLM assistance, while human reviewers calibrated prompts, resolved screening decisions, and verified outputs. They also used LLM tools in manuscript preparation and checked the final draft.

Source: Authors’ own work

  • The Covidence add-on automated abstract screening, full-text screening, and extraction, with two human reviewers calibrating criteria and prompts.Source: Source: Authors’ own work · pp. 6–7
  • LLM decisions were run three times and finalized by majority vote; abstract screening involved human consensus, while one reviewer verified full-text screening and extraction.Source: Source: Authors’ own work · p. 7
  • The extraction form captured study location, review type and stage, performance metrics, time savings, and the study team's view of LLM use.Source: Source: Authors’ own work · p. 7
  • Human reviewers assessed study quality using categories including author-affiliation rankings, sample size, funding, journal impact factor, review status, and benchmark reporting.Studies were classified as low, medium, or high quality from the mean points across categories.Source: Source: Authors’ own work · pp. 7–8
  • NotebookLM and manual review cross-checked low-precision extraction fields, and ChatGPT cleaned extracted data and helped draft text and figure code.Human experts edited and verified the final LLM-generated manuscript draft.Source: Source: Authors’ own work · pp. 8–9

Results

The review included 172 studies, spanning contributions from 43 countries and a range of automated review stages. Publication searching and data extraction were the most frequently mentioned stages.

Results

  • 3,788 records yielded 172 included studies after 447 duplicates were removed, 3,041 title/abstract exclusions, and full-text assessment of 300.Source: Results · p. 9, Figure 2 · p. 10
  • Among 47 US studies with state data, Tennessee, New York, and Massachusetts each led with 5 (10.6%); California had 4 (8.5%).Source: Results · p. 11
  • 43 countries contributed studies; the US led (n=60, 34.9%), followed by Australia (n=14, 8.14%), and the UK and China (n=13, 7.6%).Source: Results · p. 11, Figure 3 · p. 12
  • Other reported stages included title/abstract screening (n=43, 25.0%), drafting a publication (n=22, 12.8%), and full-text screening (n=14, 8.1%).Source: Results · p. 12
  • Quality and bias assessment (n=12, 7.0%), publication classification (n=10, 5.8%), other stages (n=6, 3.5%), and code and plots generation (n=4, 2.3%) were also reported.Source: Results · p. 12
  • Publication searching was the most frequently mentioned automated stage (n=60, 34.9%), followed by data extraction (n=54, 31.4%) and evidence synthesis/summarization (n=32, 18.6%).Source: Results · p. 12, Figure 4 · p. 13

Source: ChatGPT-generated code using extracted data from this manuscript (Supplemental Table S7)

GPT-based models performed better in data extraction but had lower title/abstract screening accuracy than BERT-based models. Most reviewed publications described automating a review stage rather than reviews conducted with LLM assistance.

Results

Discussion

The discussion finds many LLM review-automation projects but few full-cycle efforts. It describes an applied method spanning screening through synthesis, alongside automation of manuscript drafting and plot generation.

Discussion

  • Many LLM review-automation projects were identified, but only a few papers addressed the full review cycle.Source: Discussion · pp. 14–15
  • Journals' requests about LLM-generated content may create perceived publication barriers, but the authors lack evidence on effects on the review process.Source: Discussion · p. 15
  • The review's scale exceeded 3,000 screened abstracts and included 172 eligible full-text publications, with automation also reaching manuscript drafting and plot generation.Source: Discussion · p. 15
  • The applied method automated abstract and full-text screening, data extraction, and knowledge synthesis, demonstrating direct applicability.Source: Discussion · p. 15
  • The authors describe the method as domain-agnostic for integration into large-scale projects across domains, with expected workload reduction and efficiency gains.Source: Discussion · p. 15
  • GPT-based LLMs were the dominant type and appeared to show remarkable data-extraction results, a complex and time-consuming review stage.Source: Discussion · p. 15

Limitations

The authors caution that some extraction categories had relatively low accuracy, while recommending improvements to extraction performance and further evaluation of LLM integrations. They also propose shifting human effort toward supervising review automation.

Limitations

  • Lower-accuracy extraction categories warrant caution, and the authors identify precision and recall improvement as a research priority.They also propose exploring combinations of different LLMs and other AI models, with review-quality effects and ethical considerations assessed.Source: Limitations · p. 16
  • The authors suggest that LLMs could substitute for human reviewers, with human effort redirected to supervising the review process.This proposal follows their report of remarkable LLM accuracy, while acknowledging lower accuracy in some extraction categories.Source: Limitations · p. 16
  • The authors’ simplified research-evaluation scoring system uses arbitrary weights that may overlook key aspects such as novelty.The passage also names value, surprise, and relevance as aspects that may be overlooked.Source: Limitations · p. 16

Conclusion

The authors report rapid growth in LLM-based review automation and anticipate reduced review time with similar or higher-quality data. They also note that few reviews actually used LLMs during their creation, and that trustworthy, useful AI-driven reviews still face challenges.

Conclusion

  • LLMs are likely to reduce review time while producing similar or higher-quality data in greater quantities than manual reviews.The conclusion says simpler reviews such as scoping reviews appear within current or near-future AI capabilities, but notes particular difficulty with quantitative meta-analyses comparing treatment effects.Source: Conclusion · pp. 16–17
  • Researchers are encouraged to disclose LLM use in academic papers, including prompts and affected text sections.The passage frames disclosure as a way to promote transparency and proper acknowledgment.Source: Conclusion · p. 17
  • Few systematic reviews using LLMs were identified, despite the rapid growth of LLM use in review automation.The authors describe AI-assisted reviews as still early-stage, while reporting that they are already yielding impressive results.Source: Conclusion · pp. 16–17
  • Trustworthy and useful AI-driven reviews still face technological and ethical challenges.The authors call for assessing the short- and long-term impact of integrations on review quality and considering ethical issues to maintain credibility and trust.Source: Conclusion · p. 17

Funding

The publication reports support from the National Center for Advancing Translational Sciences of the National Institutes of Health, a biomedical informatics grant, and a Smart-state Chair endowment.

Funding

  • The publication received partial support from the National Center for Advancing Translational Sciences of the National Institutes of Health under Grant UL1 TR001450.The stated support is partial.Source: Funding · p. 17
  • Dr. Scherbakov was supported by grant T15 LM013977 for Biomedical Informatics and Data Science for Health Equity Research.The passage identifies the program as SC BIDS4Health.Source: Funding · p. 17
  • The publication also received partial support from a Smart-state Chair endowment.The funding statement says the content does not necessarily represent the official views of the National Institutes of Health.Source: Funding · p. 17

Supplementary Appendix

The supplementary appendix provides LLM prompts for review screening and extraction, operational review definitions and stages, and benchmark-table captions.

Supplementary Appendix

  • Table S1 prompts request study location, automated review stages, LLM model, performance metrics, and time savings.They also ask how metrics were calculated and how many papers or abstracts informed them.Source: Table S1 · p. 25, Supplementary Appendix · p. 26
  • A review is defined as a publication synthesizing knowledge from other publications; listed types include systematic, scoping, meta-analysis, evidence-synthesis, umbrella, rapid, literature, and narrative reviews.The listed review stages range from question generation and searching through screening, extraction, assessment, synthesis, writing, and generating plots or tables.Source: Supplementary Appendix · p. 26
  • The prompts additionally ask for review type, authors’ conclusion, author ranks, affiliations, and funding, with output formatted as an R data.frame.The specified output format is data.frame(fld1='',fld2='',fld3='',...,fld12='').Source: Supplementary Appendix · p. 26
  • The appendix captions identify benchmarks for abstract screening (100 abstracts), full-text screening (30 PDFs), and full-text extraction (15 PDFs).The captions name Tables S2, S3, and S4, respectively.Source: Table S2 · p. 26, Table S3 · p. 26, Table S4 · p. 27

Time-saving and computational costs.

The authors estimate 334 person-hours saved and approximately $500 in GPT-4o costs. The listed time-saving components, however, sum to more than that total.

Time-saving and computational costs

Figure S6. Additional extracted categories.

Figure S6 brings together additional extracted categories on LLM use across literature-review tasks, including searching, screening, data extraction, appraisal, evidence synthesis, and writing. The reported findings describe potential efficiencies and capabilities alongside concerns about accuracy, reliability, validation, and human oversight.

Figure S6. Additional extracted categories

  • The supplementary figure summarizes citation type, authors’ opinions on LLM use, reported funding sources, and overall evidence quality.Table S7 provides the complete set of extracted categories; † marks categories verified by a human reviewer for extraction precision.Source: Figure S6 · p. 28, Table S7 · p. 29
  • ChatGPT was reported to generate quality drug-repurposing hypotheses and to have capacity to accelerate candidate analyses at minimal cost.The extracted excerpts describe regression models comparing Alzheimer’s disease risk in exposed and never-exposed individuals.Source: Figure S6. Additional extracted… · p. 30
  • LLMs were not ready to replace human experts in abstract screening, although GPTscreenR offered a user-friendly framework for supporting scholarly screening decisions.The extracted text identifies GPT-4 comparison with a consensus human reviewer and decisions involving both inclusions and exclusions.Source: Figure S6. Additional extracted… · p. 30
  • Current LLMs appraised evidence worse than humans, while human–AI collaboration may reduce the second human rater’s workload for PRISMA reporting and AMSTAR methodological rigor.The supplied text does not complete the qualification after “but not for,” so no further outcome is specified.Source: Figure S6. Additional extracted… · p. 31
  • LLMs may be useful as second or third reviewers, but caution is advised when integrating models such as GPT-4 into review workflows.The extracted study describes potential value while explicitly urging caution.Source: Figure S6. Additional extracted… · p. 32
  • Modern LLMs offer a promising path toward fully automatic meta-analysis, but further improvements are needed before this is ready.The supplied excerpts state the promise and qualification without providing numerical performance results.Source: Figure S6. Additional extracted… · p. 32

Figure S6. Additional extracted categories.

  • Mistral completed a review process in 17 hours versus 100 hours for human reviewers, but sometimes lacked human reviews’ analytical depth and attention to detail [17].The model was also described as having potential to standardize preliminary literature reviews.Source: Figure S6. Additional extracted… · p. 32
  • For LLaMA title-and-abstract screening, accuracy was 74.0 and precision was 82.5.Source: Figure S6. Additional extracted… · p. 32
  • MetaMate findings suggest LLMs could assist or potentially replace human data-extraction coders, with researchers and LLMs working synergistically to improve efficiency, cost-effectiveness, and quality [19].The report also described the system as significantly faster than human coders, taking only a few minutes.Source: Figure S6. Additional extracted… · p. 33
  • LLMs did not significantly improve human performance in text-simplified screening, while automated title-and-abstract screening remained promising but not significantly better.Source: Figure S6. Additional extracted… · p. 33
  • Participants judged LLMs inadequate for producing medical systematic reviews directly because they do not adhere to formal review methods and guidelines.Source: Figure S6. Additional extracted… · p. 33
  • Instruction fine-tuning aided screening, calibration enabled targeted recall, and combining both with zero-shot ensembles saved time versus state-of-the-art approaches (Cai, 2023).The evaluation used five standard test collections.Source: Figure S6. Additional extracted… · pp. 33–34
  • Approximately 87%, 73%, and 87% were reported for HIV drug-resistance question-answering accuracy, recall, and precision, respectively, without human feedback.The system answered Boolean, numerical, and list questions about HIV.Source: Figure S6. Additional extracted… · p. 34
  • LLMs were described as useful for basic, explicit systematic-review extraction but limited when handling nuanced, subjective criteria.The cited study called for oversight.Source: Figure S6. Additional extracted… · p. 35
  • Decision-tree models were described as simple and interpretable, significantly outperforming ChatGPT while achieving performance metrics competitive with a deep-learning model.Source: Figure S6. Additional extracted… · p. 35
  • The LitLLM authors advised disclosing system use and exercising caution about possible hallucinations [35].Source: Figure S6. Additional extracted… · p. 35
  • ChatGPT was reported to review papers and offer insights into their potential for acceptance or rejection.Source: Figure S6. Additional extracted… · p. 35
  • Embedding-space separability of relevant and irrelevant articles was reported to predict ML-screening performance, with work saved compared against random sampling at 95% recall.WSS @ 95% quantifies the percentage of work saved.Source: Figure S6. Additional extracted… · p. 35

Additional extracted categories

  • One project questioned whether its conclusions reflected its analysis or GPT’s intrinsic knowledge base.The passage presents this as an unresolved concern about the system’s basis for drawing conclusions.Source: Figure S6. Additional extracted… · p. 36
  • Less than 1% of relevant articles may be omitted by an automated screen using predetermined criteria, a rate similar to human experts [40].The study also reported that AI can broaden the effectiveness of systematic-review search strategies.Source: Figure S6. Additional extracted… · p. 36
  • ChatGPT was described as a viable option for automating systematic-review processes, but integration requires careful consideration by developers.The source notes that standard classifier evaluation techniques were used, with recall and precision based on screening-task outcomes.Source: Figure S6. Additional extracted… · p. 36
  • ChatCite-generated literature summaries were reported as directly usable for drafting literature reviews.Source: Figure S6. Additional extracted… · p. 36
  • LLMs demonstrated proficiency in prioritizing candidate studies for abstract screening with the proposed QA framework [44].Source: Figure S6. Additional extracted… · p. 36
  • FACTOOL powered by GPT-4 significantly outperformed self-check baselines in scientific literature review [45].Source: Figure S6. Additional extracted… · p. 36
  • ChatGPT was reported to have potential for generating effective Boolean queries for systematic-review searches [47].The passage also reports that 60 provided feedback for further improvement.Source: Figure S6. Additional extracted… · p. 37
  • Precision was 98.0 for GPT-4 publication searching, and the best approach was described as viable and similarly effective to the available human-based approach [49].Source: Figure S6. Additional extracted… · p. 37

Figure S6. Additional extracted categories.

  • The Goldilocks study reported reduced BERT runtime and described epoch searching as laborious, recommending task-appropriate pretrained BERT-like backbones for TAR.The authors characterized this as a simpler, more effective route to improved effectiveness in TAR tasks.Source: Figure S6. Additional extracted… · p. 37
  • Aydın, 2022 [53] reported significant Ithenticate matches in paraphrased parts of a ChatGPT-generated literature review.The entry characterized the results as promising but noted this plagiarism concern.Source: Figure S6. Additional extracted… · p. 38
  • Jafari, 2024 [54] noted that GPT-4-assisted SLR selection improved on manual approaches, but its full potential required further refinement and expansion to later stages.The passage says this potential would likely be reached through ongoing refinement and expansion into subsequent stages of the SLR process.Source: Figure S6. Additional extracted… · p. 38
  • The cited passage says ChatGPT’s use in academic reviews offers advantages, identifying rapid information acquisition as a benefit, without specifying any further advantage in the supplied excerpt.Kim, 2024 noted ethical concerns.Source: Figure S6. Additional extracted… · p. 38
  • BERT-based rankers outperformed the current state-of-the-art for screening prioritisation.A separate entry reported that SciBERT-based models performed better at identifying scientifically sound work.Source: Figure S6. Additional extracted… · pp. 38–39
  • LLooM was reported to improve topic-model quality and coverage and help expert analysts uncover insights, including on familiar datasets.The entry described these as generated concepts and insights.Source: Figure S6. Additional extracted… · p. 38
  • A fashion-research entry described ChatGPT’s potential and methodological limitations, and proposed a prototype for faster, reliable discourse aligned with the study objective.It reported no specific quantitative metrics and based performance assessment on subjective author evaluation and feedback.Source: Figure S6. Additional extracted… · p. 38
  • GPT-4 was reported to streamline the review process and improve efficiency while maintaining high accuracy and quality.Domain experts scored generated related-work sections on fluency, coherence, relevance, factuality, usefulness, writing style, and overall quality.Source: Figure S6. Additional extracted… · pp. 38–39
  • Review effort fell by 17.85% to 59.04% versus no-feedback baseline at a fixed recall target.The reported range came from a relevance-feedback method.Source: Figure S6. Additional extracted… · p. 39
  • A model trained on a limited number of scientific domains generalized to new domains in cross-domain experiments despite differences in aspect representation [66].The reported differences in aspect representation occurred across domains.Source: Figure S6. Additional extracted… · p. 39
  • GPT-4 achieved 93.0 accuracy in title and abstract screening in a biomedical literature review study [67].The study also reported that ChatGPT enhanced efficiency and accuracy across several review stages and a time saving of approximately 50.Source: Figure S6. Additional extracted… · p. 39
  • ChatGPT showed moderate agreement with human gold-standard risk-of-bias assessments, but its suboptimal performance did not support recommending it for assessment [68].The passage describes the disagreement as minor but still cautions against recommendation.Source: Figure S6. Additional extracted… · p. 40
  • A survey of LLM risk-of-bias assessment reported substantial accuracy and consistency in evaluating randomized clinical trials, supporting their potential as tools in systematic reviews.The source presents LLMs as supportive tools.Source: Figure S6. Additional extracted… · p. 40
  • The results suggest that ChatGPT used as a tool for real-time evidence generation is not yet accurate and feasible; therefore, researchers should be cautious about using it.Source: Figure S6. Additional extracted… · p. 40
  • ChatGPT achieved precision=77.0 for searching publications in one evaluation.Source: Figure S6. Additional extracted… · p. 40

Additional extracted categories

  • The instruction-tuned ICL framework reported state-of-the-art results across EBM-NLP datasets, while its ICL-based version produced comparable results.The passage describes these findings as applying to all versions of the EBM-NLP datasets.Source: Figure S6. Additional extracted… · p. 41
  • Accuracy=54.0 was reported for ChatGPT 3.5 searching, while GPT-4 title-and-abstract screening had Accuracy=67.0.These are reported for different models and tasks.Source: Figure S6. Additional extracted… · p. 41
  • ChatGPT was described as promising for article screening, with high sensitivity and workload benefits; one report said it completed screening within an hour.The supplied excerpts do not specify the magnitude of the workload benefit.Source: Figure S6. Additional extracted… · p. 41
  • ChatGPT may help identify novel urogynecology research ideas, but its suggestions were described as inconsistently accurate and requiring careful auditing.The passage defines novel ideas as topics without prior systematic reviews or publications.Source: Figure S6. Additional extracted… · p. 41
  • A proposed framework was reported to improve accuracy and efficiency, while LLM-assisted data extraction was described as potentially enhancing both.The data-extraction passage frames the enhancement as potential rather than a quantified result.Source: Figure S6. Additional extracted… · pp. 41–42
  • The findings counsel caution in systematic reviews: some tasks may work under specific conditions, but the case-study passage says an independent human reviewer remains required.The cited case-study excerpt bases this requirement on agreement with a human reviewer.Source: Figure S6. Additional extracted… · pp. 41–42
  • Other reported uses included generating PICO-driven orthodontic queries, identifying scientific research gaps, and prioritizing drug-repurposing candidates from internet-based insights.The orthodontics passage notes the role of optimal prompts and configurations; the research-gap and drug-repurposing passages describe potential applications.Source: Figure S6. Additional extracted… · p. 42
  • Lozano, 2024 [91] reported publication-search precision of 22.4 for GPT-3.5, alongside tasks spanning screening, data extraction, and evidence synthesis.The excerpt does not specify a unit or further metric details for the precision value.Source: Figure S6. Additional extracted… · p. 43
  • A review-automation system was described as understanding and representing review information rather than merely copying and pasting it, with performance reported as enhanced compared with comparable tools.The supplied excerpts do not provide quantitative performance values.Source: Figure S6. Additional extracted… · p. 43
  • The review reported ChatGPT taking less time than the control arm (57 3 seconds versus 644 15 seconds), although the extracted time fragments leave decimal placement unclear.It described Perplexity.AI as able to perform only a limited review of medical literature and advised caution and human quality assurance for open AI programs.Source: Figure S6. Additional extracted… · p. 43
  • Evaluation of AI-generated literature reviews found inconsistent citation accuracy across different AI tools, despite promising results [95].The excerpts also describe discrepancies between Claude and Claude-Instant results, but do not provide their values.Source: Figure S6. Additional extracted… · pp. 43–44

Figure S6. Additional extracted categories

  • Roberts, 2023 [97] reported that ChatGPT could help automate medical-literature appraisal and identify accurately reported research.The listed study compared ChatGPT with human evaluators in assessing medical literature.Source: Figure S6. Additional extracted… · p. 44
  • The final system is a proof of concept showing that evidence-table generation can be semi-automated, representing a step toward semi-automating evidence-table generation.Source: Figure S6. Additional extracted… · p. 44
  • 65% fewer abstracts were screened by the second reviewer in a pBERT-supported workflow that sent conflicting decisions for human review.The reported reduction was in the number of abstracts screened by the second reviewer.Source: Figure S6. Additional extracted… · pp. 44–45
  • ChatGPT showed strong research-productivity potential but had minor impact on research-framework development and data analysis in the study listed as Khlaif, 2023 [102].The findings also noted that ChatGPT can be used in academic writing.Source: Figure S6. Additional extracted… · p. 45
  • 42.0% accuracy was reported for ChatGPT nephrology-literature searching, and the authors cautioned against using it as the sole reference-finding resource.They recommended checking references against reliable sources for existence, relevance, and correctness.Source: Figure S6. Additional extracted… · p. 45
  • ChatGPT conclusions resembled meta-analysis findings when fewer drugs were compared, but did not match meta-analysis or network-meta-analysis results when more drugs were compared [104].The cited comparison concerned the effectiveness and tolerance of systemic therapies for moderate-to-severe plaque psoriasis.Source: Figure S6. Additional extracted… · p. 45
  • Other extracted accounts described potential to reduce review workload and time, support lumbar-spinal-stenosis decisions, and facilitate ophthalmology work, while stressing careful validation and accuracy and bias concerns.The lumbar-spine account framed ChatGPT as decision support; the ophthalmology account also mentioned manuscript development and peer review.Source: Figure S6. Additional extracted… · pp. 45–46

Figure S6. Additional extracted categories.

Screening and workload

  • 6/390 (1.53%) articles were misclassified by the human-only process, while the hybrid workflow matched human-only inclusion/exclusion decisions for the other 384 articles.The figure describes the hybrid workflow as improving case-study accuracy and reducing human workload.Source: Figure S6. Additional extracted… · p. 50

Process efficiency

  • At least four times faster, the generative-AI-based process was reported to improve speed and comprehensiveness with human intervention and review.The comparison was with the traditional process.Source: Figure S6. Additional extracted… · p. 50

Searching for publications

  • 70.0 accuracy was reported for ChatGPT searching for publications in a literature-review case study.The study’s conclusion described ChatGPT as supporting literature scanning and evaluation with relatively accurate answers to specific questions.Source: Figure S6. Additional extracted… · p. 50

Data extraction

  • 92.0 accuracy was reported for GPT-3.5 data extraction in a meta-analysis architecture for automatic outcome-data extraction.The extracted category identifies the task as data extraction and the system as GPT/ChatGPT.Source: Figure S6. Additional extracted… · p. 50

Literature-review tools

  • 3.9/5 overall experience, 3.8/5 insight quality, and 4.1/5 search results were reported for SciSpace Literature Review.The passage describes the tool as effective for finding relevant scientific literature and providing insights.Source: Figure S6. Additional extracted… · p. 51

Writing and editorial support

  • ChatGPT 3.5 was described as useful for reviewing and refining research, correcting errors, and creating new codes.Another passage characterizes AI as a supportive, complementary tool rather than a replacement for human judgment and expertise in editorial processes.Source: Figure S6. Additional extracted… · p. 51

Figure S6. Additional extracted categories.

  • Guo described LLM support for idea generation, contextually relevant responses, partial literature-review assistance, paper design, editing, and proofreading.Source: Figure S6. Additional extracted… · p. 51
  • ChatGPT publication classification had Accuracy=88, and its filtering and categorization of relevant articles was described as saving time and effort [149].Source: Figure S6. Additional extracted… · p. 51
  • SciMine was reported to significantly outperform existing methods and achieve the best-reported results in the literature.Source: Figure S6. Additional extracted… · p. 51
  • The smooth-operator model was reported to improve existing systematic-review search queries without changing their syntactic or semantic structure [148].Source: Figure S6. Additional extracted… · p. 51
  • Researchers in a public-administration study equally reported advantages and disadvantages of using GPT-4 in collaboration on author-written text [151].Source: Figure S6. Additional extracted… · p. 52
  • BERT-based scoping searches had Precision=73.3, with results suggesting device-specific retrieval across databases while screening fewer documents than manual searches, despite risks and limitations.The results also described vast automation potential.Source: Figure S6. Additional extracted… · p. 52
  • Forefront was described as the best overall performer despite high false positives, followed by Bard and Claude.The passage notes that high false positives lowered specificity and precision.Source: Figure S6. Additional extracted… · p. 53
  • 90.0 was the reported accuracy for ChatGPT data extraction in a clinical meta-analysis study.The study’s conclusion said LLMs could support more streamlined, transparent, and reproducible clinical research.Source: Figure S6. Additional extracted… · p. 53
  • Other entries described affordances for scoping research questions, faster primary-study selection, fluent literature-review summaries needing improvement, and a need for stronger summary-quality metrics.The scoping affordances included visualizing the research space, narrowing the search strategy to available resources, and retaining search context; deep-learning-based selection was described as accelerating scanning and identification.Source: Figure S6. Additional extracted… · pp. 53–54
  • 100.0 was the reported recall for BERT title-and-abstract screening in a complex systematic-review study.The study was identified as Hasny, 2023 [165].Source: Figure S6. Additional extracted… · p. 53
  • Fine-tuned publicly available models achieved recall values of at least 90% while reducing human workload, without expensive pretraining.The reported approach was characterized as promising across accuracy, precision, recall, and F1 metrics.Source: Figure S6. Additional extracted… · p. 53
  • RefAI was reported to improve literature recommendation and summarization over existing tools while addressing fabricated papers, metadata inaccuracies, restricted recommendations, and poor reference integration.The evaluation was based on expert ratings averaged across tools.Source: Figure S6. Additional extracted… · p. 54
Loading 2409.04600v1…