Source-linked AI summary
Open Source Software for Efficient and Transparent Reviews
Rens van de Schoot, Jonathan de Bruin, Raoul Schram, Parisa Zahedi, Jan de Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, Albert Harkema, Joukje Willemsen, Yongchao Ma, Qixiang Fang, Sybren Hindriks, Lars Tummers, Daniel Oberski
TL;DR
Screening the literature for systematic reviews is labor intensive, prompting narrow searches. This paper presents ASReview, an open-source active-learning pipeline, and simulation studies show that it can identify most eligible studies after screening a much smaller fraction of records.
Problem
Screening the entire research literature is too labor intensive, so scholars often develop narrow search strategies for systematic reviews.
Method
The paper presents ASReview, an open-source machine-learning pipeline that applies active learning to screen titles and abstracts.
Results
95% of eligible studies were found after screening only 8% to 33% of studies, while 70% to 100% of relevant abstracts were found after reading 10%.
Takeaways & Limitations
ASReview can substantially reduce the amount of screening required while maintaining high-quality reviewing.
Takeaways & Limitations
ML-based screening systems, including ASReview, have remaining drawbacks and do not provide users with an accurate application-specific estimate of the system’s error rate.
Abstract
from arXiv · showhide
To help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool (ASReview) to accelerate the step of screening titles and abstracts. For many tasks - including but not limited to systematic reviews and meta-analyses - the scientific literature needs to be checked systematically. Currently, scholars and practitioners screen thousands of studies by hand to determine which studies to include in their review or meta-analysis. This is error prone and inefficient because of extremely imbalanced data: only a fraction of the screened studies is relevant. The future of systematic reviewing will be an interaction with machine learning algorithms to deal with the enormous increase of available text. We therefore developed an open source machine learning-aided pipeline applying active learning: ASReview. We demonstrate by means of simulation studies that ASReview can yield far more efficient reviewing than manual reviewing, while providing high quality. Furthermore, we describe the options of the free and open source research software and present the results from user experience tests. We invite the community to contribute to open source projects such as our own that provide measurable and reproducible improvements over current practice.
3 Utrecht University Library, Utrecht University, The Netherlands
ASReview is an open-source, free machine-learning pipeline using active learning to accelerate transparent screening of titles and abstracts for systematic reviews and meta-analyses. Simulations and user testing indicate substantial efficiency gains while retaining transparency and good usability.
- Motivation: Manual screening is error-prone and time-intensive, making adequate review impossible when literature growth exceeds available review time.
- Approach: ASReview uses active learning to prioritize relevant records and reduce the number requiring manual screening.The system is designed around researcher interaction rather than producing only a trained model.
- Software: ASReview is free, open source, transparent, extensible, supports multiple machine-learning models, and includes benchmark mode for algorithm comparison.
- Results: 83% average work saved over sampling at 95% recall, ranging from 67% to 92%.Screening 8% to 33% of studies found 95% of eligible studies, while screening 10% of abstracts found 70% to 100% of relevant abstracts.
- User experience: Participants (N=11) rated ASReview 7.9 (SD = 0.9) on a one-to-ten scale.
- Evaluation and limitations: The authors report that ASReview’s default parameters performed well on average across the examined real-world systematic-review applications.They also note that drawbacks of machine-learning screening systems remain, including uncertainty about application-specific error rates.
Appendix
The appendix surveys software tools that support systematic reviews and defines the criteria used to compare them. It includes tools with varying support for open source access, machine learning, and active learning.
- Table A1 overviews software tools supporting systematic reviews.The overview is based on Harrison et al. (2020).
- Listed tools include DBPedia5, DistillerSR, EROS, GAPScreener, JBI-SUMARI, Lingo 3d, Rayyan, RobotAnalyst, and others.The appendix also lists Pimiento, ReVis, RevMan, SESRA, SLR-Tool, SLRTOOL, SRDB.PRO, SyRF, and Systematic Review Accelerator.
- The comparison considers open-source availability, open-source code URLs, machine learning, and active learning.The notes define Open Source as publicly available application source materials and distinguish machine learning and active learning by their goal of reducing abstracts needed for screening.