Source-linked AI summary
Artificial intelligence to automate the systematic review of scientific literature
José de la Torre-López, Aurora Ramírez, José Raúl Romero
TL;DR
Preparing an SLR is costly and error-prone when researchers must process large literature collections, motivating AI support for repetitive tasks. The paper surveys AI techniques, tools, supported tasks, and human involvement across 34 primary studies. It finds strongest interest in machine-learning support for paper screening, while planning and reporting remain less automated.
Problem
Preparing an SLR requires time-intensive searching, screening, extraction, and reporting across potentially thousands of papers, while existing studies do not comprehensively cover AI across the whole process.
Method
The paper conducts a survey of AI-based SLR automation, analyzing 34 primary studies for supported phases, techniques, methods, tools, evaluation, and human involvement.
Results
The survey finds clear interest in machine learning for paper screening, while ontology- and NLP-based approaches for semantic information are less numerous.
Takeaways & Limitations
Future efforts should support planning and reporting, whose tasks are more difficult to automate, including possible advances in automatic writing through deep-learning approaches.
Takeaways & Limitations
Human involvement remains necessary because current AI techniques lack humans’ holistic view of the SLR process.
Abstract
from arXiv · showhide
Artificial intelligence (AI) has acquired notorious relevance in modern computing as it effectively solves complex tasks traditionally done by humans. AI provides methods to represent and infer knowledge, efficiently manipulate texts and learn from vast amount of data. These characteristics are applicable in many activities that human find laborious or repetitive, as is the case of the analysis of scientific literature. Manually preparing and writing a systematic literature review (SLR) takes considerable time and effort, since it requires planning a strategy, conducting the literature search and analysis, and reporting the findings. Depending on the area under study, the number of papers retrieved can be of hundreds or thousands, meaning that filtering those relevant ones and extracting the key information becomes a costly and error-prone process. However, some of the involved tasks are repetitive and, therefore, subject to automation by means of AI. In this paper, we present a survey of AI techniques proposed in the last 15 years to help researchers conduct systematic analyses of scientific literature. We describe the tasks currently supported, the types of algorithms applied, and available tools proposed in 34 primary studies. This survey also provides a historical perspective of the evolution of the field and the role that humans can play in an increasingly automated SLR process.
1 Introduction
AI-based SLR automation addresses the time, effort, and error risks of reviewing extensive scientific literature. This survey examines AI techniques, supported tasks, human involvement, and 34 primary studies.
- SLRs require searching, screening, extracting, and synthesizing literature, but extensive collections make these activities time-consuming and potentially error-prone.A rigorous methodology can mitigate potential biases and errors.
- Existing SLR tools support importing search results, applying inclusion criteria, and analyzing author and citation metadata.
- The survey focuses on AI-driven automation of SLR tasks while preserving human involvement in the process.
- Previous secondary studies examined general automation, individual AI techniques, or particular SLR tasks rather than the whole AI-supported SLR process.
- The study asks which SLR phases AI automates, which techniques support them, and how humans remain involved.
- The authors identify 34 primary studies from more than 9,000 references and analyze their purposes, methods, inputs, outputs, evaluation, and corpora.
2 Background
An SLR is a rigorous secondary study that synthesizes scientific literature through a defined, evidence-based methodology. It offers broad research overviews but requires substantial time and careful planning.
- An SLR rigorously unifies and analyzes scientific literature to synthesize knowledge, discuss proposals, and identify trends.
- Its evidence-based methodology includes defining research questions and using a replicable procedure to find primary studies for information extraction.
- SLRs provide authors with in-depth topic knowledge and readers with comprehensible, up-to-date overviews and access to key studies.
- SLRs also require substantial time and can be difficult when evaluating the quality of primary studies.
- Planning phase: Planning defines the review’s need, research questions, protocol, search procedure, and information sources.
3 Methodology
The survey combines automatic and manual searching with staged screening, eligibility criteria, and structured data extraction. It ultimately analyzes 34 primary studies selected from thousands of references.
- The methodology follows defined steps to retrieve papers and extract information from them.
- The search combines automatic queries across five databases with manual searching and uses general automation terms to reduce technique-specific bias.
- 9,027 references yielded 44 candidates after title-and-abstract inspection, while backward snowballing added 6 papers to reach 50 candidates.
- Candidates were screened using language, accessibility, peer-review, scope, and explicit AI-based SLR-automation criteria.
- 34 primary studies were selected, comprising 32% conference papers and 68% journal papers.
- Each study was analyzed with a data-extraction form covering study metadata, AI categories, human involvement, experimental evaluation, and reproducibility.
4 AI techniques for SLR automation
The paper organizes AI techniques for SLR automation by the three SLR phases: planning, conducting, and reporting.
- AI techniques are organized according to the planning, conducting, and reporting phases of an SLR.
4.1 AI techniques for the planning phase
AI techniques in the planning phase support early topic mapping and analysis of how review activities are organized. They use clustering and process mining to inform scope assessment, prioritization, and resource decisions.
- Topic mapping: Unsupervised document clustering groups similar papers from titles and abstracts without requiring predefined topic labels.Lingo3G can assign papers to multiple clusters and generate hierarchical topic classifications for reviewer mapping.
- Topic mapping: The clustering approach was evaluated by comparing automatically generated topics with topics from previous health-research scoping reviews.
- Process analysis: Process mining analyzes event logs to identify relevant SLR activities and their temporal distribution.Dependency graphs reveal task-sequence patterns, while fuzzy mining abstracts review models by excluding less relevant activities or characteristics.
- Process analysis: These planning-phase analyses can support decisions about available resources and task prioritization before the review proceeds.
4.2 AI techniques for the conducting phase
The conducting phase receives the greatest AI attention, especially for selecting primary studies. Methods combine text representations, classifiers, active learning, visualization, clustering, ontologies, and sentence-level extraction to support different conducting tasks.
- Phase overview: 59% of the primary studies address conducting-phase tasks, with primary-study selection supported by 18 papers.Machine learning is the most widely used AI branch in this phase, often combined with NLP and text mining.
- Paper selection: ML-based paper selection extracts textual features and trains classifiers to distinguish papers included in or excluded from an SLR.Titles and abstracts are commonly processed with NLP and text-mining methods.
- Paper selection: Supervised selection methods include neural networks, SVM, logistic regression, Naive Bayes, and ensemble classifiers trained on labelled literature.Representations include bag-of-words, TF-IDF features, topic features, and specialized medical terms.
- Paper selection: Active learning reduces labelling demands by querying a human reviewer during training, often focusing on uncertain predictions.FASTREAD uses SVM-based configurations with alternative training, querying, stopping, and class-balancing strategies.
- Candidate inspection: Visualization and clustering help reviewers inspect candidate-paper quality through document maps, content similarity, citation relationships, and grouped papers.Document maps use cosine similarity over bag-of-words representations weighted by TF-IDF, with k-means clustering.
- Data extraction: Ontology-based and learning-based extraction methods represent paper concepts or identify sentences describing treatment comparisons and experimental results.Ontology systems can generate PICO-based descriptions, while SVM classifiers identify relevant sentences and entities.
4.3 AI techniques for the reporting phase
AI support for the reporting phase remains limited, covering automatic report generation and report evaluation. Proposed methods generate study summaries or questions about whether reports reflect relevant study information.
- Reporting scope: Current AI approaches support only two reporting tasks: writing the SLR report and evaluating it.
- Report generation: Deep-learning and NLP systems have been proposed to generate summaries of selected primary studies as a starting point for SLR reports.The approach is conceptual and combines NLP with deep neural networks.
- Report evaluation: NLP-generated questions can evaluate whether an SLR report reflects study subjects, aims, contributions, methods, datasets, results, strengths, and limitations.The proposed system uses the LBJ named-entity tagger.
4.4 Previous analyses of the field and tool evaluations
Previous analyses examine SLR-support tools, AI techniques, workload reduction, development principles, and practical tool use. Their scope varies by domain, task, and whether they analyze scientific literature or accessible tools.
- Field perspective: Other studies compare existing tools or analyze literature on AI for SLR automation, complementing the field's historical perspective.
- Tool analyses: A software-engineering tool mapping study found that text mining, including visualization-integrated methods, appeared in 57% of analyzed tools.The study analyzed 14 papers and included tools evaluated on software-testing and cost-estimation corpora.
- Tool analyses: An evidence-based-medicine overview included Abstrackr as its only AI-based tool and mentioned other techniques without analyzing them in depth.
- ML analyses: Two studies of machine-learning methods for paper selection reported workload reductions varying from 30–70% across experiments.They also identified challenges including imbalanced data, where relevant studies form a small proportion of candidates.
- Field principles: Proposed automation principles emphasize efficiency, coverage of multiple tasks, method flexibility, replicability, and open-source resources.
- Practical guidance: A practical guide for medical SLRs analyzed 13 tools from the SR Toolbox catalogue, focusing on their scope, strengths, and limitations.It omitted scientific literature unless a supporting tool was available.
5 Analysis of current trends
AI-based SLR research covers all review phases, but paper selection dominates current work. Machine learning—especially active learning and SVM classification—predominates, while human involvement and broader task coverage remain important considerations.
- SLR phases currently supported: All SLR phases have been addressed by at least one primary study, but conducting—especially automatic primary-study selection—is the most studied phase.Planning and reporting remain unsupported by complete automation reviews despite some AI applications in those phases.
- SLR phases currently supported: Primary-study selection has continued attracting attention since the first paper, with five new methods proposed during the last four years.Supporting tools have also been evaluated.
- Selection of AI techniques: Machine learning is the most frequent AI area, using supervised and active learning for classification and less often unsupervised learning for clustering.Active learning is the reference approach for paper selection because it models the cost of obtaining labels.
- Selection of AI techniques: SVM appears in 13 out of 17 classification papers, while NB, DT, LR and neural networks are used less often.The small number of papers limits conclusions about why particular algorithms were chosen.
- Selection of AI techniques: Paper-selection methods usually extract features from titles and abstracts, whereas only two methods add citations or cross-references.The latter develop visualization and clustering mechanisms to assist analysis.
- Selection of AI techniques: Most AI methods make a binary selection decision rather than ranking papers or simulating criteria-guided evaluation.A few methods instead produce rankings that distinguish relevant, irrelevant and borderline papers.
- Human intervention: Among the reviewed approaches, 68% are classified as fully automated and 32% as semi-automated.Fully automated approaches may still rely on a corpus previously created or annotated by humans.
6 Open issues and challenges
AI-based SLR automation remains concentrated on paper selection, while other tasks, techniques, domains, and evaluation settings remain less developed. The authors also identify trust, interpretability, benchmarking, and researcher oversight as continuing challenges.
- Task coverage: AI-based SLR research is strongly biased toward paper selection, while other tasks receive less mature support.AI-driven writing tasks such as formulating research questions, defining criteria, and reporting results remain largely unexplored.
- Algorithmic coverage: Optimisation and search techniques have not yet been applied to SLR task resolution, despite their potential for planning and resource allocation.The authors suggest prioritising databases or assigning papers to reviewers according to their skills.
- Trust and interpretability: Limited interpretability creates a trust challenge for researchers from diverse disciplines who are not necessarily AI experts.Rule-based systems and small decision trees have been barely explored, while explainable methods could complement black-box solutions.
- Evaluation scope: Most proposals have been validated in medicine or computing, and broader evaluation is constrained by the limited availability of replicable benchmarks.Selection decisions are not always explicitly linked to inclusion and exclusion criteria, complicating performance comparisons across fields.
- Human oversight: Researchers remain responsible for methodological safeguards and analytical work even when tools accelerate SLR tasks.The authors emphasise clear protocols, replication checkpoints, paper summaries, and trend analysis.
7 Conclusions
The survey finds that AI, especially machine learning, is being used most extensively to support paper screening in SLRs, with ontologies and NLP addressing semantic information less frequently. It concludes that planning and reporting require further support because they are more difficult to automate.
- Conclusions: AI has been effective for automating costly and repetitive SLR tasks, especially paper screening among thousands of candidate papers.The survey reports clear interest in applying machine learning to identify relevant works.
- Conclusions: Ontologies and NLP support semantic information in SLRs, but studies in these areas remain far less abundant than paper-screening research.The conclusion contrasts their limited coverage with the stronger concentration on machine-learning approaches to screening.
- Conclusions: Future work should support SLR planning and reporting, whose tasks are more difficult to automate.The authors also anticipate progress in automatic writing following conceptual deep-learning approaches.
- Conclusions: Detailed literature-search results and tool analyses are available from the survey website.The resource is provided at https://www.uco.es/kdis/ai4slr/survey.
Competing interests.
The paper declares that the authors have no competing interests.
- Competing interests: The authors declare no competing interests.The declaration appears in the competing-interests statement.
- Competing interests: The listed contributions cover conceptualization, methodology, analysis, writing, supervision, and funding acquisition.All authors are credited for conceptualization and methodology.