PaperlayerView exact revision on arXiv

Source-linked AI summary

Artificial intelligence to automate the systematic review of scientific literature

José de la Torre-López, Aurora Ramírez, José Raúl Romero

arXiv:2401.10917v1cs.IRcs.AI

TL;DR

Systematic literature reviews require time-consuming searching, screening, and information extraction across large paper collections, while existing secondary studies cover automation broadly or focus on a specific AI technique or task. This paper surveys 34 primary studies of AI for SLR and finds particular interest in machine-learning support for paper screening, with much less work on ontologies and NLP for semantic information.

  • Problem

    Existing secondary studies considered automation broadly or focused on a particular AI technique or SLR task, rather than surveying AI across tasks.

    Source: §1 · p. 3

  • Method

    The paper surveys 34 selected studies of AI techniques for SLR and examines support across tasks and human involvement.

    Source: §1 · p. 3, §3.1 · p. 6

  • Results

    AI, especially ML, is most prominent for paper screening; ontologies and NLP address semantic information, but studies of these areas are far fewer.

    Source: §7 · p. 21

  • Takeaways & Limitations

    Planning and reporting remain priorities for future support because their tasks are more difficult to automate.

    Source: §7 · p. 21

    Current AI techniques still lack humans’ holistic view of the SLR process, so the human role should not be diminished.

    Source: §1 · p. 3

Abstract

from arXiv · show

Artificial intelligence (AI) has acquired notorious relevance in modern computing as it effectively solves complex tasks traditionally done by humans. AI provides methods to represent and infer knowledge, efficiently manipulate texts and learn from vast amount of data. These characteristics are applicable in many activities that human find laborious or repetitive, as is the case of the analysis of scientific literature. Manually preparing and writing a systematic literature review (SLR) takes considerable time and effort, since it requires planning a strategy, conducting the literature search and analysis, and reporting the findings. Depending on the area under study, the number of papers retrieved can be of hundreds or thousands, meaning that filtering those relevant ones and extracting the key information becomes a costly and error-prone process. However, some of the involved tasks are repetitive and, therefore, subject to automation by means of AI. In this paper, we present a survey of AI techniques proposed in the last 15 years to help researchers conduct systematic analyses of scientific literature. We describe the tasks currently supported, the types of algorithms applied, and available tools proposed in 34 primary studies. This survey also provides a historical perspective of the evolution of the field and the role that humans can play in an increasingly automated SLR process.

1 Introduction

The paper surveys AI techniques for automating systematic literature review (SLR) tasks, examining the tasks and methods involved, how humans participate, and open challenges.

1 Introduction

  • An SLR uses a rigorous methodology to find relevant papers, extract information, and present findings, but searching, screening, and extracting information can be costly and error-prone.A literature review can provide a comprehensive and historical overview of a topic, while rigorous methods can mitigate potential biases and errors.Source: §1 · p. 2
  • SLR tools support importing search results, screening papers, and visualizing author and citation metadata; automated approaches can save time and resources, though practical use may be limited by learning curves and limited evaluation.Source: §1 · p. 2
  • AI-based SLR automation dates to a 2006 neural network for selecting primary studies, followed by text-mining approaches and more recent ML and NLP methods.The authors caution that humans should retain a role because current AI techniques lack a holistic view of the SLR process.Source: §1 · pp. 2–3
  • The survey asks which SLR phases AI automates, which techniques support them, and how much humans remain involved, addressing gaps in broader or task- and technique-specific prior reviews.The authors aim to explain AI concepts applicable across the SLR process and examine human involvement in semi-automated reviews.Source: §1 · p. 3
  • Analysis of 34 primary studies selected from more than 9,000 references finds uneven task coverage, continued use of early ML techniques, newer approaches with greater human involvement, and challenges including reproducibility.The authors examine methods’ inputs, outputs, algorithmic choices, and experimental evaluations; they also identify unsupported tasks and AI techniques not yet explored.Source: §1 · p. 3

2 Background

A systematic literature review (SLR) synthesizes and critically examines scientific literature through an evidence-based, replicable methodology. It proceeds through planning, conducting, and reporting phases, offering useful overviews while requiring substantial time and careful quality assessment.

2 Background

  • An SLR synthesizes current knowledge, critically discusses proposals, and identifies trends through evidence-based research with defined questions and a replicable search for primary studies.Information is then extracted from the relevant papers.Source: §2 · pp. 3–4
  • SLRs let authors study topics in depth and give readers up-to-date field overviews, but they take time and can be difficult to replicate when methodology is unclear.Graduate students are particularly encouraged to undertake SLRs; challenges include evaluating primary-study quality and replicability.Source: §2 · p. 4
  • Kitchenham and Charters divide the SLR process into planning, conducting, and reporting, beginning with research questions, a protocol, and predefined selection and extraction guidance.Questions may use PICO or SPICE structures.Source: §2 · p. 4
  • During conducting, reviewers search databases and other sources, screen studies, extract information, and synthesize data; reporting presents findings and checks the report’s completeness and quality.Searches may use supporting methods, grey literature and snowballing; selection applies inclusion and exclusion criteria, while reporting checklists are available.Source: §2 · p. 4

3 Methodology

The methodology combines database searching and manual snowballing, screens candidate papers against eligibility criteria, and extracts study characteristics through a reviewer-check process.

3.1 Search strategy

  • The automatic search queried five databases using general automation terms to avoid bias toward specific techniques and unwieldy search strings.Searches covered titles, keywords, and abstracts, and the query was adapted to each source; manual searching used backward snowballing.Source: §3.1 · p. 5, Figure 2 · p. 5
  • The search returned 9,027 references, removed 2,417 duplicates, shortlisted 44 through title-and-abstract screening, and added six via backward snowballing to reach 50 candidates.Eligibility criteria excluded papers that were not in English, lacked accessible full text, or lacked peer-review evidence; 34 studies meeting the content criteria were selected.Source: §3.1 · pp. 5–6
  • Among the 34 selected studies, 32% were conference papers and 68% journal papers; 57% of journal papers (13) appeared in the last five years.The first study appeared in 2006, and no other proposals appeared until 2009; Fig. 3 presents the yearly distribution.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 Data extraction

  • Each study was reviewed by one author, with a second reviewer consulted when uncertain, using a form to capture study metadata and AI-approach characteristics.The form recorded SLR phases and tasks, AI areas and techniques, human involvement, study type, empirical corpora and evaluation metrics, and public availability of algorithms, datasets, and tools.Source: §3.2 · pp. 6–7

4 AI techniques for SLR automation

The section organizes AI techniques for SLR automation by three phases: planning, conducting, and reporting.

Planning

  • Planning is covered as the first SLR phase in Section 4.1.Source: §4 · p. 7

Conducting

  • Conducting is covered as the second SLR phase in Section 4.2.Source: §4 · p. 7

Reporting

  • Reporting is covered as the third SLR phase in Section 4.3.Source: §4 · p. 7

4.1 AI techniques for the planning phase

Planning-phase AI can help scope a review by clustering papers and help allocate resources by mining records of SLR activities. The survey describes both approaches and their methods.

4.1 AI techniques for the planning phase

  • Lingo3G3 clusters papers by title and abstract for scoping reviews, allowing overlapping and hierarchical clusters that reviewers can map to concepts.It was evaluated against topics assigned by manual review in previous health-institution scoping reviews.Source: §4.1 · p. 7
  • Pham et al. applied process mining to event logs from 12 simulated manual SLR processes to extract task assignments, timelines, and person-hour effort.The approach is intended to inform understanding of effort and the usual organization of SLR activities.Source: §4.1 · p. 7
  • Heuristic mining identifies frequent activities and task sequences, while fuzzy mining abstracts review models by simplifying less significant events and relationships.Fuzzy mining uses significance and correlation to decide which events and relationships to highlight, aggregate, or remove.Source: §4.1 · pp. 7–8

4.2 AI techniques for the conducting phase

The conducting phase receives substantial AI attention, especially for selecting primary studies; machine learning is commonly combined with NLP and text mining, while other methods support selection and data extraction.

Supervised learning techniques for paper selection

  • Supervised learning studies use varied paper representations and classifiers, with SVM and Naive Bayes most frequently applied.The pioneering text-mining and neural-network work introduced WSS%95; later studies used SVMlight and found that adding non-topic papers increased performance, while another trained on 19 systematic reviews. Studies compared SVM and logistic regression against human screening, built a two-SVM ensemble with Platt scaling, and tested feature engineering and multiple classifiers. Reported conditions mattered: SVM performed better with title or title-plus-abstract features, Rocchio and NB with abstracts alone, and LMT with bag-of-words plus biomedical terms.Source: Supervised learning techniques for… · pp. 8–10

Active learning techniques for paper selection

  • Active learning reduces reliance on costly labels by querying human reviewers, with SVM commonly serving as the underlying classifier.Abstrackr lets reviewers label papers and highlight supporting terms; other strategies propagate labels to similar papers and query predictions by uncertainty. FASTREAD explores 32 learning-model configurations and reports that one abstract-method configuration outperformed state-of-the-art algorithms; FAST2 adds domain knowledge, a logistic-regression stopping estimate, and disagreement revision.Source: Active learning techniques for paper… · pp. 10–11

Other methods to support paper selection

  • Other selection-support methods combine search-string generation, interactive paper exploration, citation information, and automated empirical-study classification.One method infers search strings from accepted papers and uses active or reinforcement learning to guide interactive selection. Document and citation maps support reviewer inspection, while SCAS combines a StArt score with citation status in a decision tree; another study classifies empirical versus non-empirical papers and finds that full-text words slightly improve performance.Source: Other methods to support paper selection · pp. 11–12

AI techniques for data extraction and summarisation

  • Data-extraction methods use ontologies to represent study concepts and relationships, and machine learning to identify sentences describing treatment comparisons.SLROnt supports reasoning about primary studies using abstract information; another approach maps biomedical abstract terms to concepts from three medical ontologies and fills a PICO-based template, using machine learning for entity recognition. A separate method trains linear- and Gaussian-kernel SVMs on 100 sentences to identify entities and treatment characteristics in comparison sentences.Source: AI techniques for data extraction and… · pp. 12–13

4.3 AI techniques for the reporting phase

The reporting phase has received little attention, and current AI approaches support only report writing and report evaluation. Proposed methods generate summaries of selected studies and questions to assess whether reports reflect their relevant aspects.

4.3 AI techniques for the reporting phase

  • Current AI approaches support only two reporting tasks: writing the SLR report and evaluating it.This phase has received little attention so far.Source: §4.3 · p. 13
  • For report writing, Teslyuk et al. propose extracting up to five sentences around citations with NLP, encoding them with BioBERT, and generating summaries with an LSTM.The approach uses a pre-trained biomedical language representation model and a deep neural network to transform the encoded sentences into study summaries.Source: §4.3 · p. 13
  • For evaluation, Liu et al. propose NLP-generated questions about studies’ aims, contributions, methods, datasets, results, strengths, and limitations.LBJ, phrase parsers, and regular expressions generate questions about sentences explaining cited works; LBJ uses functions, constraints, and inference for NLP tasks.Source: §4.3 · p. 13

4.4 Previous analyses of the field and tool evaluations

Earlier analyses compare SLR tools and examine AI-related research from different perspectives, complementing the survey and adding a historical view of the field.

Previous analyses of the field and tool evaluations

  • 57% of tools in a 14-paper software-engineering mapping study used text mining, and the tools and effectiveness experiments were judged early-stage and preliminary.The study also covered extensions of Revis and the SLROnt ontology, evaluated on software-testing and cost-estimation paper corpora, respectively.Source: §4.4 · p. 14
  • Tsafnat et al. surveyed AI-based SLR automation in evidence-based medicine, analysing Abstrackr while noting other techniques without in-depth analysis.They identified potential in automatic hypothesis generation, reasoning about inclusion criteria, NLP-based duplicate detection, ML-and-heuristic abstract screening, and NLP plus optical character recognition for multilingual text analysis.Source: §4.4 · p. 14
  • 30-70% workload reduction was achievable across paper-selection ML experiments, while active learning ensured higher recall.The studies identify class imbalance as a challenge and suggest class weighting or undersampling as possible remedies; one also reviews approaches and their workload effects and challenges.Source: §4.4 · p. 14
  • 66% of feature representations in a paper-screening text-mining analysis used term frequency, and SVMs and ensembles were the most widely applied of 13 classifiers.The study characterises preprocessing methods by feature extraction, classifier, evaluation measures, and paper corpus.Source: §4.4 · p. 14
  • Beller et al. proposed SLR-automation principles emphasizing efficiency, coverage of multiple tasks, flexible combinations of methods, and replicability through open-source resources.Their principles were derived from an international meeting of ICASR members.Source: §4.4 · pp. 14–15
  • 13 tools in Marshall et al.'s practical guide were classified by SLR task, and most were framed as assistants whose results users must validate.The guide targets non-experts in medicine and covers tools in the SR Toolbox catalogue, omitting scientific literature unless a supporting tool was available; it cautions that most tools were prototypes or research-oriented, while this survey provides an up-to-date AI-tool analysis in supplementary material.Source: §4.4 · p. 15

5 Analysis of current trends

The review finds that AI studies span all SLR phases but concentrate on primary-study selection, where machine learning approaches predominate and human involvement varies from none to interactive review.

5.1 SLR phases currently supported

  • All SLR phases have at least one AI study, but conducting—especially primary-study selection—dominates; most tools remain unavailable and only two papers address multiple tasks.AI-based methods have been reported to reduce screening burden by up to 60% and save more than 80 hours. Abstrackr and EPPI-Reviewer are relatively popular in medicine, while five new selection methods appeared in the last four years [26, 27, 34, 35, 40].Source: §5.1 · p. 15
  • Selection has attracted attention since the first paper, and supporting tools have also been evaluated.The five new methods were proposed in the last four years.Source: §5.1 · p. 15

5.2 Selection of AI techniques

  • Machine learning is the most frequent AI area; active learning is the reference approach for paper selection, while imbalance is addressed with algorithms [29] or balancing techniques.Active learning models labeling cost rather than assuming an unlimited supply of labeled training data.Source: §5.2 · p. 16
  • SVMs appear in 13 of 17 classification papers; NB, DT, LR, and neural networks appear in 5, 3, 2, and 2, respectively.The paper notes that the number of studies is too low to explain why particular algorithms were chosen.Source: §5.2 · p. 16
  • Selection methods usually build text features from titles and abstracts, often weighting terms with TF-IDF; Abstrackr also lets researchers interactively highlight relevant and irrelevant words.Some methods use keywords, and LDA can extract a predefined number of high-level topics; processing more paper text could capture information beyond headers but requires handling more words.Source: §5.2 · p. 16
  • Most methods make a binary selection, while a few rank papers; only two methods add citation or cross-reference information.Most systems return a reduced list rather than simulate criteria-guided evaluation; the methods using citations and cross-references develop visualization and clustering to support analysis.Source: §5.2 · pp. 16–17

5.3 Human intervention

  • The review classifies 68% of approaches as fully automated and 32% as semi-automated; human-created training data alone does not make an approach semi-automated.Semi-automated approaches explicitly require human intervention during execution.Source: §5.3 · p. 17
  • Human roles range from Abstrackr reviewers’ interactive feedback to labeling or validation in active-learning methods.Abstrackr users highlight terms and label papers, with borderline papers routed to more experienced reviewers; researchers have used it to reproduce real screening and measure time saved and selection precision. Other studies describe human-oracle strategies, including prioritizing uncertain or positive papers, and FAST2 rechecks papers where reviewer and learner strongly disagree. In one simulated evaluation, a classifier achieved 92% performance with 5% of papers labeled.Source: §5.3 · pp. 17–18

6 Open issues and challenges

The review finds that AI-based SLR automation is concentrated on paper selection, while other tasks and several AI approaches remain less developed. It identifies needs for more specialised and interpretable methods, human interaction, broader evaluation, and accessible research artefacts.

6 Open issues and challenges

  • AI-based SLR research is strongly biased toward paper selection, while AI-driven writing tasks such as formulating research questions, setting criteria, and reporting results remain a challenge.Work on other tasks has appeared, but is less mature than paper-selection algorithms.Source: §6 · pp. 18–19
  • Optimisation and search remain unexplored for SLR tasks, while knowledge representation and NLP are less common, often preliminary, and lack development tools and frameworks.The review suggests optimisation could help prioritise databases or assign papers to reviewers according to their skills.Source: §6 · p. 19
  • For paper selection, the review calls for studying alternatives to SVM and algorithms designed for SLR tasks, including methods that combine input types and offer more flexible outputs.SVM is a reference algorithm, possibly because of the high-dimensional bag-of-words representation.Source: §6 · p. 19
  • Text-analysis methods need adaptation to disciplinary vocabulary; relying on titles and abstracts may miss emerging terminology, motivating use of paper sections, metadata, citations, and research-community links.The review names co-authorships and cross-references as ways to identify emerging topics without established terminology.Source: §6 · p. 19
  • Human involvement remains limited mainly to supplying paper-selection labels, and interactive AI could support planning and writing while adapting results to user preferences.The review also identifies trust as a challenge for non-AI-expert end users and calls for more interpretable models and explainable methods alongside black-box systems.Source: §6 · pp. 19–20
  • AI-based SLR methods need evaluation across more topics and fields because terminology, paper types, and available benchmarks can constrain replication and cross-field comparison.Selection decisions in available datasets may not be explicitly tied to inclusion and exclusion criteria.Source: §6 · p. 20
  • Open science requires accessible, functional implementations and corpora, as well as common experimental frameworks for comparing proposals.The review notes that some previously public tools and algorithms are no longer accessible.Source: §6 · p. 20
  • FAST2 found almost 95% of primary studies with less than 10% of papers screened, while Abstrackr found only 10% after the same 300 papers were screened.The authors found both tools useful and intuitive despite configuration issues, and suggested showing reviewers why papers were selected and allowing keywords to be added at the beginning.Source: §6 · pp. 20–21

7 Conclusions

AI has been used most prominently to support paper screening in systematic literature reviews, while planning and reporting remain difficult to automate. Ontologies and NLP have also been applied to semantic information, but fewer studies address these tasks.

7 Conclusions

  • AI, especially machine learning, has attracted clear interest for automating paper screening, the burdensome task of finding relevant works among thousands of candidates.The conclusion notes that many other SLR tasks have not yet been automated.Source: §7 · p. 21
  • Ontologies and NLP have been used to handle semantic information, but studies addressing these tasks remain far less abundant.Source: §7 · p. 21
  • Future work should support planning and reporting, whose tasks are harder to automate; deep-learning conceptual approaches suggest automatic writing may advance in the near future.Source: §7 · p. 21

Competing interests.

The authors declare no competing interests; the declarations also assign conceptualization and methodology to all authors and specify individual roles in analysis, writing, funding, and supervision.

Competing interests.

Declarations.

  • All authors handled conceptualization and methodology, while named authors undertook analysis, writing, funding acquisition, and supervision.J. de la Torre-López and A. Ramírez conducted formal analysis and investigation; J. de la Torre-López prepared the original draft; A. Ramírez and J.R. Romero reviewed and edited; J.R. Romero acquired funding and supervised.Source: Declarations · p. 21
Loading 2401.10917v1…