Source-linked AI summary
Leveraging Large Language Models for Systematic Literature Review of Disease Spread Models
Orhan Yagizer Cinar, Timur Emre Ozkose, Emma Von Hoene, Amira Roess, Taylor Anderson, Hamdi Kavak
TL;DR
SLRs are costly and labor-intensive, while evidence for LLM-based extraction at full review scale remains limited outside clinical research. This study develops and evaluates a zero-shot API pipeline on 536 COVID-19 ABM papers against human-extracted data. GPT-4.1 and GPT-5.0 achieved moderate-to-high paper-level agreement, with GPT-5.0 performing better, while field complexity and PDF parsing constrain interpretation.
Problem
Evidence remains limited on how well LLM-based extraction performs at the full scale of systematic reviews, especially for modeling and simulation.
Method
An automated zero-shot API pipeline processed full-text papers and compared GPT-4.1 and GPT-5.0 structured outputs with a human-extracted dataset using field-specific similarity metrics.
Results
77.95% for GPT-4.1 and 81.67% for GPT-5.0 mean paper-level accuracy, with GPT-5.0’s improvement statistically significant.
Takeaways & Limitations
Inter-model agreement can provide an additional diagnostic for interpreting LLM extraction quality and possible inconsistencies in human reference data.
Takeaways & Limitations
Internal PDF parsing may omit document elements such as multi-column layouts, tables, footnotes, and appendices, affecting some extraction fields.
Abstract
from arXiv · showhide
Recent advancements in Large Language Models (LLMs) have created new opportunities to streamline and potentially automate many research processes, including systematic literature reviews (SLRs). This study reports an LLM pipeline development for extracting model-relevant information from 536 peer-reviewed agent-based modeling papers. We compare the results with those of a human-conducted SLR. Our results show paper-level accuracies of approximately 77.95% for GPT-4.1 and 81.67% for GPT-5.0. Field-level accuracy ranges from 32.40% to 100.00%, with more complex or subjective fields performing less reliably. Importantly, we find that agreement between LLMs is a potential indicator of output quality: low agreement may signal hallucinations, whereas high agreement combined with low accuracy may point to noise or errors in the human dataset. Overall, our study provides practical insights into prompt development and highlights both the potential and limitations of using LLMs for full-scale SLRs in the modeling and simulation domain.
1 INTRODUCTION
SLRs provide rigorous, reproducible evidence synthesis but require substantial time and cost, motivating LLM support. This study evaluates LLM extraction for large-scale modeling and simulation reviews against human-extracted data.
- SLRs use transparent, reproducible methods to identify and synthesize evidence, but require extensive searching, screening, assessment, and manual extraction.
- LLM research has addressed multiple SLR stages, including literature search, study selection, and data extraction.A review identified 37 papers on LLM use for SLRs.
- Data extraction is especially important because it is labor-intensive and vulnerable to reviewer subjectivity.
- Evidence remains limited on LLM performance at the full scale of systematic reviews, particularly for modeling and simulation fields.
- The study evaluates GPT-4.1 and GPT-5.0 against human-extracted data from 536 COVID-19 agent-based modeling papers using an automated field-level comparison pipeline.The pipeline focuses on extraction rather than screening or synthesis.
- The paper compares human-extracted and LLM-extracted data and compares outputs from GPT-5.0 and GPT-4.1.
2 LARGE LANGUAGE MODELS IN SYSTEMATIC LITERATURE REVIEWS
Prior LLM-assisted SLR extraction studies report promising results but generally use small samples, less reproducible interfaces, and predominantly clinical evidence. This study addresses the resulting domain and scale gap for modeling and simulation.
- Scientific research processes, including literature review, have long been viewed as sufficiently formalizable for computational support.
- PRISMA-compliant SLRs include defining objectives, searching, screening, extracting data, synthesizing, and reporting evidence.
- Existing zero-shot extraction studies report better performance for explicit variables than subjective variables, with prompt refinement improving accuracy in small samples.One cited study examined 10 COVID-19 simulation studies.
- Across multiple LLM versions, reported extraction results include higher accuracy for Claude 2 and 91% versus 89% for LLM-assisted versus human extraction.
- Existing evidence is largely based on small article samples, less reproducible web interfaces, and clinical studies, limiting direct extension to modeling and simulation.
3 METHODS
The study builds a reproducible API-based pipeline that processes each paper independently and evaluates structured LLM outputs against a human reference using field-specific similarity measures. The design covers zero-shot extraction, validation, full-text processing, and comparison across field types.
- Human-Extracted Reference Data: The pipeline processes 536 human-reviewed COVID-19 ABM papers and compares GPT-generated structured extraction with the human reference dataset.
- Human-Extracted Reference Data: Human reviewers used a standardized extraction form, pilot training, paired independent review, and consensus resolution of discrepancies.
- Human-Extracted Reference Data: The reference schema contains 21 data fields covering model purpose, reuse, platforms, documentation, assumptions, code availability, disciplines, evaluation, and uncertainty.
- LLM Pipeline: The API pipeline uses secure, programmatic calls, controlled batching, detailed logging, and independent paper processing to support reproducibility and scale.
- LLM Pipeline: GPT-4.1 and GPT-5.0 were evaluated zero-shot with default API parameters and no adaptation or tuning on the human dataset.
- LLM Pipeline: A uniform structured prompt specified extraction instructions, permitted categorical responses, and JSON-only output constraints.
- LLM Pipeline: Each full-text PDF was processed in one API call, creating a direct mapping between one paper’s LLM extraction and human extraction.
- LLM Pipeline: Model responses underwent JSON parsing, rule-based structural repair, and lightweight-LLM correction when earlier validation steps failed.
4 RESULTS
LLM extraction matched human SLR data unevenly across fields and papers, with stronger performance for explicit or simpler fields and GPT-5.0 outperforming GPT-4.1 overall.
- Field-level Accuracy: 100% accuracy was achieved by both models for explicitly stated model purpose, while sub-classifications reached 32.4% for GPT-4.1 and 37.4% for GPT-5.0.The sub-classifications field had zero class variance in the human reference, limiting its interpretation as a discriminative extraction test.
- Field-level Accuracy: High-scoring fields included model purpose, geographic specificity, standard documenting protocols, and stated assumptions and limitations.The authors describe these fields as potentially accelerating SLRs without sacrificing too much accuracy.
- Field-level Accuracy: 81.6% and 85.0% were the accuracies for free-text fields, compared with 81.2% and 84.0% for single-choice fields and 58.3% and 66.0% for multi-choice fields.The results associate lower multi-choice performance with the complexity of check-all-that-apply tasks and subjective fields.
- Paper-level Accuracy: 77.95% was the mean paper-level accuracy for GPT-4.1 versus Human, compared with 81.67% for GPT-5.0 versus Human and 82.43% for model agreement.The GPT-5.0 distribution was shifted toward higher accuracy, with more papers concentrated in the 80–90% range.
- Paper-level Accuracy: +3.73 percentage points was GPT-5.0’s mean extraction-accuracy advantage over GPT-4.1, a statistically significant difference across 536 paired papers.The paired comparison reported t(535) = 6.44, p < .001, and Cohen’s d = 0.28.
5 DISCUSSION, CONCLUSION, AND FUTURE WORK
Across 536 COVID-19 agent-based modeling papers, LLM extraction achieved moderate-to-high agreement with human reference data, while performance varied by field structure and subjectivity. Inter-model agreement offers a possible diagnostic for unstable outputs or inconsistencies in human labels, but the zero-shot design and possible contamination constrain interpretation.
- Results: 77.95% mean paper-level accuracy for GPT-4.1 and 81.67% for GPT-5.0 were achieved against the human reference dataset.The GPT-5.0 improvement was statistically significant: t(535) = 6.44, p < .001.
- Field-level performance: Single-choice fields generally outperformed multi-choice fields because multi-choice classifications require more interpretation and domain-specific judgment.Examples of difficult multi-choice fields include “Assumptions and Limitations”, “Main disciplines”, and “Sub-classifications”.
- Field-level performance: Free-text fields achieved accuracy and agreement similar to single-choice fields when they captured straightforward information such as country, platform, or model link.The reported similarity is attributed to the straightforward nature of the extracted data.
- Inter-model agreement: Low GPT-4.1–GPT-5.0 agreement may indicate instability, prompt sensitivity, or hallucination, whereas higher inter-model than human-model agreement may indicate reference-data inconsistencies.The field “The model extends or uses an existing model” had 87.1% inter-model agreement versus 77.6% and 80.0% agreement with the human reference, but this diagnostic remains proposed rather than demonstrated correction.
- Limitations: The study’s zero-shot, conservative configuration supports reproducibility but likely underestimates performance achievable with domain-adapted prompting, few-shot examples, retrieval augmentation, or assisted workflows.The authors present the design as a baseline under default usage conditions.
- Contributions: The study extends prior work by applying zero-shot extraction to modeling and simulation reviews rather than predominantly clinical reviews, and by assessing inter-model agreement as an auditing signal.The authors caution that agreement could also reflect training-data overlap, prompt artifacts, or PDF-parser artifacts.
- Limitations: Possible training-data overlap could inflate accuracy, although the authors note that judgment-based labels and poor performance on contextual fields make memorization an unlikely primary driver.The authors recommend contamination-detection approaches where feasible.
- Future work: Future work will test ensembles for stability and human-adjudication flags, and extend evaluation to screening, selection, synthesis, and reporting.These directions aim to assess broader end-to-end systematic review workflows.
NOTES
The paper acknowledges contributors to the original data extraction and provides public access to study materials, code, and scoring references.
- Acknowledgments: Sara Von Hoene and five additional collaborators are acknowledged for extracting the original data.The listed collaborators are Szandra Peter, Ethan Hopson, Emily Csizmadia, Faith Fenyk, and Kai Barner.
- Data and code availability: Study titles, pipeline-development Python code, and detailed scoring-logic references are available for download.The materials are hosted at OSF under DOI 10.17605/OSF.IO/MTRCP.
- Data and code availability: The supplementary materials can be accessed through the OSF repository at https://osf.io/mtrcp.The repository includes study titles, code, and scoring references.