PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews

Shreyansh Padarha, Ryan Othniel Kearns, Tristan Naidoo, Lingyi Yang, Łukasz Borchmann, Piotr BŁaszczyk, Christian Morgenstern, Ruth McCabe, Sangeeta Bhatia, Philip H. Torr, Jakob Foerster, Scott A. Hale, Thomas Rawson, Anne Cori, Elizaveta Semenova, Adam Mahdi

arXiv:2603.22327v2cs.IRcs.AIcs.DL

TL;DR

AI-based systematic literature review remains difficult to evaluate reliably in high-stakes epidemiology. AgentSLR evaluates five frontier models across review stages and finds no model dominates, while average field-level extraction F1 stays at or below 0.67, supporting human-supervised use.

  • Problem

    LLM evidence-synthesis workflows remain insufficiently reliable for high-stakes epidemiological reviews, with errors requiring expert review.

    Source: §1 · p. 1

  • Method

    AgentSLR combines an expert-annotated epidemiological SLR dataset with a workflow and stage-isolated metrics for retrieval, screening, and structured extraction.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    No evaluated model dominates across the workflow, and no model exceeds an average field-level extraction F1 of 0.67.

    Source: §6.1 · p. 8

  • Takeaways & Limitations

    The results support human-supervised deployment of LLMs in epidemiology rather than unsupervised use.

    Source: §7 · p. 9

    The evaluation covers open-access articles from queried sources, approximately 27.0% of the PERG corpus, and excludes multilingual evidence, meta-analysis, and final review writing.

    Source: §6.4 · p. 9

Abstract

from arXiv · show

Systematic literature reviews (SLRs) are a demanding and high-stakes form of scientific knowledge synthesis that remains underspecified as an evaluation setting for large language models (LLMs). We introduce AgentSLR, a large-scale evaluation harness comprising an SLR automation workflow and an expert annotated dataset covering 16,248 articles, designed to test LLM capabilities across the stages of SLRs in epidemiology. Reference annotations were derived from peer-reviewed studies on WHO priority pathogens and produced by domain experts. The harness evaluates each review stage as a separate unit with dedicated metrics enabling targeted failure analysis. We evaluated five frontier reasoning models and found that no single model dominated across all tasks, showing sub-task specialisation often hidden by aggregate benchmarks. Structured data extraction is a major bottleneck, with no model exceeding an average field-level F1 of 0.67. Estimated costs vary substantially, by up to 96 times across evaluated models. Documented failure modes suggest that the evaluated models are not yet reliable enough for unsupervised deployment in epidemiology, where findings can inform public policy.

1 Introduction

AgentSLR evaluates whether LLMs can perform evidence-handling tasks in epidemiological systematic literature reviews, using expert annotations from WHO priority-pathogen reviews as the reference standard. The study addresses limited reliability in multi-stage evidence synthesis and finds that no single evaluated model consistently dominates across workflow stages.

Problem

  • Although LLMs can reduce title-and-abstract screening time, false inclusions, false exclusions, omitted scope-limiting details, and compounded errors constrain reliability in longer evidence-synthesis workflows.These limitations require expert review and can make conclusions appear broader than the source articles support.Source: §1 · p. 1

Motivation

  • Epidemiological systematic reviews provide a rigorous test because key parameters can vary across age group, geography, and clinical severity, requiring models to link numbers to the correct context.Relevant parameters include the basic reproduction number, serial interval, and case-fatality ratio.Source: §1 · pp. 1–2

Contributions

  • 16,248 article records, 3,808 parameter extractions, 687 transmission-model extractions, and 189 outbreak extractions form an expert-annotated evaluation dataset from WHO priority-pathogen reviews.The annotations were produced in collaboration with the Pathogen Epidemiology Review Group and matched to expert screening labels.Source: §1 · p. 2
  • AgentSLR evaluates retrieval, screening, and structured extraction in LLM-assisted epidemiological systematic literature reviews, scoring extraction at the record level.Record-level scoring allows evaluation when the same evidence can be represented in more than one correct way.Source: §1 · p. 2
  • No single model consistently dominates across workflow stages, while refusal behaviour in closed systems underscores the importance of open, locally reproducible infrastructure.The evaluation covers five frontier reasoning LLMs.Source: §1 · p. 2

2 Related Work

Prior work evaluates systematic-review automation across retrieval, screening, extraction, and evidence synthesis, but aggregate pipelines remain domain-specific. AgentSLR addresses this gap with an epidemiological evaluation harness that tests complex evidence structuring against expert annotations from WHO priority-pathogen reviews.

Stage-specific review automation

  • LLMs transfer screening logic across review stages with high sensitivity and specificity, but extraction performance declines on complex fields and human-in-the-loop workflows generally outperform LLM-only approaches.Systematic-review evaluation spans retrieval, screening, and evidence structuring.Source: §2 · p. 2

Workflow-level pipelines

  • Aggregate review pipelines have emerged, but existing systems are tailored to individual domains such as bee ecotoxicology and Cochrane-style clinical reviews.Epidemiology therefore requires a dedicated pipeline or evaluation harness.Source: §2 · p. 3

Evaluation criteria

  • Recent evaluations test value matching, recall-focused screening, cost-sensitive analysis, within-paper evidence retrieval, and agreement with Cochrane conclusions [23].These studies examine distinct review and research-synthesis capabilities that LLMs may reliably support.Source: §2 · p. 3

AgentSLR contribution

  • AgentSLR evaluates epidemiological SLRs against expert annotations from peer-reviewed WHO priority-pathogen reviews and requires evidence-family identification, schema selection, uncertainty bounds, population context, and alignment beyond exact match.Papers may contain multiple estimates across age groups, locations, or periods, with values implicit in disaggregated tables and lacking canonical alignment to human annotations.Source: §2 · p. 3

3 Evaluation Harness

AgentSLR combines an epidemiological SLR workflow with stage-isolated evaluation metrics mapped to expert labels. Its protocol separately evaluates screening and structured extraction, while report generation remains outside the evaluated construct.

3 Evaluation Harness

  • AgentSLR pairs a human-like epidemiological SLR workflow with an evaluation protocol that maps stage outputs to expert labels using isolated metrics.The harness is designed as an open-source evaluation framework with workflow replication and targeted failure analysis across evidence-handling stages.Source: §3 · p. 3

3.1 Building SLR Workflows

  • The workflow covers database retrieval, screening, PDF-to-Markdown conversion, schema-constrained extraction, population tagging, and standardized SLR-like artefact generation.Retrieval uses OpenAlex, PubMed, and Europe PMC; extraction targets epidemiological parameters, transmission models, and concluded outbreaks.Source: §3.1 · pp. 3–4
  • Report generation is excluded from evaluation because the metrics target evidence handling, while narrative synthesis requires matched reference reports and introduces additional safety risks.The workflow still produces figures, tables, and aggregate statistics through standardized code and LLM self-refinement.Source: §3.1 · p. 4

3.2 Evaluation Protocol

  • Screening is evaluated against human PERG labels using precision, recall, and macro F1 across abstract-to-full-text funnel variants.The protocol compares AI abstract-to-AI full-text, human abstract-to-AI full-text, and AI direct full-text screening.Source: §3.2 · p. 4
  • Structured extraction separates presence detection, record counting, and field-level content accuracy, using weighted Jaccard matching with a one-to-one assignment algorithm.This design distinguishes over-extraction, missed records, and inaccurate fields when generated records are only partially correct.Source: §3.2 · p. 4

4 Experiment Settings

AgentSLR evaluates LLM extraction against epidemiologists’ human-reference labels from WHO priority-pathogen SLRs, using a released open-access benchmark of 16,248 matched article records and structured extractions. The comparison covers five reasoning-capable models across screening and pathogen-specific extraction tasks.

4.1 Dataset

  • Human-reference labels come from peer-reviewed SLRs conducted by PERG for nine WHO priority pathogens, with five SLRs published and two in data extraction.The dataset extends open-source resources and is released with associated code for reproducibility.Source: §4.1 · pp. 4–5
  • The benchmark covers seven pathogens for screening and four for data extraction, with parameter and transmission extraction across four pathogens and outbreak extraction only for Lassa and Zika.Marburg, MERS, and Nipah are excluded because of inconsistent formats or incomplete extraction phases.Source: §4.1 · p. 5
  • The released benchmark contains 16,248 matched article records, 3,808 parameter extractions, 687 transmission model extractions, and 189 outbreak extractions.The matched open-access subset represents approximately 27.0% of the PERG corpus; a 1,004-article closed-access comparison found broadly comparable performance across stages.Source: §4.1 · p. 5

4.2 Models

  • The evaluation compares GPT-5.2, gpt-oss-120b, Kimi K2.5, GLM-4.7, and DeepSeek-V3.2, using high reasoning settings where possible and up to 64K tokens per pass.Claude Opus 4.5 and Sonnet 4.5 were not evaluated after streaming refusals; open-weight models ran on vllm with NVIDIA H200 hardware.Source: §4.2 · p. 5

5 Results

Across five models, AgentSLR finds no universal leader: performance varies by SLR subtask, extraction remains the main bottleneck, and screening design, expert oversight, and cost materially affect deployment suitability.

5.1 Scientific Synthesis Is Not a Single Capability

  • No single model dominates the workflow: Kimi-K2.5 and gpt-oss-120b lead screening, while extraction leaders vary by data type.Kimi-K2.5 leads title and abstract screening (F1 = 0.77) and gpt-oss-120b leads full-text screening (F1 = 0.87); Kimi-K2.5 leads parameter extraction (F1 = 0.63), GLM-4.7 model extraction, and GPT-5.2 outbreak extraction.Source: Figure 2 · p. 6, §5.1 · p. 6

5.2 Structured Evidence Extraction Is the Bottleneck Across Models

  • With no LLM exceeding average extraction F1 of 0.67, field-level evidence extraction trails flagging across structured data types.Mean flagging F1 ranges from 0.72 to 0.80, versus 0.61 to 0.67 for field-level extraction; parameter extraction remains below F1 = 0.60 for every model, while GPT-5.2 reaches 0.84 for outbreak extraction on Lassa and Zika.Source: §5.2 · p. 6, Table 2 · p. 6

5.3 Screening Strategy Controls What Evidence Survives

  • Recall rises from 0.81 to 0.92 when human abstract decisions condition full-text screening, while direct full-text screening reaches 0.89 but costs 2.3× more.The fully automated two-stage strategy achieves F1 = 0.77; human-conditioned screening gives F1 = 0.87, whereas direct screening gives F1 = 0.73 and raises OCR costs from USD 36.6 to USD 303.2.Source: §5.3 · p. 7, Figure 3 · p. 7

5.4 Expert Review Separates Utility from Autonomy

  • Expert review supports supervised assistance rather than autonomy: field-level accuracy is 0.77–0.83, but competence is only 2.80 for model extraction.Across six epidemiologists, competence scores are 4.23 for parameters, 3.90 for outbreaks, and 2.80 for models; recurring failures involve document structure, cited prior work, and contextual inference.Source: §5.4 · p. 7, Table 3 · p. 7

5.5 Performance Does Not Scale Reliably with Cost

  • At competitive average performance, gpt-oss-120b costs USD 13.9 per pathogen run, whereas GPT-5.2 costs USD 1,348.2 and performs slightly worse.gpt-oss-120b achieves F1 = 0.70, GPT-5.2 F1 = 0.69, and Kimi-K2.5 has the best overall F1 = 0.74 at $277; GPT-5.2 produces 91.1K output tokens per parameter article versus 3.0K for DeepSeek-V3.2.Source: §5.5 · p. 8

6 Discussion

AgentSLR shows that epidemiological systematic-review automation comprises separable capabilities, with structured extraction remaining a major bottleneck and no model dominating across stages. The system can substantially accelerate evidence triage, but deployment requires expert oversight because infrastructure constraints, structured errors, and evaluation limitations remain consequential.

6.1 Key Findings

  • No evaluated model dominates the workflow, and average field-level extraction F1 remains at or below 0.67 across all five LLMs.Extraction performance converges across models under tool-calling because heterogeneous reporting requires schema-grounded normalization across study designs.Source: §6.1 · p. 8

6.2 Deployment Risk

  • gpt-oss-120b achieves competitive average performance at over 96× lower cost than GPT-5.2, while screening, provenance, and pathogen-specific errors create distinct deployment risks.False negatives can permanently remove evidence, provenance failures can misattach correct values, and gaps such as Zika extraction can propagate systematic bias.Source: §6.2 · p. 8

6.3 Assisted Evidence Synthesis

  • 20 hours versus 385 human labour hours yields a 19.3× efficiency gain and 58× fewer calendar days, but AgentSLR remains an assistive tool rather than an expert replacement.The clearest short-term use is accelerating full-text screening and supervised extraction, where outputs provide a correctable starting point and support feasible living-review updates [41].Source: Figure 5 · p. 9, §6.3 · p. 9

6.4 Limitations and Future Work

  • The evaluation covers only about 27.0% of the PERG corpus through open-access English evidence and excludes meta-analysis and final review writing.Future work should measure human uplift, productionize the annotation interface, extend evaluation across domains, and test heterogeneous model routing across complementary tasks.Source: §6.4 · p. 9

7 Conclusion … A.1 SLR Automation and Workflows

AgentSLR evaluates LLM-assisted systematic reviews in infectious disease epidemiology through stage-specific, expert-grounded assessment. The framework exposes both assistive value and persistent limitations, motivating human-supervised deployment and reproducible evaluation.

7 Conclusion

  • AgentSLR decomposes epidemiological review production into retrieval, screening, and schema-grounded extraction, enabling stage-level diagnosis against human reference annotations.The framework evaluates scientific knowledge synthesis in a high-stakes, expert-grounded setting rather than relying on aggregate automation claims.Source: §7 · p. 9
  • Current LLMs provide assistive value especially for screening and correctable preliminary extraction, but structured extraction, error structure, cost, and model availability remain limiting factors.These findings support human-supervised deployment in epidemiology and position SLRs as a demanding test of reliable scientific evidence synthesis.Source: §7 · p. 9

A Extended Related Work

  • The related-work review distinguishes SLR automation workflows from benchmark and evaluation studies in scientific evidence synthesis.This organization reflects both domain-specific automation challenges and broader evaluation practices for SLRs.Source: A Extended Related Work · p. 16

A.1 SLR Automation and Workflows

  • Prior systems target isolated stages or clinical and biological domains, use incompatible schemas, and cannot transfer directly to epidemiological parameter extraction.These differences make direct quantitative comparison methodologically ill-posed, so Table 4 provides a qualitative comparison of domain, methodology, and evaluation rigour.Source: §A.1 · p. 16, Table 4 · p. 16
  • AgentSLR is the only compared framework combining open code, open-weight compatibility, human references, independent evaluation, and stage-specific metrics.Table 4 marks these five properties as present or absent across related pipelines; ✓ and × indicate presence and absence.Source: Table 4 · p. 16
  • AgentSLR is the first pipeline targeting WHO-priority-pathogen epidemiological reviews, supporting open-weight models with disclosed source code and expert-annotation evaluation without LLM-as-a-judge.Open-weight compatibility supports version pinning and local deployment, while stage isolation enables component-level failure attribution.Source: §A.1 · p. 16

A.2 Comparison with Benchmarks

AgentSLR occupies a distinct position among adjacent benchmarks by evaluating evidence handling across epidemiological SLR stages, from screening through schema-validated records. Unlike neighbouring evaluations of retrieval, conclusion matching, or clinical workflows, it targets preservation and normalization of heterogeneous epidemiological evidence with uncertainty and population context.

Clinical workflow evaluation

  • TrialReviewBench [23] spans search, screening, extraction, and synthesis for clinical oncology reviews, whereas AgentSLR targets epidemiological parameters, transmission models, and outbreaks.The comparator is broad and PRISMA-aligned, but its extraction targets are intervention-centred study characteristics and outcomes.Source: §A.2 · pp. 16–17

Evidence reasoning

  • MedEvidence and EvidenceBench test conclusion matching and supporting-sentence retrieval, but neither measures evidence survival through screening or conversion into structured epidemiological records.MedEvidence also exposes failures involving uncertainty and scientific scepticism.Source: §A.2 · p. 17

Screening metrics

  • LLM4SCREENLIT motivates risk-aware screening evaluation through recall, lost evidence, confusion matrices, and cost sensitivity, which AgentSLR extends to extraction with flagging, count, and field-level matching metrics.This extension addresses evidence handling beyond binary screening.Source: §A.2 · p. 17

SciLitBench

  • SciLitBench is an AI-assisted literature-review dataset, but AgentSLR omits detailed comparison because its public draft contains unresolved placeholders and undisclosed evaluation labels and design.The cited placeholder example is “TODOXXX”.Source: §A.2 · p. 17

AgentSLR positioning

  • AgentSLR fills a gap between clinical workflow automation and biomedical evidence reasoning by evaluating screening and record matching for epidemiological parameters, transmission models, and outbreaks.Its harness uses PERG expert annotations from peer-reviewed WHO priority pathogen reviews and does not evaluate final meta-analysis or narrative report writing.Source: §A.2 · p. 17

B Data Representativeness & Ecological Validity · C Evaluation Constructs · C.1 Article Screening

AgentSLR evaluates evidence-handling fidelity across representative epidemiological SLR data, while explicitly bounding claims to screening and structured representation rather than synthesis. Its metrics and screening configurations target distinct inclusion, coverage, counting, and extraction failures, with results interpreted cautiously because accessibility and construct-validity limitations remain.

B Data Representativeness & Ecological Validity

  • The evaluation spans screening across seven WHO priority pathogens and structured extraction across four pathogens with complete, consistently formatted SLR artefacts.The underlying reviews come from the Pathogen Epidemiology Review Group, whose curated article sets and extracted data are distributed through the epireview and priority-pathogen R packages.Source: B Data Representativeness & Ecological… · p. 18
  • Full-text screening differed by −5.8 percentage points between open-access and closed-access samples, while abstract screening and parameter extraction had overlapping confidence intervals, indicating broadly comparable workflow trends.Table 6 reports macro F1 with 95% bootstrap confidence intervals for matched open-access and 1,004 closed-access articles processed with gpt-oss-120b under identical conditions; the full-text interval excluded zero, whereas the other intervals included zero.Source: Table 6 · p. 18, B Data Representativeness & Ecological… · p. 18
  • Absolute performance estimates should be interpreted relative to the lawfully retrievable evaluation corpus because the released data exclude OCR text and PDF-derived Markdown unless redistribution is permitted.The open-access benchmark was compared with closed-access articles retrieved through institutional access, while released materials contain identifiers, metadata, and annotations rather than restricted document text.Source: B Data Representativeness & Ecological… · p. 18

C Evaluation Constructs

  • AgentSLR’s construct is evidence handling fidelity: identifying, retaining, and structurally representing claims from primary literature consistently with expert review practice.The construct excludes cross-study aggregation, heterogeneous-evidence weighting, and meta-analytic conclusions, so findings do not test full scientific knowledge synthesis.Source: C Evaluation Constructs · p. 19
  • Macro F1, precision, recall, and extraction decomposition measure inclusion fidelity, missed relevant articles, evidence visibility, record counts, and field-value correctness.Bipartite matching permits partial credit for partially correct records, while field weights and Jaccard similarity for multi-value fields are explicit modelling choices; cross-model rankings are therefore treated as more robust than absolute scores.Source: C Evaluation Constructs · p. 19
  • Construct validity is limited by omitted synthesis and report writing, OCR and prompting variance, field-weight choices, and disagreement in expert reference labels.The harness reports standard deviations across pathogens, relies on comparative rankings, and treats fresh-reviewer accuracy estimates from Section C.3 as a ceiling for automated metrics rather than a pure system-error measure.Source: C Evaluation Constructs · p. 19

C.1 Article Screening

  • Article screening is a binary include-or-exclude decision evaluated with precision, recall, and macro-F1, with macro averaging preventing class imbalance from allowing the majority class to dominate.Recall is reported because missed relevant articles propagate irrecoverably to downstream stages, while precision measures the reliability of inclusion decisions.Source: §C.1 · p. 19, C Evaluation Constructs · p. 19
  • Full-text screening is evaluated after abstract screening using three configurations: AI abstract→AI full-text, Human abstract→AI full-text, and AI direct full-text.The first two configurations force final exclusion after an abstract exclusion by AI or PERG, respectively; the direct configuration evaluates the AI full-text decision without abstract filtering.Source: §C.1 · p. 19

C.2 Data Extraction … C.3 Human Expert Validation

The extraction evaluation combines schema-filtered reference data with Flagging, Count, and field-level Extraction metrics, using optimal matching to separate over-extraction from field inaccuracies. Parameter, model, and outbreak schemas are specialized, while expert validation samples outputs across pathogens and data types but does not guarantee parity.

C.2 Data Extraction

  • Extraction performance is measured with precision and recall for Flagging, Count, and Extraction, covering relevance identification, extraction volume, and field-level accuracy.Flagging evaluates article–data_type labels; Count compares extraction volumes; Extraction evaluates matched metadata fields.Source: §C.2 · p. 20
  • Optimal one-to-one matching uses weighted field similarity and Jonker–Volgenant assignment, so unmatched extractions incur Count penalties without contaminating field-level Extraction scores.Similarity combines normalized field weights and Jaccard comparisons, while assignment maximizes total similarity.Source: §C.2 · pp. 20–21

Data Extraction: Parameters

  • Parameter extraction resolves nine parameter classes separately and weights categorical, value, uncertainty, and population fields equally across its staged evaluation.Perturbing within-group field weights by ±10% leaves cross-model rankings unchanged.Source: Data Extraction: Parameters · p. 21

Data Extraction: Transmission Models

  • Reference-data filtering removed invalid entries at rates of 3.85%–23.14% across pathogens, while AI-extracted annotations had 0% invalid entries under structured schema enforcement.The reported filtering statistics cover transmission-model annotations, and the tables report zero invalid AI entries across evaluated models.Source: Data Extraction: Transmission Models · p. 22, Table 8 · p. 23

Data Extraction: Outbreaks

  • Reference outbreak annotations had 0%–9.43% invalid entries removed across pathogens, whereas AI-extracted outbreak values had 0% invalid entries under schema enforcement.Outbreak extraction uses 17 fields grouped into temporal, geographic, case-burden, and epidemiological-context categories.Source: Data Extraction: Outbreaks · pp. 22–23, Table 9 · p. 23
  • Outbreak field weights prioritize country, start year, confirmed cases, and deaths, but equal weighting did not significantly change extraction F1 at the 95% level.The weighting scheme reflects a cascading importance order and was checked on n = 169 outbreak records.Source: Data Extraction: Outbreaks · p. 23

C.3 Human Expert Validation

  • Six epidemiologists validated AgentSLR extractions across parameters, models, and outbreaks, with assignments based on pathogen and workflow expertise rather than random allocation.Three experts validated parameters, two validated models, and one validated all three modalities.Source: §C.3 · p. 23
  • Validation sampled screened articles without replacement across four pathogens and assessed relevance, field accuracy, pipeline competence, and time, but did not guarantee parity across pathogens or data types.Experts used a 1–7 Likert scale and could provide qualitative error descriptions; extraction workloads varied by article and data type.Source: §C.3 · pp. 23–24

D Pipeline Statistics: Data Processed & Time … D.2 Token Usage and Operational Cost of AgentSLR harness

AgentSLR processes progressively fewer articles across SLR stages, while automation substantially reduces estimated runtime relative to human review. Parameter extraction dominates token usage and API cost despite title-and-abstract screening handling the largest article volume.

Article counts across SLR stages

  • Article counts decrease substantially from title-and-abstract screening through full-text screening to data extraction as progressively applied relevance criteria filter records at successive review stages.These counts represent estimated annotator workload at each phase rather than finalised inclusion totals.Source: Table 10 · p. 25, Article counts across SLR stages · p. 25

Runtime estimation methodology

  • Runtime estimates combine average stage-specific article counts with per-article processing times for PERG and AgentSLR, using direct execution logs for gpt-oss-120b.The gpt-oss-120b timings represent the upper bound across the evaluated model-runtime spectrum.Source: Runtime estimation methodology · p. 25, Table 11 · p. 25

PERG runtime calculations

  • 197.50 hours are estimated for PERG data extraction, compared with 114.15 hours for title-and-abstract screening and 73.47 hours for full-text screening.The estimates assume 30 minutes per extraction article, 45 seconds per title-and-abstract article, and 4 minutes per full-text article.Source: PERG runtime calculations · p. 26

AgentSLR workflow (gpt-oss-120b) runtime calculations

  • AgentSLR requires 0.63 seconds per article for article retrieval and title-and-abstract screening; with 9,132 articles retrieved or screened on average, each stage is estimated to take 1.6 hours.Source: AgentSLR workflow (gpt-oss-120b)… · p. 26
  • PDF-to-Markdown conversion takes 2.8 hours for 9,132 articles in parallel, versus 16.47 seconds per document under sequential execution.Parallel execution uses 14 concurrent requests and averages 1.1 seconds per document.Source: Table 11 · p. 25, AgentSLR workflow (gpt-oss-120b)… · p. 26
  • AgentSLR requires 0.62 hours for full-text screening and 13.4 hours for data extraction, with extraction taking 122.1 seconds per article across 395 articles.Data extraction comprises outbreak identification, model extraction, and parameter extraction.Source: AgentSLR workflow (gpt-oss-120b)… · p. 26

D.2 Token Usage and Operational Cost of AgentSLR harness

  • Parameter extraction dominates AgentSLR’s aggregate token usage and API cost, whereas title-and-abstract screening processes the most articles but contributes comparatively little cost.Costs are based on per-article token usage, average stage-specific article counts, and published input/output token pricing.Source: Figure 6 · p. 26, §D.2 · p. 27
  • Table 12 reports mean input tokens, output tokens, and USD cost per article for each pipeline stage and model, with minima and maxima marked within rows and subcolumns.Aggregate stage costs are derived from these per-article statistics and the average article counts reaching each stage.Source: §D.2 · p. 26, Table 12 · p. 27

E Extended Evaluation Results … Full-text Screening

Extended evaluation shows strong specialization across models and substantial variation by screening stage, extraction task, pathogen, and field type. Field-level analyses and screening ablations identify recall–precision trade-offs, overcounting, and difficulty with nuanced contextual information.

Full-text Screening

Parameters

  • Parameter extraction averages remain close across models, but field behavior diverges: GPT-5.2 leads Extraction F1 at 0.59, while gpt-oss-120b combines flagging recall 0.92 with count precision 0.83 and recall 0.47.Value and population fields are weakest, whereas single-type uncertainty and method classification are generally strong; exact-match criteria may penalize equivalent numeric formats.Source: Parameter Extraction · p. 29, Table 15 · p. 29, Parameters · pp. 31–32

E.1 Results Across Models

E.2 Field-Level Results Possible with AgentSLR

  • Field-level evaluation of gpt-oss-120b exposes localized errors beyond aggregate scores, including overflagging parameters, undercounting repeated extractions, overestimating transmission-model counts, and difficulty with context-dependent outbreak fields.Across transmission screening and extraction, precision ranges from 0.61 to 0.70 and recall from 0.75 to 0.81, while temporal and case-burden outbreak fields remain comparatively robust.Source: §E.2 · p. 31, Parameters · p. 31, Transmission Models · p. 32

Transmission Models

  • Transmission-model screening recalls 0.86–0.99, but model-count precision is only 0.48–0.60, while nuanced fields such as transmission routes, assumptions, and interventions remain hardest.Core structural fields perform better, with model type and theoretical-versus-data-fitted classification reaching balanced precision and recall between 0.62 and 0.89.Source: Transmission Model Extraction · p. 30, Transmission Models · p. 32, Table 20 · p. 33

Outbreaks

  • Outbreak extraction is strongest for temporal and case-burden fields but weaker for specific locations and epidemiological context, with overall precision/recall of 0.85/0.73 for Lassa and 0.84/0.78 for Zika.Lassa counting reaches precision 0.83 and recall 1.00, whereas Zika counting falls to 0.49/0.45; country identification is perfect but specific-location extraction is 0.50/0.56 for Lassa and 0.50/0.50 for Zika.Source: Table 21 · p. 33, Outbreaks · p. 34, Table 22 · p. 34

Title and Abstract Screening

  • For article screening ablations, title-and-abstract screening achieves precision 0.79, recall 0.72, and F1 0.74, with MERS strongest at F1 0.78 and Marburg weakest at 0.69.The pattern favors specificity over recall, producing fewer irrelevant inclusions at the cost of more false negatives.Source: Title and Abstract Screening · p. 35, Table 23 · p. 35

Full-text Screening

  • Human abstract →AI full-text screening performs best overall at precision 0.83 and recall 0.92, while AI abstract →AI full-text recall is 0.81 and direct AI full-text reaches recall 0.89 with precision 0.68.The results indicate a precision–recall trade-off and error propagation when AI abstract screening gates full-text screening.Source: Full-text Screening · p. 35, Table 24 · p. 35

F Extended Expert Validation Results … G.5 PDF Retrieval

Extended expert validation found uneven AgentSLR performance across outbreak and parameter fields, while the retrieval workflow combines standardized epidemiological queries with database-specific adaptations, pathogen modifications, deduplication, and cascading PDF acquisition.

F Extended Expert Validation Results

  • The six-researcher validation survey audited relevance and field correctness rather than duplicating PERG annotation, limiting separation of model errors from reference noise.The survey collected 62 parameter, 50 model, and 31 outbreak submissions, and filtered reference entries with invalid enum values before evaluation.Source: F Extended Expert Validation Results · p. 36
  • 0.61 overall flagging precision and 0.80 overall extraction accuracy were expert-rated for outbreak outputs, with field-level scores varying substantially.Country reached 0.95, while start year was 0.84 and end year was 0.50; asymptomatic and severe cases each reached 1.00.Source: Table 25 · p. 36, F Extended Expert Validation Results · p. 37
  • Disaggregated validation exposed weaknesses masked by averages, including attack-rate confusion and temporal outbreak features with group accuracy 0.62.The workflow with gpt-oss-120b confused attack rate with seroprevalence, while population-context fields were difficult to apply consistently across articles.Source: F Extended Expert Validation Results · p. 37

G Article Search and Retrieval

  • AgentSLR constructs a standardized epidemiological base query and adapts it across bibliographic databases to retrieve literature on transmission, severity, temporal, heterogeneity, and evolutionary domains.PubMed and Europe PMC support wildcard truncation, whereas OpenAlex requires expanded morphological variants.Source: G Article Search and Retrieval · p. 38

G.1 Base Search Query (PubMed and Europe PMC)

  • The PubMed and Europe PMC query uses Boolean groups and truncation operators to capture pathogen identifiers alongside epidemiological terms and optional inclusion or exclusion criteria.Its domains include transmission, modeling, severity, infectious periods, heterogeneity, infectivity, immunity, evolution, outbreaks, and risk factors.Source: §G.1 · p. 38

G.2 OpenAlex Adapted Queries

  • OpenAlex queries expand truncated terms into common morphological variants because its API does not support wildcard operators and strips those characters during processing.The expanded vocabulary covers transmission, severity, infectious periods, heterogeneity, infectivity, immunity, evolution, outbreaks, and risk factors.Source: §G.2 · p. 38

G.3 Pathogen-Specific Query Modifications

  • Pathogen-specific modifications preserve shared identifiers and exclusion criteria while adding SARS COVID-19 exclusions, Zika vector terms, and virus-specific identifiers or spelling variants.PubMed and Europe PMC retain wildcard forms, whereas OpenAlex uses expanded variants.Source: §G.3 · p. 39, Table 26 · p. 39

G.4 Metadata Extraction and Deduplication

  • Metadata extraction combines database-specific identifiers with five-level deduplication, and PDF retrieval cross-references identifiers before trying up to four open-access sources in priority order.Deduplication uses DOI, PMID, PMCID, OpenAlex ID, and title-year combinations; the first successful PDF retrieval stops the cascade.Source: §G.4 · p. 40

G.5.1 Implementation Details · G.6 Final Quality Control · H Article Screening Criteria and Prompts

The workflow validates and quality-controls retrieved PDFs before screening articles for epidemiological relevance. Screening uses staged abstract and full-text prompts with explicit inclusion, exclusion, and quantitative-data criteria.

G.5.1 Implementation Details

  • PDF downloads use 64 KB HTTP streaming, two-stage validation, a 500 MB size limit, and 16-worker concurrency with per-source rate limits.Validation checks PDF magic bytes and rejects HTML access-denial pages; invalid or oversized files are discarded immediately.Source: §G.5.1 · pp. 40–41
  • Retrieved PDFs are cached, checkpointed every 50 records, standardised by identifier-priority filenames, and augmented with source, timestamp, and error provenance.Caches store successful URLs and negative markers to avoid redundant API calls, while filenames prioritize PMID, PMCID, DOI hash, then title hash.Source: Table 29 · p. 41, §G.5.1 · p. 41

G.6 Final Quality Control

  • Deduplication and quality filtering remove records without abstracts, duplicate article IDs or DOIs, and files that fail validation.For duplicate DOIs, the first occurrence is retained.Source: §G.6 · p. 41

H Article Screening Criteria and Prompts

  • Article screening proceeds from abstracts to full texts using structured prompts containing objectives, criteria, article content, chain-of-thought instructions, and parsable output requests.Prompt sensitivity tests with open-source models motivated a modified ScreenPrompt formulation replacing yes/no with include/exclude, similar to Alhetelah and Ahmad.Source: H Article Screening Criteria and Prompts · pp. 42, 44
  • Inclusion requires the target pathogen, English peer-reviewed original research, an eligible human or parameter-reporting animal population, relevant epidemiological content, and extractable quantitative models or parameters.Eligible content covers outbreaks, transmission and timing measures, severity, genetic evolution, overdispersion, seroprevalence, risk factors, and vector parameters.Source: H Article Screening Criteria and Prompts · p. 42
  • Exclusion criteria remove non-target, non-English, non-eligible publication or study types, studies lacking required transmission parameters, accidental laboratory outbreaks, and, at full text, case reports with fewer than 10 human cases.Full-text screening also excludes literature reviews and meta-analyses, while abstract screening emphasizes broad inclusion of potentially relevant studies.Source: H Article Screening Criteria and Prompts · pp. 43–44
  • Prompts require criterion-by-criterion reasoning and a final parsable decision, outputting INCLUDE when inclusion is advised or uncertainty persists and EXCLUDE otherwise.Full-text prompts explicitly assess whether articles contain extractable quantitative data, models, or parameters relevant to disease transmission and outbreak response.Source: H Article Screening Criteria and Prompts · pp. 43–44

I Data Extraction Process … Outbreak report prompts

This section describes AgentSLR’s structured, multi-stage extraction workflow for epidemiological models and outbreak reports, including parameter-specific schemas, validation rules, pathogen-specific criteria, and provenance tracing. It also outlines the constrained, evidence-grounded process for generating outbreak reports from extracted records.

I Data Extraction Process

  • The extraction stage uses structured tool-calling with validation to capture epidemiological parameters from full-text articles.The broader workflow also covers transmission models and outbreak data, with validation applied to each tool output.Source: I Data Extraction Process · p. 45

I.1 Parameters

  • The parameter schema covers 47 epidemiological parameter types spanning mutation, transmission, delays, severity, seroprevalence, and risk factors. [11]The catalogue is based on PERG’s REDCap data-entry tool and defines extraction criteria at the parameter-class level.Source: §I.1 · p. 45
  • The five-step parameter pipeline screens full texts, extracts relevant quotations, and then standardizes parameter values, uncertainty bounds, metadata, and population context.The screening model uses parameter-class-specific guidance and is instructed to extract only estimates derived from or fitted to actual data.Source: §I.1 · pp. 46–47
  • Subsequent extraction runs on each identified summary rather than the full article, reducing inference time and API cost before structuring values and uncertainty bounds.If no relevant parameter is identified, the process terminates; otherwise, a new context performs value extraction using parameter-specific instructions.Source: §I.1 · pp. 47–48
  • The schema preserves measurement context by recording genome regions for genetic estimates, distinguishing delay types, and classifying central estimates by available uncertainty information.Genome-site fields can identify genes, segments, codon positions, whole genomes, or experimental inserts; central values prioritize means with SD, variance, or confidence intervals, then medians with IQRs or credible intervals.Source: §I.1 · pp. 48–50
  • Parameter extraction targets values estimated from actual data, using distinct tools and schemas for parameter classes before adding population context and aggregating suitable disaggregated estimates.The workflow excludes purely theoretical transmission-model parameters and uses a shared population-context tool, followed by aggregation under PERG’s rule of three and pathogen-specific location rules.Source: §I.1 · pp. 50, 51, 53, 55
  • Aggregation returns ranges for sufficiently disaggregated parameters, while preserving disaggregation types and avoiding aggregation when population information is inadequate or location rules prohibit it.For lassa, SARS, zika, and nipah, geographic disaggregation is generally not aggregated down to country administrative level 2, whereas neighborhood estimates still follow the rule of three.Source: §I.1 · p. 55

I.2 Models

  • Transmission-model extraction covers mechanistic models and uses controlled vocabularies to encode framework, compartmental architecture, stochasticity, routes, uncertainty, spatial structure, spillover, assumptions, and interventions.Purely statistical analyses, regression-based forecasts without transmission mechanisms, and risk-factor studies are excluded; the schema also records whether models were fitted to data.Source: §I.2 · pp. 56–57
  • The model schema additionally records code availability and programming language as reproducibility indicators, alongside categories of evaluated control measures.Intervention categories include vaccination, quarantine, treatment, contact tracing, vector or animal control, behavioural changes, and other specified measures.Source: §I.2 · p. 57
  • Model extraction screens articles for dynamic transmission models, then calls extract_model_data once per identified model to populate a validated standardized schema.The workflow excludes regression-only, purely statistical, and unimplemented models; controlled vocabularies and logical validation reject invalid outputs and prompt corrections.Source: §I.2 · p. 58

I.3 Outbreaks

  • Outbreak extraction captures epidemiological characteristics as reported, without calculations or inferred missing values, using a controlled schema for structured analysis.Fields cover temporal and geographic scope, case counts, deaths, transmission descriptions, and demographic breakdowns, with nulls permitted when information is unstated.Source: §I.3 · p. 60, Table 37 · p. 61
  • Pathogen-specific inclusion rules retain all reported Marburg and Lassa outbreaks, but require at least 10 cases for Zika and RVF and include even single OROV cases.For Zika, the threshold reflects concerns that smaller events may be incompletely documented and contribute little to population-level immunity estimates.Source: §I.3 · pp. 60, 64
  • Outbreak screening identifies concluded real-world events with defined case counts, then calls extract_outbreak_data once per distinct outbreak differentiated by location, time, or both.The workflow excludes ongoing, modeled, forecasted, routine-surveillance, and single-case reports, while validation enforces required fields, WHO country names, and data-type rules.Source: §I.3 · p. 63
  • Each extracted outbreak value is linked to supporting quotations, equations, or table citations, creating an auditable provenance trail for validation and downstream synthesis.Multi-select fields require independent textual support for every selected option.Source: §I.3 · p. 65

J Report Generation: Building Systematic Living Reviews

  • Report generation is outside AgentSLR’s evaluation because the available datasets cannot produce data-matched reference reports, and report generation and meta-analysis remain future work.Consulted epidemiologists identified screening and extraction as the primary workflow bottlenecks, while unconstrained summaries tended toward over-interpretation.Source: J Report Generation: Building… · p. 66

J.1 Deterministic Report Assembly

  • Outbreak reports aggregate schema-validated extraction records into summary tables, figures, Markdown, and PDF, with the lightweight build typically taking less than 5 minutes per pathogen.The report build is omitted from the main runtime breakdown because retrieval, screening, and extraction dominate runtime.Source: Inputs and derived artefacts · p. 66

Evidence packet construction

J.2 Evidence grounded narrative refinement

  • An LLM revision loop refines the draft under mandatory asset-presence and table-preservation checks, appending missing artefacts verbatim when necessary.Every required figure path must appear, and provided table values must remain unchanged.Source: §J.2 · p. 67, Self-refinement loop and… · p. 67

Outbreak report prompts

  • The prompts require evidence-linked outbreak surveillance writing, restrict interpretation to labelled AI-Interpretation blocks, and evaluate fidelity, coverage, traceability, clarity, completeness, interpretation, and formatting.Reports must preserve all figures and tables, avoid unsupported numbers or external facts, and cover temporal, geographic, severity, detection, demographic, and data-quality patterns.Source: Shared dimensions · p. 67, Outbreak report prompts · pp. 68–69

K Living Systematic Reviews with AgentSLR … M.2 Current Status and Field Testing

AgentSLR supports structured, human-audited epidemiological synthesis across WHO priority pathogens, from protocol-driven searching through extraction, meta-analysis, and living-review generation. Its outputs include validated reviews for four pathogens and preliminary summaries for five others, while beta field testing targets CCHF and RVF.

K Living Systematic Reviews with AgentSLR

  • Nine WHO priority pathogens received AgentSLR living reviews, with four validated against expert annotations and five remaining preliminary because PERG review has not yet commenced or is at an early stage.Each pathogen produces transmission-modelling and outbreak-surveillance documents alongside structured datasets and visualisations.Source: K Living Systematic Reviews with… · p. 72
  • 513 extracted Ebola models and 1,104 outbreak records were synthesised into complementary transmission-modelling and surveillance living reviews, with evidence-based descriptions and labelled AI-generated interpretation blocks.The reviews cover model architectures, stochasticity, code availability, outbreak timing, geography, and detection methods.Source: K Living Systematic Reviews with… · p. 72, Figure 7 · p. 72
  • 115 RVF models and 59 CCHF outbreak records were summarised in preliminary reviews, demonstrating evidence synthesis for understudied pathogens in under 48 hours despite lacking completed PERG validation.RVF results emphasised compartmental and vector-to-human models, while CCHF results mapped geographic and temporal outbreak patterns.Source: K Living Systematic Reviews with… · pp. 72–73, Figure 8 · p. 73

L The PERG Review Pipeline (Human Reference Workflow); Step 1: Paper search (protocol-driven, pathogen-specific); Step 3: Full-text review (confirm “extractability"); Step 5: Meta-analysis

Step 2: Title and abstract screening (broad triage against explicit criteria)

  • Title-and-abstract triage deliberately favors sensitivity, retaining English-language, peer-reviewed human studies containing epidemiological, modelling, outbreak, severity, evolutionary, serological, or transmission evidence while excluding specified nonprimary or unsuitable records.Explicit exclusions include non-English items, conference materials, in-vitro-only studies, most solely animal studies, and case studies with fewer than 10 cases.Source: Step 2: Title and abstract screening… · p. 74

Step 4: Parameter extraction (read, highlight, enter structured fields)

  • PERG extraction records article metadata, models, epidemiological parameters, contextual comparability fields, and structured outbreak characteristics exactly as reported, with wiki guidance supporting consistency across reviewers and pathogens.Reviewers read PDFs, highlight candidate information, and enter it into pathogen-specific REDCap projects without calculating missing values or inferring unreported fields.Source: Step 4: Parameter extraction (read,… · p. 75

M AgentSLR Annotation Tool (Beta); M.1 Human-in-the-Loop Validation

  • AgentSLR’s beta annotation tool combines LLM structured extraction with provenance-linked evidence and dual-panel expert verification, allowing reviewers to verify, modify, or reject individual predictions before synthesis.The interface adapts extraction fields to content types and records revised entries for quality control and system evaluation.Source: M AgentSLR Annotation Tool (Beta) · p. 76, §M.1 · p. 76, Figure 9 · p. 76

M.2 Current Status and Field Testing

  • The tool remains in beta after epidemiologist pilot testing, is designed to generalise beyond epidemiology through schema reconfiguration, and plans field testing for CCHF and RVF using standardised parameter, model, and outbreak schemas.These tests are aligned with the remaining priority-pathogen PERG workflow.Source: §M.2 · p. 77

M.3 Transparency and Reproducibility

AgentSLR supports transparent, reproducible synthesis by linking its structured database persistently to source text, enabling complete disaggregation and traceability to primary literature.

M.3 Transparency and Reproducibility

  • AgentSLR’s persistent database–source-text links create an audit trail that disaggregates reports and traces every reported parameter, model, and outbreak to its exact location in the primary literature.This supports scientific reproducibility.Source: §M.3 · p. 77

Related papers

Loading 2603.22327v2…