PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews

Shreyansh Padarha, Ryan Othniel Kearns, Tristan Naidoo, Lingyi Yang, Łukasz Borchmann, Piotr BŁaszczyk, Christian Morgenstern, Ruth McCabe, Sangeeta Bhatia, Philip H. Torr, Jakob Foerster, Scott A. Hale, Thomas Rawson, Anne Cori, Elizaveta Semenova, Adam Mahdi

arXiv:2603.22327v2cs.IRcs.AIcs.DL

TL;DR

Epidemiological systematic literature reviews are an important but underspecified, high-stakes setting for evaluating LLM scientific knowledge synthesis. The paper introduces AgentSLR, an expert-grounded workflow and dataset with stage-specific metrics, and finds specialization across tasks, a field-level extraction ceiling, and substantial deployment constraints. These results support human-supervised rather than unsupervised use in epidemiology.

  • Problem

    LLM evaluation for high-stakes systematic literature review evidence handling remains underspecified, despite epidemiology requiring context-sensitive synthesis across heterogeneous articles.

  • Method

    AgentSLR combines an SLR automation workflow with expert annotations from peer-reviewed WHO priority-pathogen reviews and stage-specific evaluation metrics.

  • Results

    No evaluated model dominates across the workflow, and no model exceeds an average field-level extraction F1 of 0.67.

  • Takeaways & Limitations

    The findings support human-supervised deployment in epidemiology and emphasize persistent limitations in structured extraction, error structure, cost and model availability.

  • Takeaways & Limitations

    The evaluation covers approximately 27.0% of the PERG corpus through open-access routes, uses English-only screening, and does not evaluate meta-analysis or final review writing.

Abstract

from arXiv · show

Systematic literature reviews (SLRs) are a demanding and high-stakes form of scientific knowledge synthesis that remains underspecified as an evaluation setting for large language models (LLMs). We introduce AgentSLR, a large-scale evaluation harness comprising an SLR automation workflow and an expert annotated dataset covering 16,248 articles, designed to test LLM capabilities across the stages of SLRs in epidemiology. Reference annotations were derived from peer-reviewed studies on WHO priority pathogens and produced by domain experts. The harness evaluates each review stage as a separate unit with dedicated metrics enabling targeted failure analysis. We evaluated five frontier reasoning models and found that no single model dominated across all tasks, showing sub-task specialisation often hidden by aggregate benchmarks. Structured data extraction is a major bottleneck, with no model exceeding an average field-level F1 of 0.67. Estimated costs vary substantially, by up to 96 times across evaluated models. Documented failure modes suggest that the evaluated models are not yet reliable enough for unsupervised deployment in epidemiology, where findings can inform public policy.

1 Introduction

AgentSLR frames epidemiological systematic literature reviews as a rigorous test of LLM evidence handling, from retrieval through synthesis. It combines expert references with stage-specific evaluation to expose reliability, context, and workflow-scaling problems.

  • SLRs require screening, extraction and synthesis across thousands of scientific articles.
  • LLM workflows can accelerate screening but still produce false inclusions, false exclusions and summaries that overgeneralize study findings.Errors can compound when models are chained across longer workflows.
  • Epidemiological SLRs test whether models associate reported parameters with the correct age group, geography or clinical severity.The same numbers may be reported differently across studies, so retrieval without contextual matching is insufficient.
  • AgentSLR uses verified expert annotations from WHO-designated priority-pathogen reviews as the human reference standard.
  • 16,248 article records, 3,808 parameter extractions, 687 transmission model extractions and 189 outbreak extractions comprise the released evaluation dataset.The harness covers retrieval, screening and structured extraction, with record-level extraction scoring.
  • No single evaluated model consistently dominates across workflow stages.The contribution statement also identifies refusal behaviour in closed systems as a failure mode relevant to scientific evidence synthesis.

2 Related Work

Prior work shows value in stage-specific LLM assistance but persistent degradation on complex extraction and domain-specific workflow limitations. AgentSLR extends this line with an expert-annotated epidemiological evaluation requiring context-sensitive, non-exact record matching.

  • LLMs can transfer screening logic across title, abstract and full-text stages without task-specific fine-tuning.Prior studies report high sensitivity and specificity across multiple systematic reviews.
  • Human-incorporated workflows generally outperform LLM-only approaches when data-extraction fields become complex.LLMs perform better on constrained schemas than on complex fields.
  • Existing aggregate review pipelines are often tailored to individual domains, motivating a dedicated epidemiological evaluation harness.Examples include bee ecotoxicology and Cochrane-style clinical reviews.
  • AgentSLR requires models to identify evidence families, apply appropriate schemas and recover values with uncertainty bounds and population context.Records may contain multiple estimates in disaggregated tables without canonical alignment to expert annotations.
  • AgentSLR is presented as the first evaluation of LLMs on epidemiological SLRs against expert annotations from peer-reviewed WHO priority-pathogen reviews.

3 Evaluation Harness

AgentSLR reproduces core evidence-handling stages and evaluates them with stage-isolated metrics against expert labels. Its extraction protocol separates detection, record volume and field correctness rather than relying on exact match.

  • AgentSLR combines a human-like epidemiological SLR workflow with an evaluation protocol mapping outputs to expert labels through stage-isolated metrics.
  • The workflow retrieves and deduplicates records, screens titles, abstracts and full texts, converts PDFs with OCR, and extracts structured evidence.Full-text screening applies stricter criteria for extractable epidemiological parameters and excludes specified study types.
  • Data extraction first flags evidence presence, then uses category-specific tool calls and population tagging for validated structured outputs.Population tags include age groups, geographic locations and clinical severity, supporting aggregation across contexts.
  • Report generation is outside the evaluation construct because generated reports require data-matched reference reports and separate narrative-synthesis tests.Interpreting aggregate public-health statistics also introduces safety risks beyond this study’s scope.
  • Screening metrics compare binary include/exclude decisions with human labels using precision, recall and macro F1 across funnel configurations.
  • Extraction evaluation separates presence detection, record counts and field-level content, allowing partial credit for partly useful records.Reference and generated records are matched within articles using weighted Jaccard similarity and one-to-one assignment.

4 Experiment Settings

The experiment uses expert annotations from priority-pathogen reviews to benchmark screening and extraction across pathogen-specific subsets. AgentSLR evaluates five reasoning-capable models with a released matched dataset and supports open- and closed-weight systems.

  • PERG’s peer-reviewed systematic reviews provide human-reference labels for the experiment.Seven pathogens contribute screening data, while four contribute parameter and transmission-model extraction data; outbreak extraction covers Lassa and Zika.
  • 16,248 downloaded article records matched to PERG labels are included in the released dataset.The dataset also contains structured parameter, transmission-model and outbreak extraction records.
  • Table 1 reports released benchmark record counts and identifies the matched downloaded-article subset.Its symbols indicate whether each priority-pathogen review was published, undergoing data extraction or awaiting screening as of March 2026.
  • AgentSLR evaluates GPT-5.2, gpt-oss-120b, Kimi K2.5, GLM-4.7 and DeepSeek-V3.2.The harness supports both open- and closed-weight models through schematised tool calls and requests.

5 Results

AgentSLR results show strong specialization across review stages rather than one universally dominant model, with structured extraction remaining the main bottleneck. Screening strategies and workflow costs also vary substantially, while expert validation supports human review rather than autonomous use.

  • Scientific Synthesis Is Not a Single Capability: Kimi-K2.5 leads title and abstract screening (F1 = 0.77), while gpt-oss-120b leads full-text screening (F1 = 0.87).No single model dominates across all stages.
  • Structured Evidence Extraction Is the Bottleneck Across Models: No LLM exceeds an average field-level extraction F1 of 0.67, making structured evidence extraction the principal cross-model bottleneck.Flagging data-type presence is more reliable than producing complete structured records.
  • Article Screening Strategies: Recall rises from 0.81 under automated two-stage screening to 0.92 when full-text screening is conditioned on human abstract decisions.The human-conditioned strategy also achieves F1 = 0.87 against PERG decisions.
  • Expert Review Separates Utility from Autonomy: Expert evaluation indicates that extraction outputs can provide a correctable starting point but require human review rather than full autonomous use.Experts assessed outputs alongside corresponding articles across parameter, model, and outbreak extractions.
  • Performance Does Not Scale Reliably with Cost: gpt-oss-120b achieves F1 = 0.70 at USD 13.9 per pathogen run, whereas GPT-5.2 costs USD 1,348.2 with F1 = 0.69.The reported cost difference exceeds 96×, showing that higher operating cost does not reliably predict higher performance.

6 Discussion

AgentSLR exposes substantial trade-offs between performance, cost, speed, and deployment risk in epidemiological SLR workflows. The results support assisted review rather than unsupervised expert replacement, while leaving important scope boundaries for future evaluation.

  • Key Findings: No evaluated model exceeds an average field-level extraction F1 of 0.67, with performance differences narrowing at extraction relative to screening.This convergence highlights the difficulty of schema-grounded extraction from heterogeneous scientific reporting.
  • Assisted Evidence Synthesis: 20 hours versus 385 human labour hours yields a 19.3× efficiency gain and 58× fewer calendar days for the full workflow.Full-text screening is under two seconds per article for the system versus four minutes for humans, or 118× faster.
  • Assisted Evidence Synthesis: LLMs can reduce evidence-triage time, but the authors state they are not yet at the capability threshold to replace expert judgment.The reported efficiency is conditional on stages meeting acceptable quality thresholds.
  • Assisted Evidence Synthesis: The short-term role proposed for LLMs is accelerating human-led review where missed evidence is costly, including full-text screening and supervised data extraction.The authors identify moderate-supervision use for parameter and outbreak extraction and suggest applications to living reviews.
  • Limitations and Future Work: The evaluation covers approximately 27.0% of the PERG corpus through open-access routes, uses English-only screening, and omits meta-analysis and final review writing.These boundaries limit direct interpretation for multilingual, closed-access, and later-stage review tasks.

7 Conclusion

AgentSLR frames epidemiological SLRs as a demanding evaluation setting for scientific evidence handling, addressing gaps in existing workflow benchmarks. Its stage-isolated, expert-referenced design supports targeted failure analysis rather than aggregate automation claims.

  • AgentSLR evaluates retrieval, screening and structured extraction against expert-curated annotations, enabling stage-level diagnosis.The framework uses human references and decomposes review production into separate components.
  • The harness targets evidence handling fidelity rather than complete scientific knowledge synthesis or final meta-analysis and report writing.Its scope is limited to preserving and structurally representing evidence through selected SLR stages.
  • Existing LLM-based SLR systems use incompatible domain-specific schemas, making direct quantitative comparison methodologically ill-posed.Prior systems address individual stages or bespoke clinical and biological workflows.
  • AgentSLR jointly provides open-code, open-weight, expert-referenced, independently evaluated and stage-specific workflow assessment.These properties support reproducibility, local deployment and component-level failure attribution.

A.2 Comparison with Benchmarks

AgentSLR occupies a distinct position among benchmarks for scientific evidence synthesis by evaluating epidemiological SLR evidence handling across stages and structured records. It complements, rather than replaces, clinical workflow, evidence-reasoning and screening benchmarks.

  • Clinical workflow evaluation: AgentSLR evaluates epidemiological parameters, transmission models and outbreaks, unlike clinical workflow benchmarks focused on interventions, study characteristics and outcomes.TrialReviewBench spans PRISMA-aligned clinical review stages, whereas AgentSLR targets WHO priority pathogen reviews.
  • Evidence reasoning: EvidenceBench and MedEvidence isolate retrieval or conclusion-matching skills but do not measure evidence survival through screening and heterogeneous record conversion.AgentSLR extends evaluation toward schema-grounded evidence handling.
  • Screening metrics: AgentSLR follows risk-aware screening recommendations while extending evaluation to flagging, count and field-level extraction metrics.This connects binary screening evaluation with structured extraction assessment.
  • Adjacent datasets: SciLitBench is omitted from detailed comparison because its public draft contains unresolved placeholders and undisclosed evaluation labels and design.The stated issue prevents a detailed methodological comparison.
  • AgentSLR positioning: AgentSLR is positioned between clinical SLR automation and biomedical evidence-reasoning benchmarks, not as a general deep-research-agent benchmark.Its contribution is epidemiological evidence handling with expert annotations, stage-isolated screening metrics and record matching.

B Data Representativeness & Ecological Validity

The evaluation uses PERG SLR data across WHO priority pathogens, but performance estimates are bounded by lawful article access and construct-validity constraints. The harness therefore emphasizes comparative interpretation and acknowledges limits from coverage, OCR, weighting and reference disagreement.

  • Data coverage: Screening covers up to seven WHO priority pathogens, while structured extraction covers four pathogens with published PERG SLR artefacts.The extraction set includes Ebola, Lassa, SARS-CoV-1 and Zika.
  • Data representativeness: 27.0% open-access overlap motivates a representativeness check against 1,004 closed-access articles from the broader PERG population corpus.The comparison directly tests whether open-access availability biases performance estimates.
  • Data representativeness: Full-text screening differs by −5.8 percentage points between the open-access and population samples, while abstract screening and parameter extraction have overlapping confidence intervals.Table 6 reports stage-level macro F1 with 95% bootstrap confidence intervals.
  • Access boundary: Absolute performance estimates should be interpreted relative to the lawfully retrievable evaluation corpus.The released materials exclude OCR text or PDF-derived Markdown when redistribution is not permitted by source licences.
  • Evaluation design: Macro-F1 weights included and excluded articles equally, and recall is reported because missed relevant articles propagate costly screening errors.Screening is evaluated against PERG reference labels using article-level precision, recall and F1.
  • Construct validity: AgentSLR measures evidence handling fidelity, not meta-analytic synthesis or report writing, so its results bound only one SLR component.Additional variance arises from OCR error, prompt formulation and field-weight choices, while PERG labels may contain reviewer disagreement.
  • Evaluation design: Full-text screening uses separate AI- and PERG-conditioned configurations to distinguish stage-specific from holistic performance.Abstract decisions can force final exclusion depending on the evaluation configuration.

C.2 Data Extraction

AgentSLR evaluates extraction with separate flagging, count, and field-level measures, using schema validation and similarity-based matching to compare LLM outputs with expert references.

  • Three measures—Flagging, Count, and Extraction—use precision and recall to evaluate relevant-data identification, extraction volume, and field contents.
  • Optimal one-to-one correspondences match reference and LLM extractions within each article using weighted field similarity rather than exact record identity.The matching maximises total similarity with a modified Jonker–Volgenant algorithm.
  • An unmatched LLM extraction incurs a Count false-positive penalty but does not contaminate field-level Extraction scores.The example matches two reference models to two of three LLM extractions, leaving the spurious extraction unmatched.
  • Parameter extraction resolves nine parameter classes and weights fields so value, uncertainty, and population-context sub-stages contribute equally to similarity.The key fields include parameter class, value, unit, uncertainty, and population context.
  • Reference-data validation removed invalid entries ranging from 3.85% for Lassa to 23.14% for Zika, while AI-extracted transmission-model and outbreak annotations showed 0% invalid entries.The AI-extracted zero-invalid result is attributed to structured schema enforcement.

D Pipeline Statistics: Data Processed & Time

Pipeline statistics quantify workload, runtime, tokens, and cost across successive SLR stages, showing that automation sharply reduces processing time while parameter extraction dominates compute and cost.

  • 9,132 articles undergo title-and-abstract screening on average, 1,102 undergo full-text screening, and 395 reach data extraction.These counts represent successive filtering stages and annotator workload rather than final inclusion totals.
  • 197.50 hours is the estimated human time for data extraction, exceeding title-and-abstract screening at 114.15 hours and full-text screening at 73.47 hours.
  • 13.4 hours is the estimated AgentSLR runtime for data extraction, compared with 1.6 hours for article retrieval and title-and-abstract screening and 0.62 hours for full-text screening.The workflow timing uses gpt-oss-120b.
  • Stage costs combine mean per-article input and output tokens, average article counts, and model-specific published API pricing.
  • Parameter extraction dominates overall compute and cost because it uses substantially more input and output tokens than other stages.Title-and-abstract screening processes the most articles but contributes comparatively little to total cost.

E Extended Evaluation Results

Extended results show substantial task specialization: model rankings change across screening and extraction tasks, while structured extraction remains uneven across fields and epidemiological contexts.

  • Kimi-K2.5 leads title-and-abstract screening with F1 0.77, whereas gpt-oss-120b leads full-text screening with F1 0.77.The title-and-abstract spread is driven largely by recall, while gpt-oss-120b’s full-text advantage comes from recall.
  • Kimi-K2.5 and GLM-4.7 achieve the highest average parameter-extraction F1 at 0.63, while GPT-5.2 leads field-level Extraction with F1 0.59.Cross-model average differences are attributed mainly to flagging and counting behaviour.
  • GLM-4.7 leads transmission-model extraction with average F1 0.85, while GPT-5.2 leads outbreak extraction with average F1 0.77.
  • Parameter flagging reaches recall 0.92 but precision 0.51, whereas parameter counts reach precision 0.83 but recall 0.47.Flagged parameters can be discarded through structured tool calls, but multiple extractions of one class are often underestimated.
  • Transmission-model counts show recall 0.97–1.00 but precision 0.48–0.60, indicating systematic overestimation of models per article.
  • Field difficulty varies by modality: core transmission-model characteristics are strong, while assumptions, interventions, and transmission routes are weaker.Overall precision ranges from 0.61 to 0.70 and recall from 0.75 to 0.81 across screening and extraction tasks.
  • Outbreak extraction is strongest for countries, death counts, and temporal features but weaker for specific locations and context-dependent fields.Country identification reaches 1.00 precision and recall for both pathogens, while specific-location precision is 0.50 for both.
  • Human abstract →AI full-text screening achieves precision 0.83 and recall 0.92, outperforming AI abstract →AI full-text on recall at 0.81.Direct AI full-text screening raises recall to 0.89 but lowers precision to 0.68.

F Extended Expert Validation Results

Expert validation audited AgentSLR outputs across parameters, transmission models, and outbreaks, revealing strong performance on some fields but substantial weaknesses in relevance and temporal or contextual extraction.

  • Validation design: Six experts audited 62 parameter, 50 model, and 31 outbreak submissions against source articles, evaluating relevance and populated-field correctness.The survey audited harness outputs rather than re-annotating the PERG reference set.
  • Outbreaks: 0.61 overall flagging precision and 0.80 overall extraction accuracy were reported for outbreaks.These aggregate scores summarize expert-rated performance for the outbreak category.
  • Outbreaks: 0.95 country and 0.80 location extraction accuracy contrasted with 0.50 end-year and 0.50 duration-in-months accuracy.Geographic fields were substantially more accurate than several temporal fields.
  • Outbreaks: 1.00 accuracy was achieved for asymptomatic and severe cases, while suspected cases reached 0.64 and deaths 0.71.Case-burden fields varied considerably in expert-rated extraction accuracy.
  • Scoring: Flagging precision was averaged over sampled parameter classes, while extraction accuracy was normalised across groups of similar fields.The sampled parameter classes excluded mutation rate and relative contribution because they were absent from the random subsample.
  • Parameters: Parameter population context and temporal outbreak features were difficult, with temporal group accuracy reported at 0.62.Experts also observed confusion between attack rate and seroprevalence in parameter flagging.

G.5.1 Implementation Details

The implementation details define database-aware retrieval, PDF validation, staged screening, and explicit epidemiological inclusion and exclusion criteria before extraction.

  • Retrieval: PDF downloads are validated through magic-byte checks and HTML access-denial inspection, with files over 500 MB or failing validation discarded.Downloads use 64 KB streaming chunks and concurrent processing with 16 workers while respecting source rate limits.
  • Retrieval: Records lacking abstracts, duplicate article IDs or DOIs, and files failing validation are removed during quality filtering.For duplicate DOIs, the first occurrence is retained.
  • Screening: Screening proceeds from titles and abstracts to full text using predefined inclusion and exclusion criteria.The prompts require all inclusion criteria and exclusion when any exclusion criterion applies.
  • Inclusion criteria: Included studies must be English-language, peer-reviewed original research concerning the target pathogen and containing extractable quantitative parameters, models, or outbreak data.The full-text criteria include human subjects, with limited animal-study exceptions for relevant transmission or vector parameters.
  • Screening rationale: The workflow preserves broad abstract-stage coverage because relevant evidence may only become clear in the full text.The stated aim is inclusive screening that filters out clearly irrelevant studies while retaining potentially pertinent evidence.
  • Exclusion criteria: Studies are excluded for non-English publication, non-primary formats, in-vitro-only designs, unsupported animal studies, small case reports, or accidental laboratory outbreaks.The full-text prompt defines small case studies as those with fewer than 10 human cases.

I Data Extraction Process

AgentSLR extracts epidemiological parameters, transmission models, and outbreaks through schema-constrained, multi-stage processing that first flags relevant evidence and then captures structured values and context.

  • Scope: Three extraction categories are processed: epidemiological parameters, transmission models, and concluded outbreaks.Each category uses a multi-stage workflow with validation on tool outputs.
  • Parameter schema: The reference catalogue contains 47 epidemiological parameter types, including reproduction numbers, seroprevalence, risk factors, and severity measures.Definitions distinguish basic and effective reproduction numbers and case-fatality from infection-fatality rates.
  • Parameters: A five-step parameter pipeline identifies relevant estimates, standardises them, and records population context and uncertainty from full-text articles.The pipeline is designed to mirror careful human reading and extraction.
  • Parameters: Parameter screening uses tool-calling to find potentially discontiguous quotations, then extracts values, uncertainty intervals, and study populations from focused snippets.Separate summaries are returned when multiple estimates of the same parameter occur.
  • Extraction rules: Extraction is restricted to parameters estimated from or fitted to actual data, excluding purely theoretical transmission-model parameters.The model is instructed not to call the extraction tool when no relevant information is found.

K Living Systematic Reviews with AgentSLR

AgentSLR generated structured living reviews for WHO priority pathogens, combining transmission-model and outbreak-surveillance documents while distinguishing validated outputs from preliminary syntheses.

  • Living reviews: Nine WHO priority pathogens received AgentSLR living reviews, each comprising transmission-modelling and outbreak-surveillance documents.The reviews were generated from the corpus extracted with gpt-oss-120b.
  • Validated examples: The Ebola transmission review summarised 513 extracted models, while its outbreak review synthesised 1,104 records spanning nearly six decades.The model review covered architecture, stochasticity, and code availability distributions.
  • Preliminary reviews: RVF and CCHF received preliminary reviews before PERG systematic screening began for those pathogens.These outputs demonstrate workflow capacity but lack the validation rigour applied to validated-pathogen reviews.
  • Preliminary reviews: Preliminary evidence summaries for RVF and CCHF were generated in under 48 hours of wall-clock time.The text frames these syntheses as potentially useful for resource allocation and hypothesis generation.
  • Review structure: Both review types use consistent schemas covering model characteristics, transmission pathways, reproducibility, outbreak timelines, geography, detection, and case counts.A standardised artefact structure is maintained across pathogen reviews.
  • Expert workflow: PERG’s end-to-end expert workflow spans protocolised search, two-stage screening, REDCap extraction, meta-analysis, and practical review write-up.AgentSLR’s living-review outputs are therefore situated relative to an expert-led evidence-synthesis process.

M.3 Transparency and Reproducibility

AgentSLR maintains a persistent link between structured data and source text, making synthesized reports fully disaggregable and supporting reproducible tracing to primary literature.

  • M.3 Transparency and Reproducibility: Persistent links connect the structured database to the source text, allowing reports to be fully disaggregated.This design keeps synthesized outputs connected to their underlying evidence.
  • M.3 Transparency and Reproducibility: The audit trail lets researchers trace every reported parameter, model, and outbreak to its exact location in the primary literature.The passage identifies this traceability as critical for scientific reproducibility.
Loading 2603.22327v2…