Source-linked AI summary
SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews
Miguel Zabaleta, Baihan Lin
TL;DR
LLM evaluations for systematic reviews often isolate stages, leaving unclear where automation is dependable across the full evidence-synthesis pipeline. SciLitBench evaluates screening and schema-guided extraction together, finding that protocol design supports reliable screening while open-ended extraction remains substantially less reliable.
Problem
Systematic-review stages differ in reliability because fixed bibliographic attributes are easier to extract than researcher-defined categories and open-ended evidence.
Method
SciLitBench benchmarks title and abstract screening, full-text screening, and schema-guided extraction across a multi-stage review corpus, using reproducible open-weight model evaluations.
Results
The benchmark reveals a practical boundary: protocol design and recall-oriented selection can support screening, whereas no evaluated model recovered more than approximately one third of annotated evidence in two open-ended fields.
Takeaways & Limitations
Evaluating evidence-synthesis systems requires measuring study preservation, evidence recovery, uncertainty and omissions, and reproducibility rather than plausible review outputs alone.
Takeaways & Limitations
The benchmark is derived from a single review of computational methods for literature-review automation, so its design principles should not be treated as universal laws.
Abstract
from arXiv · showhide
Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically examine review stages in isolation. We introduce SciLitBench, a multi-stage benchmark spanning title and abstract screening, full-text screening, and schema-guided data extraction, with 42,981 retrieved records, 1,012 full texts, and annotations for 888 included papers. Across 22 open-weight LLMs from six model families, explicit inclusion and exclusion criteria improve title and abstract screening $F_2$ by 28.8\%, while researcher-authored rationales improve full-text screening by 15\%. Data extraction reveals a different reliability regime: performance declines from 0.97 accuracy for publication year to 0.37 Jaccard overlap for computational approach, while the strongest models recover only 30\% of annotated evaluation evidence and 25\% of limitations. SciLitBench identifies a practical boundary between high-recall screening and evidence-complete extraction and provides a reproducible resource for evaluating LLM-assisted evidence synthesis.
1 Results
SCILITBENCH evaluates LLMs across screening and extraction in a unified review pipeline, finding that protocol design improves screening while open-ended extraction remains substantially less reliable. Reliability declines from straightforward bibliographic fields to ambiguous evidence recovery.
- Title and abstract screening: 17.0% relative F2 gain came from explicit inclusion prompting, rising to 28.8% when explicit exclusion reasoning was added.The improvements held across model families and sizes without requiring fine-tuning or additional compute.
- Title and abstract screening: F2 reached 0.808 with threshold-3 voting in the strong pool and 0.803 with threshold-9 voting in the lenient pool.Both configurations outperformed their averaged baselines and best single models.
- Full-text screening: Researcher-authored rationales increased full-text screening F2 from 0.551 to 0.633, a 15% relative improvement.The rationales paired inclusion decisions with quoted evidence and concise justifications.
- Data extraction: Data extraction performance fell from 0.97 accuracy for publication year to 0.37 Jaccard overlap for approach, while calibrated recall reached only 0.30 for evaluation results and 0.25 for limitations.The extraction fields span low-ambiguity attributes, schema-dependent categories, and open-ended evidence.
- End-to-end reliability: In a cascade, 97.6% of relevant records survived title and abstract screening, but only 4 of 33 papers were fully correct across extraction fields.This corresponded to 12.1% complete-extraction accuracy conditional on reaching extraction.
2 Discussion
SCILITBENCH finds that reliable screening depends on protocol design and recall-oriented operation, while evidence-complete extraction becomes less reliable as targets require interpretation. The benchmark therefore frames automation as a researcher-controlled system whose reproducibility and evidentiary visibility matter alongside model performance.
- Reliability boundary: Screening can be made dependable through protocol design and recall-oriented model selection, whereas evidence-complete extraction remains substantially less reliable for interpretive targets.Screening and extraction fail differently: screening preserves relevant records under asymmetric error costs, while extraction requires defining, representing, and comparing evidence.
- Protocol design: Explicit criteria, researcher-authored rationales, calibrated voting, and context-aware model selection can improve screening reliability beyond model scale alone.The relevant evaluation unit is the review system, including instructions, evidence representation, aggregation, and human verification.
- Screening trade-offs: Screening has no universally optimal threshold because false negatives can remove relevant studies, while false positives primarily increase reviewer workload.A useful system should expose the recall–precision trade-off and let researchers choose an operating point according to the consequences of omission.
- Extraction reliability: Open-ended extraction deteriorates from canonical attributes toward approaches, evaluation findings, and limitations that require researcher-defined categories and semantic interpretation.For the two open-ended fields, no evaluated model recovered more than approximately one third of annotated evidence, and some precise outputs covered only a small subset.
- Data release: Harmonization preserves raw human and machine annotations while mapping them into analysis-ready variables for downstream study beyond model benchmarking.This downstream release step makes the extracted dataset queryable while retaining the underlying evidence.
- Researcher control: Human judgment moves upstream into criteria, schemas, rationales, granularity decisions, and adjudication rather than disappearing from the review process.LLMs may reduce repetitive work while epistemic decisions remain inspectable and researcher-controlled.
- Reproducibility: Open-weight models support reproducibility because their evaluated state, prompts, inference settings, and outputs can be preserved for rerunning comparisons.This does not make them intrinsically unbiased or scientifically valid; training data, implementations, quantization, and model families remain sources of variation.
- Scope: Because SCILITBENCH derives from one review and uses researcher-in-the-loop schemas plus calibrated LaaJ evaluation, its design principles are corpus-bound rather than universal laws.The released annotations, schemas, prompts, calibration artifacts, and raw outputs allow future work to test alternative definitions and evaluation procedures.
3 Methods
The benchmark uses a unified, staged workflow to evaluate open-weight LLMs for screening and schema-guided extraction, with frozen protocols, audits, calibrated metrics, and released artifacts.
- Corpus and splits: 42,981 candidate records were searched, while title and abstract screening used a manually annotated 2,000-record seed set and full-text screening used 200 manually annotated papers.Frozen development, few-shot, and evaluation splits support reproducible assessment.
- Benchmark design: SCILITBENCH evaluates 22 open-weight LLMs across standardized screening and extraction protocols, using multiple model families and capacity ranges.Models were evaluated under frozen zero-shot and few-shot prompts, with stage-specific procedures.
- Evaluation criteria: F2 was the primary screening-selection metric because false negatives can remove relevant studies from downstream review stages.Configurations were first required to achieve perfect recall, after which the highest-precision system was selected.
- Screening workflow: The title-and-abstract workflow combined model evaluation, voting-pool selection, high-recall curation, and manual re-annotation of predicted positives.The selected voting configuration used three strong-model votes and four lenient-model votes, producing 1,767 additional included records.
- Data extraction: Data extraction covered six fields across 888 included full texts, ranging from low-ambiguity publication year to open-ended evaluation results and limitations.Closed fields used accuracy or Jaccard overlap, while open-ended fields used calibrated recall and precision.
- Calibration: Calibration classified model-field-metric estimates as supported, unstable, or unsupported; 69 of 88 combinations were supported.Support required observed calibration classes, an adjusted judge signal above 0.10, and confidence-interval width below 0.50.
- Reproducibility: The benchmark releases experiments, artifacts, and a harmonized dataset for comparing systems, auditing behavior, and studying review-related evidence.Released materials include frozen splits, prompts, calibration labels, and supporting documentation.
A Search query
The search query combines review-related terms with broad artificial-intelligence and language-model terms to retrieve literature on automated evidence synthesis.
- Review terms: The review component includes systematic, scoping, narrative, umbrella, rapid, integrative, and other review terms, plus evidence synthesis and meta-analysis.These terms are searched in title, abstract, and keyword fields.
- Query structure: The query links the review and AI term groups with a Boolean AND condition.A record must match terms from both conceptual groups to enter the search result set.
- AI terms: The technology component includes large language models, NLP, artificial intelligence, transformers, and named model families and systems.The query includes variants such as LLM, GPT, Llama, Mistral, BERT, and biomedical model names.
B Prompts
The prompts progressively add explicit inclusion and exclusion rules, while full-text prompts vary in structure and reasoning instructions for identifying AI-assisted review processes.
- B.1 Title and abstract screening: The basic title-and-abstract prompt asks whether a paper uses AI to automate any part of a scientific review paper.It requires a yes-or-no answer only.
- B.1 Title and abstract screening: The expanded title-and-abstract prompt defines inclusion as algorithmic execution of an evidence-synthesis step using AI, ML, NLP, or LLMs.Examples include search, deduplication, screening, extraction, study classification, snowballing, and meta-analysis.
- Prompt comparison: The prompt variants differ in how explicitly they encode inclusion, exclusion, and reasoning requirements.This design permits comparison of structured and unstructured instructions across screening stages.
- B.1 Title and abstract screening: The title-and-abstract appendix labels the expanded configuration as an Inclusion + Exclusion System Prompt.The prompt explicitly adds exclusion guidance for out-of-scope literature reviews.
- B.2 Full-text screening: The unstructured full-text prompt asks the model to analyze the paper and discuss whether it uses AI to automate part of a scientific review process.It provides an expert role but no numbered reasoning procedure.
- B.2 Full-text screening: The chain-of-thought full-text prompt decomposes the decision into review identification, AI-use assessment, and a final conclusion.The numbered sequence makes the required reasoning stages explicit.
C Benchmark results
Benchmark results compare ensemble-based screening with closed-field extraction, showing different evaluation structures for title-and-abstract screening, full-text screening, and field-specific extraction.
- Title and abstract screening: Table 2 reports recall-weighted F2, precision, and recall across title-and-abstract prompting, few-shot, and ensemble configurations.Strong and lenient voting pools are combined, and the selected curation system must achieve recall = 1.0.
- Full-text screening: Table 3 compares full-text models and routing ensembles using F2, with Llama 3.3-70B in zero-shot under the unstructured prompt as the best configuration.None of the evaluated ensembles improves over that single model in this setting.
- Closed-field extraction: Closed-field extraction scores publication year with accuracy and domain, review stage, and approach with Jaccard overlap on 888 papers.The best open-weight row reports the highest value separately for each field rather than one deployable model.
- Open-field extraction: Table 5 reports calibrated recall and precision for open-field extraction, with unsupported estimates marked by dashes.Only estimates meeting the stated support criteria are treated as supported in the main analysis.
D Additional performance analysis
Few-shot prompting produces modest, heterogeneous changes in title-and-abstract screening, while chain-of-thought prompting yields a modest but family-dependent gain in full-text screening.
- Few-shot prompting yields modest, heterogeneous F2 changes in title-and-abstract screening as in-context examples increase from 0 to 2 to 5.Some families improve slightly, whereas others show no benefit or slight regressions.
- 0.681 → 0.708 F2: Chain-of-thought prompting modestly improves full-text screening overall.The comparison is between unstructured and criteria-focused two-step zero-shot prompting.
- Gemma, Qwen, and Phi improve with chain-of-thought prompting, while GPT, Llama, and M show slight degradation.The effects vary across model families rather than moving uniformly in one direction.