Source-linked AI summary
A Benchmark Framework for Screening Automation in Systematic Reviews
Gauransh Kumar, Luciano Marchezan, Guillaume Genois, Kévin Delcourt, Eugene Syriani
TL;DR
SR screening is labor-intensive, and conventional evaluation metrics can mislead on highly imbalanced datasets. This paper presents SRBench, a curated benchmark with imbalance-aware evaluation, and PromptSR for experimentation and analysis; its use case demonstrates useful performance across several SRs while exposing failure cases.
Problem
Standard classification metrics can mislead in SR screening because relevant studies are a small fraction of candidates and false negatives and false positives have different consequences.
Source: §2 · p. 2Method
Results
With a minimal prompt configuration, LLM-based screening achieved useful performance across several SRs while exposing failure cases that conventional metrics could hide.
Source: §5.3 · p. 5Takeaways & Limitations
Performance variability across SR datasets indicates that approaches performing well in one review context may not generalize reliably to others.
Source: §5.3 · p. 6The use-case evaluation considers only one model and one baseline prompting strategy, so it is a proof of concept rather than an exhaustive performance assessment.
Source: §5.3 · p. 7
Abstract
from arXiv · showhide
Systematic reviews (SR) are essential for evidence-based research, but their screening phase is highly time-consuming and labor-intensive. Large language models (LLMs) offer a promising opportunity to reduce this workload by assisting with article relevance classification. However, existing evaluation approaches often rely on traditional metrics that may be misleading for highly imbalanced SR screening datasets. This paper presents a benchmark dataset of $45\,064$ labeled entries for evaluating LLM performance in SR screening across 32 curated secondary studies. It proposes an evaluation framework that accounts for class imbalance, i.e., the natural prevalence of excluded articles relative to included articles in SRs. It also introduces PromptSR, a tool designed to support prompt experimentation, experiment management, and result analysis for LLM-based screening. We also present a use case demonstrating the application of SRBench and PromptSR.
1 Introduction
Systematic-review screening is labor-intensive, and evaluating automation is difficult because screening datasets are imbalanced and existing benchmarks and tools leave gaps. The paper introduces SRBench and PromptSR and demonstrates their use for evaluating LLM-assisted screening.
1 Introduction
- SRs require multiple stages, including screening studies against predefined inclusion and exclusion criteria, making the process time-consuming and labor-intensive.LLMs are presented as a promising opportunity to improve review efficiency and consistency.Source: §1 · p. 1
- Imbalanced screening datasets can make conventional metrics misleading, including by inflating accuracy when a model predicts the majority class.The paper identifies a need for benchmarks that account for class imbalance and SR-screening priorities.Source: §1 · p. 1
- SRBench combines imbalance-aware evaluation metrics with 45 064 labeled studies from 32 secondary studies, while PromptSR supports experimentation and analysis.The framework is demonstrated in a use case evaluating LLM performance in SR screening.Source: §1 · pp. 1–2
2 Background and Related Work
Existing SR tools and benchmarks provide useful support, but they do not fully address controlled LLM screening experiments or the imbalance and asymmetric error costs of SR screening. These gaps motivate dedicated tooling and evaluation frameworks.
2 Background and Related Work
- Screening remains labor-intensive despite tools that assist review workflows; existing approaches may require substantial training data and generalize poorly across SR topics.Prior LLM studies also identify reliability, consistency, and generalizability as concerns.Source: §2 · p. 2
- Process-management, AI-assisted, and reviewer-trained tools lack infrastructure for systematically varying prompts across datasets with imbalance-appropriate evaluation.PromptSR is proposed to support reproducible prompt-engineering experiments for LLM-based screening.Source: §2 · p. 2
- Prior work on LLMs in evidence synthesis reports promise but limited large-scale validation, mixed performance, and sensitivity to prompting, validation, model selection, and human-in-the-loop strategies.SESR enables cross-study evaluation, but its assessment primarily uses standard classification metrics.Source: §2 · p. 2
- Standard classification metrics do not capture SR screening’s imbalance and asymmetric error costs: false negatives can omit relevant studies, while false positives add manual work.The authors argue that this mismatch can make performance conclusions overly optimistic or misleading.Source: §2 · p. 2
3 SRBench
SRBench combines evaluation metrics and a curated dataset of software-engineering SR studies to assess automated screening, including LLMs. Its construction extends SESR, standardizes and validates study data, and broadens coverage across review contexts.
3 SRBench
- SRBench combines evaluation metrics and labeled primary studies from published software-engineering SRs to assess automated screening tools, including LLMs.The benchmark also includes baseline results from a use-case application.Source: §3 · p. 2
3.1 Evaluation Metrics
- BAcc and MCC are the primary measures: BAcc treats both classes equally, while MCC accounts for all four prediction outcomes proportionally to class distribution.Recall, precision, NPV, and specificity complement them; accuracy is collected but not used as a primary indicator.Source: §3.1 · p. 3
3.2 Dataset
- The dataset extends SESR with 14 SRs, reaching 32 SRs and 45 064 primary studies, all with titles and abstracts.The reported corpus is smaller than Huotala et al.’s reported corpus because entries without abstracts were excluded from the replication package.Source: §3.2 · p. 3
3.2.1 Dataset construction.
- Dataset construction combined author outreach, metadata retrieval and standardization, and validation that corrected errors in the original SESR data.Authors provided 14 of 21 requested responses; validation identified duplicates, non-article items, unfindable entries, and non-English articles, among other issues.Source: §3.2.1 · p. 3
3.2.2 Data characteristics.
- The benchmark spans 35 ACM CCS categories and heterogeneous review sizes and inclusion rates, from ARCHIML at 0.015 to SE4MVP at 0.856.The added SRs introduce 11 categories missing from the original SESR dataset.Source: §3.2.2 · p. 3
4 The PromptSR Tool
PromptSR supports the workflow of configuring prompts, running screening experiments across SRBench datasets, and analyzing results. Its configurable features and modular components support comparison of prompt variants and screening approaches.
4 The PromptSR Tool
- PromptSR coordinates experiment configuration, screening execution across 32 datasets, and SR-specific metric computation without custom scripting.Its described modules include data processing, prompt templating, and asynchronous experiment scheduling.Source: §4 · p. 3
- The architecture is presented as modular, with a data-processing module and a template engine that produces the model prompt.The scheduler manages asynchronous execution; the figure is captioned as PromptSR’s technical architecture.Source: §4 · p. 3, Figure 1 · p. 4
- PromptSR’s configurable prompt features cover context, research questions, inclusion and exclusion criteria, few-shot examples, output formats, uncertainty, confidence, reasoning, and leniency.Features can be toggled individually or in combination for ablation studies.Source: §4 · p. 4
- The system supports multiple model backends and stores predictions, confidence scores, and evaluation metrics in a structured database for reproducibility.Experiments can run via a multithreaded scheduler locally or on HPC clusters; analysis includes tables and multiple plot types.Source: §4 · p. 5
5 Use Case Application
The use case runs a minimal prompt across all 32 SRBench datasets to demonstrate SRBench and PromptSR’s evaluation workflow. Results show useful screening performance alongside failures that imbalance-aware metrics reveal, while the discussion identifies limits in the evaluation and benchmark scope.
5.1 Procedure
- The use case configures exclusion-criteria prompts without few-shot examples, chain-of-thought, or uncertainty outputs, then screens all 32 datasets through PromptSR.Titles and abstracts are screened independently by Qwen3-30B-A3B-Thinking-2507; predictions, confidence scores, token use, and metadata are logged.Source: §5.1 · p. 5, §5 · p. 5
5.2 Results and Analysis
- MODELAUTOCODE4WIRELESS reaches BAcc 0.88 and MCC 0.75, while RL4SE has the highest BAcc at 0.91 with MCC 0.58.CODESMELDETECT and OPINIONMINESD each have MCC 0.49, and MPM4CPS has MCC 0.48.Source: §5.2 · p. 5
- LEARNSOFTCONFSPEC combines accuracy 0.971 and F1 0.985 with MCC 0.00 and BAcc 0.485, while TESTNN has MCC 0.00 and BAcc 0.211 despite F1 0.595.The authors attribute these cases to single-class prediction; failures tend to involve very small datasets or atypically high inclusion rates.Source: §5.2 · p. 5
- For MODELAUTOCODE4WIRELESS, SRBench’s results are accuracy 0.88 and F1 0.89, compared with SERS’s aggregate accuracy 0.56 and F1 0.53.SERS values aggregate results across its evaluated LLMs, and the paper cautions that accuracy and F1 do not fully capture screening performance.Source: §5.2 · p. 5
5.3 Discussion and Limitations
- SRBench spans 35 ACM CCS categories and supports studies of generalization across review settings, including prompt-feature ablations and comparisons of model families and workflows.PromptSR’s stored confidence scores and prediction metadata could also support research on calibration, selective prediction, and adaptive reviewer assistance.Source: §5.3 · pp. 6–7
- MCC and BAcc expose majority-class collapse that conventional accuracy and F1 can obscure, supporting more trustworthy interpretation of screening quality.The discussion highlights the asymmetric consequences of screening errors as a reason to account for class imbalance.Source: §5.3 · pp. 5–6
5.2 Results and Analysis
- 0.69 median BAcc and 0.32 median MCC across datasets; Table 2 presents the five best and worst cases ranked by MCC.The reported medians describe performance across the full benchmark, while the table excerpt focuses on selected cases.Source: §5.2 · p. 5, Table 2 · p. 7
5.3 Discussion and Limitations
- The use case tests only one model and one baseline prompt, while benchmark construction requires substantial manual effort and the benchmark covers title-and-abstract screening only.The authors describe the experiment as a proof of concept, not a definitive comparison; extending curated data to other domains requires effort and expertise.Source: §5.3 · p. 7
6 Conclusion
The paper presents SRBench, an imbalance-aware framework and dataset for evaluating screening automation in software engineering systematic reviews, alongside PromptSR for prompt experimentation and analysis. Together, they support reproducible, controlled evaluation across diverse SR datasets.
6 Conclusion
- SRBench uses imbalance-aware metrics and includes 45 064 labeled entries from 32 software engineering SRs.The framework is intended to provide a more trustworthy interpretation of screening performance.Source: §6 · p. 7
- PromptSR supports prompt experimentation, execution, and analysis for screening approaches.Source: §6 · p. 7
- Future work will expand SRBench to additional domains and review types, evaluate more LLMs and prompting strategies, and improve screening automation.Planned improvements include dataset curation and error handling in large-scale experiments.Source: §6 · p. 7