PaperlayerView exact revision on arXiv

Source-linked AI summary

Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews?

Aleksi Huotala, Miikka Kuutila, Mika Mäntylä

arXiv:2610.10633v1cs.SEcs.AI

TL;DR

Paper screening makes systematic reviews time-consuming, while earlier LLM results may not reflect newer models. The authors benchmark new LLMs and examine sampling, prompts, agreement, and criteria refinement. They find at most marginal performance improvement, with differences between studies exceeding those between models, and conclude that LLMs are not reliable enough to replace human screeners.

  • Problem

    Screening is a time-consuming systematic-review phase, and earlier benchmarks used one model generation, prompt, and output schema, leaving newer-model performance unclear.

    Source: §1 · p. 2

  • Method

    The study benchmarks new LLMs using SESR-Eval and a power-sampled smaller dataset, and examines prompts, decision aggregation, model agreement, and criteria refinement.

    Source: §4.2 · pp. 13–14, §7 · p. 44

  • Results

    At most marginal improvements followed the 18-month gap, while performance differences between secondary studies remained larger than differences between LLMs.

    Source: §7 · p. 44

  • Takeaways & Limitations

    LLMs are not yet reliable enough to replace humans in systematic-review screening, though they may serve as useful companions.

    Source: §7 · p. 45

    The evaluation reflects baseline prompting; the authors assume performance may improve with one-shot, few-shot, and few-shot chain-of-thought prompting.

    Source: §6.5 · p. 43

Abstract

from arXiv · show

Screening in systematic reviews (SRs) is manual and time-consuming. Prior work has explored large language models (LLMs) for automating this step, but LLMs are evolving rapidly, so earlier performance claims may no longer accurately reflect their screening performance. We used an existing software engineering SR screening benchmark (SESR-Eval) as our data. We also power-sampled a new, smaller dataset (SESR-Eval-Mini) that allows evaluation at lower costs. Using this data, we evaluated eight new LLMs for screening performance. Additionally, we tested different prompts, analyzed LLM agreement in screening decisions and criteria, and examined the effect of refining the inclusion and exclusion criteria on screening performance. The eight new LLMs performed marginally better than the seven old ones: avg. MCC across secondary studies rose from 0.347 to 0.365. Differences between secondary studies are still bigger than between LLMs. Computing the overall screening decision from criterion-level decisions degraded screening performance only slightly. LLMs generally agree with each other in their corresponding screening decisions (mean Gwet's AC1 = 0.830), though certain inclusion and exclusion criteria showed larger disagreement than others. Refining the inclusion and exclusion criteria slightly improved recall and made decisions easier for some LLMs, but overall impacts of criteria refinement were modest. LLMs are not yet ready to replace humans in paper screening and the advantages new, more costly models bring, appear to be very limited. Agent-based approaches, prompt engineering, and further criteria refinement are three potential future research avenues.

1 Introduction

Paper screening is a time-consuming part of systematic reviews, and earlier LLM benchmarks left open whether newer models, prompts, or output schemas would improve performance. This study benchmarks newer models and examines sampling, decision aggregation, model agreement, and criteria refinement.

1 Introduction

  • A single systematic review averages 67 weeks, and screening takes about 89 seconds per paper, making screening a substantial workload.Thousands of papers may require screening.Source: §1 · p. 2
  • Earlier SESR-Eval work benchmarked one generation of models using one prompt and output schema, leaving newer models and alternative designs untested.The full benchmark contains 34,528 papers, and newer models can be costly to run.Source: §1 · p. 2
  • The study benchmarks eight new models, evaluates two on full SESR-Eval and all eight on a 5% sample, and investigates prompts, output structuring, agreement, and criteria refinement.Its questions also examine criterion-level and overall screening decisions.Source: §1 · p. 3

2 Related Work

Prior work has evaluated LLM screening across software engineering and other fields, reporting varied performance and sensitivity to study and task conditions. The literature also leaves limited evidence on criterion-level behavior, inter-model agreement, sampling, and criteria refinement.

2 Related Work

  • LLM-assisted screening has been studied in software engineering, medicine, biomedicine, health technology assessment, veterinary medicine, and artificial intelligence.The review organizes prior work into software engineering and other disciplines.Source: §2 · p. 4

2.1 Software Engineering

  • Software engineering studies report varied screening performance, with differences between secondary studies often exceeding differences between models.Prior SESR-Eval work concluded that no model achieved high recall with reasonable precision; other studies reported domain-specific errors and criterion-structure effects.Source: §2.1 · p. 5

2.2 Other Disciplines

  • Studies in other disciplines report widely varying screening outcomes, and prompt or task choices can materially affect performance.Prompt changes affected performance in Dennstädt et al. (2024), while Adam et al. (2026) reported effects associated with prompt structure and input choice.Source: §2.2 · pp. 6–7

2.3 Summary

  • Prior work often treats screening as binary classification, with limited analysis of criterion-level decisions, inter-LLM agreement, sampling, or criteria refinement.Across studies, performance is also affected by criteria, prompts, domain-specific issues, and model cost.Source: §2.3 · p. 8

3 Background

The background frames title-abstract screening as decisions made against multiple inclusion and exclusion criteria, whose individual judgments must be combined. It also motivates examining whether decisions differ across criteria and raters.

3.1 Inclusion and exclusion criteria

  • Inclusion and exclusion criteria guide title-abstract decisions, should map to review questions, and should be formulated for reliable assessment.Criteria may be refined during the literature search process.Source: §3.1 · p. 8

3.2 Title-abstract screening as a classification task

  • Screening combines criterion-level judgments into an overall decision, commonly excluding a paper if any exclusion criterion applies or an inclusion criterion fails.Criteria can differ in how objectively they can be assessed, motivating analysis of agreement and refinement.Source: §3.2 · p. 9, §3.1 · p. 9
  • LLMs may generate criterion judgments alongside an overall decision, or the overall decision may be aggregated from criterion judgments; the latter allows flexible decision boundaries but risks false exclusions from a single error.Generating an overall decision separately can produce contradictions with criterion-level judgments, depending on generation order.Source: §3.2 · p. 9

3.3 Agreement between multiple raters

  • When individual reviewers are more than 50% accurate and decide independently, majority-vote accuracy increases with the number of reviewers.High agreement between LLMs is ambiguous because models may make correlated errors.Source: §3.3 · p. 10

4 Methodology

The study benchmarks newer LLMs against earlier models, creates a lower-cost sample of SESR-Eval, and tests prompt/schema changes and criterion-derived overall decisions. Its design also examines LLM agreement and criteria refinement.

4.1 LLM selection (all RQs)

  • Eight new and seven previously tested LLMs were selected, with new models chosen from eight publishers’ highest-global-average LiveBench models and mostly run at low reasoning effort.The earlier benchmark’s two poorest-performing models were excluded; two new models were tested on the full dataset, while all eight were tested on the sample.Source: §4.1 · pp. 11–12

4.2 Data sampling – SESR-Eval-Mini (all RQs)

  • The SESR-Eval-Mini dataset contains 1,726 papers, a 5% power-allocated sample using α = 0.5 from SESR-Eval’s 34,528 papers across 24 secondary studies.The sample’s overall included:excluded ratio was 21:79, compared with 12:88 in the full dataset; the RQ4 subset contained 448 papers from seven studies.Source: §4.2 · pp. 13–14, Table 3 · p. 13

4.3 Benchmarking the LLMs (all RQs)

  • The full-data benchmark reused seven earlier LLMs and tested two new ones; all eight new LLMs were benchmarked on the 5% sample for RQ1.2, RQ2, and RQ4.The other six new models were not run on the full dataset because the authors estimated that doing so would cost about $6,500.Source: Table 4 · p. 14, §4.3 · p. 15
  • The study assessed screening with accuracy, precision, recall, F1, MCC, and lost evidence, using raw model responses to derive overall and criterion-level decisions for agreement analyses.The full-dataset responses also supplied results derived for the sampled data in the agreement analyses.Source: §4.3 · pp. 15–16

4.4 Separating the computation of the overall screening decision (RQ2)

  • For RQ2 and RQ4, V2 refined the prompt and output schema, while V1 preserved the prior study’s prompt and schema for directly comparable RQs.V2 removed the overall-decision field, added criterion-name validation, separated inclusion and exclusion criterion classes, and clarified their probability outputs.Source: §4.4 · pp. 16–17, Figure 3 · p. 16
  • Overall decisions for RQ2 and RQ4 were computed from inclusion and exclusion criterion decisions using each study’s rules, supplied to LLMs as Boolean formulas.The criteria were converted into YAML files and checked against the originals before use.Source: §4.4 · p. 17

4.5 LLM agreement - Overall decision and criteria (RQ3.1 & RQ3.2)

The authors analyze agreement from LLM screening responses, test criteria revisions on difficult studies, and use metrics and statistical procedures suited to imbalanced data and comparisons across studies.

4.5 LLM agreement - Overall decision and criteria (RQ3.1 & RQ3.2)

  • Agreement analysis used parsed raw responses for overall binary decisions and criterion-level decisions, with Gwet’s AC1 chosen to avoid kappa’s prevalence problem.Gwet’s AC1 uses observed agreement and chance agreement; it does not artificially deflate agreement when one category is prevalent.Source: §4.5 · p. 17, §4.7.4 · pp. 20–21

4.6 Modifying criteria (RQ4)

  • For RQ4, eight new LLMs re-screened 448 papers from seven challenging secondary studies using the V2 prompt and output schema after criteria revisions were verified.The revised criteria were tested on 5% sampled data, with accuracy, precision, recall, F1, MCC, Gwet’s AC1, and lost evidence computed.Source: §4.6 · p. 18
  • Criteria revisions expanded acronyms, split compound rules, removed title/abstract-unverifiable criteria, converted negations, minimized rewording, and clarified vague terms from the SR.The revisions aimed to improve clarity and interpretability while preserving the underlying SR’s intended meaning.Source: §4.6.1 · p. 18, §4.6.2 · p. 19, §4.6.3 · p. 19

4.7 Statistical methods used

  • Accuracy, precision, recall, and F1 supported comparison with prior work, while MCC was reported for ranking and to account for class imbalance.MCC uses the full confusion matrix; the dataset’s included-to-excluded ratio was 12:88, and the 5% sample’s was 21:79.Source: §4.7.1 · pp. 19–20, §4.7.2 · p. 20

4.7.3 Lost evidence

  • Lost evidence counts and quantifies relevant studies incorrectly excluded, distinguishing LLMs whose F1 scores may be similar but false negatives differ.The authors note that irrelevant papers are usually caught during full-text screening.Source: §4.7.3 · p. 20

4.7.5 Bootstrap confidence intervals and regression model for statistical significance

  • Because composite F1 and MCC do not suit traditional significance tests, the authors estimated 95% percentile bootstrap confidence intervals from 2,000 within-study resamples.Paired intervals for same-primary-study conditions were treated as meaningful when they excluded zero.Source: §4.7.5 · p. 21
  • A logistic regression predicted screening-decision correctness from the LLM and primary study, using the highest-probability model and study as reference classes.Source: §4.7.5 · p. 21

4.8 AI disclosure

  • Claude Code supported script development and table generation, but authors independently reviewed and validated all outputs and retained responsibility.Source: §4.8 · p. 21

5.1 RQ1.1: Paper screening on full dataset

On the full dataset, the two newer models were marginally better on secondary studies but worse on primary studies than the older models. Performance also varied substantially across secondary studies, while the newer models were slower and more costly.

5.1 RQ1.1: Paper screening on full dataset

  • On primary studies, DeepSeek V4 Pro and Qwen 3.6 Plus did not exceed the old leaders' MCC of 0.46 or Llama 4 Maverick's F1 of 0.53.The two newer models each had F1=0.48; their lost-evidence shares were 53% and 48%.Source: Table 6 · p. 22, §5.1 · pp. 23–24
  • On secondary studies, DeepSeek V4 Pro, Qwen 3.6 Plus, GPT-4o, and Llama 4 Maverick tied at MCC=0.31, while average MCC varied widely by study.Across studies, MCC ranged from -0.11 to 0.64 and lost evidence from 18% to 85%.Source: Table 7 · p. 22, Table 8 · p. 23, §5.1 · p. 24
  • DeepSeek V4 Pro and Qwen 3.6 Plus were the slowest models at 103.0s and 94.7s per paper and cost $12.1 and $11.2 per secondary study.GPT-4.1 mini screened a paper in 10.5s.Source: §5.1 · p. 24
  • Across secondary studies, average MCC was 0.309 for new models versus 0.294 for old models; across primary studies, it was 0.405 versus 0.429.The newer models therefore did not show a consistent advantage across study types.Source: Table 7 · p. 22, Table 6 · p. 22, §5.1 · p. 24

5.2 RQ1.2: Paper screening on sampled dataset

On the sampled benchmark, new models were only marginally better on secondary studies, while old models remained better on primary studies. Differences among secondary studies remained substantial, and model costs and screening speeds varied.

5.2 RQ1.2: Paper screening on sampled dataset

  • The old GPT family led secondary-study F1 with a shared mean of 0.531, while DeepSeek V4 Pro and Qwen 3.6 Plus tied at 0.50.Across new models, per-study MCC ranged from -0.27 to 1.00.Source: §5.2 · p. 29
  • Screening one paper cost less than a dollar per secondary study for 11 models, but DeepSeek V4 Pro took 103.0s and Kimi K2.6 took 225.6s per paper.No model was faster than GPT-4.1 mini at 10.5s per paper.Source: §5.2 · p. 29
  • Across secondary studies, MCC averaged over LLMs ranged from -0.20 to 0.72, and F1 ranged from 0.15 to 0.87.The authors characterize secondary-study differences as larger than LLM differences.Source: Table 11 · p. 27, §5.2 · p. 30
  • Across secondary studies, average MCC was 0.365 for new models versus 0.347 for old models, but the primary-study comparison reversed.Claude Opus 4.7 led secondary-study MCC at 0.42, while primary-study average MCC was 0.412 for new models and 0.433 for old models.Source: Table 9 · p. 25, Table 10 · p. 26, §5.2 · pp. 29–30

5.3 RQ2: Computing the overall decision from criterion-level decisions

Computing the overall decision from criterion-level decisions reduced average screening performance and increased missed relevant papers. The authors conclude this stricter approach cannot be recommended because false negatives are more costly than false positives.

5.3 RQ2: Computing the overall decision from criterion-level decisions

  • Average accuracy, precision, recall, F1, and MCC decreased under criterion-level aggregation, with statistically significant reductions for some models and metrics.Lost evidence increased significantly for most LLMs.Source: §5.3 · p. 30, Table 13 · p. 31, Table 14 · p. 31
  • Recall fell from 0.41 to 0.38 when overall decisions were computed from criterion-level decisions, with false negatives increasing by 10 on average.The authors say this approach makes screening stricter but not more precise.Source: Table 13 · p. 31, Table 14 · p. 31, §5.3 · p. 32

5.4 RQ3.1: LLM agreement - Overall decision

LLMs generally agreed on overall screening decisions, with lower agreement on the sampled data than on the full dataset. The lowest-agreement pairs included GPT-4.1 or GPT-4o.

5.4 RQ3.1: LLM agreement - Overall decision

  • Mean Gwet's AC1 was 0.864 on the full dataset and 0.830 on the 5% sample; nine of 36 sampled-data model pairs had agreement below 0.80.Those nine pairs each included GPT-4.1 or GPT-4o.Source: §5.4 · p. 32

5.5 RQ3.2: LLM agreement - Individual criteria

Agreement varies substantially across individual criteria and secondary studies: exclusion criteria have higher mean agreement than inclusion criteria, while simple, objective criteria tend to align more consistently.

5.5 RQ3.2: LLM agreement - Individual criteria

  • 0.868 vs 0.778: Exclusion criteria had statistically significantly higher mean inter-model agreement than inclusion criteria.Source: §5.5 · p. 32
  • Inclusion-criterion agreement varied widely across studies, and about half of studies also showed substantial variation among exclusion criteria.For inclusion criteria, agreement ranged from 0.709 to 0.965 in Stradowski and Madeyski (2023), and from 0.065 to 0.834 in Alonso et al. (2023).Source: §5.5 · p. 32
  • Simple, objective criteria tended to yield higher agreement; agreement for excluding non-English papers ranged from 0.911 to 1.000 across studies.Source: §5.5 · p. 34
  • Some apparently straightforward criteria still had low agreement, including a prioritization criterion at AC1 0.259 and an MVP criterion at AC1 0.065.Source: §5.5 · p. 34
  • Criteria combining multiple conditions also showed low agreement, including examples with AC1 0.530 and 0.663.Agreement also depends on the screened papers; vagueness alone did not correspond to the same agreement level across criteria.Source: §5.5 · pp. 34–35

5.6 RQ4: Refining the inclusion and exclusion criteria

Refining screening criteria modestly improved some agreement and recall measures, but effects varied across models and studies. The overall changes were mixed rather than uniformly beneficial.

5.6 RQ4: Refining the inclusion and exclusion criteria

  • Refinement targeted studies with low screening accuracy and low criterion agreement, based on the assumption that increased LLM agreement would improve accuracy.Source: §5.6 · p. 35

5.6.1 Agreement – Gwet’s AC1

  • Mean Gwet’s AC1 increased from 0.847 to 0.866, although some refined criteria gained agreement while others lost it.The reported comparison includes criteria with statistically significant changes in either direction.Source: §5.6.1 · p. 35, Table 17 · p. 36
  • Some criteria tied to publication metadata were deleted because title and abstract screening could not determine them without metadata access.Other reworded criteria showed only small agreement changes, such as increases of 0.031 and 0.016.Source: §5.6.1 · p. 35
  • Splitting a multi-condition criterion raised AC1 from 0.475 to 1.000 for the resulting English-language criterion; expanding an acronym and flipping a criterion raised AC1 from 0.138 to 0.750.Source: §5.6.1 · pp. 37, 39

5.6.2 Screening accuracy

  • Lost evidence decreased by 6 overall, but model-level reductions ranged from 1 for GLM 5.1 to 21 for DeepSeek V4 Pro.Source: Table 18 · p. 37, §5.6.2 · p. 39
  • Average F1 rose from 0.16 to 0.19, while MCC fell from 0.12 to 0.11; recall improved for every tested LLM.Across studies, average recall rose from 0.32 to 0.37, while average accuracy did not improve.Source: Table 19 · p. 38, §5.6.2 · p. 39, Table 20 · p. 39

6 Discussion

Newer LLMs were at best marginally more accurate on the full benchmark, while a small-sample comparison found a narrow lead for Claude Opus 4.7. Across the discussion, the authors report modest effects from prompt and criteria changes, and identify several limits on interpreting the results.

6.1 LLM performance (RQ1)

  • On SESR-Eval-Mini, Claude Opus 4.7 reached MCC=0.46 versus MCC=0.45 for GPT-4.1 mini and Llama 4 Maverick, while newer models were at best marginally more accurate on the full dataset.DeepSeek V4 Pro and Qwen 3.6 Plus did not reach GPT-4.1 mini's F1=0.50 and MCC=0.46 on the full dataset; Claude Opus 4.7 was not tested there.Source: §6.1 · p. 40
  • Added SR context and criterion-focused agents are proposed improvement avenues, though multiple agents may increase costs by multiplying LLM calls per paper.The discussion also notes that human domain expertise and knowledge of SR goals may be difficult for LLMs to match.Source: §6.1 · pp. 40–41

6.2 Effects of prompt design and output structure on screening performance (RQ2)

  • Deriving overall screening decisions from criterion-level decisions reduced all five reported metrics, while V2 lost more relevant evidence and its performance declined slightly.The authors suggest that more radical prompt changes or system-prompt modifications may be needed for larger effects.Source: §6.2 · p. 41

6.3 LLM agreement with each other and across inclusion and exclusion criteria (RQ3.1 & RQ3.2)

  • Overall include/exclude agreement was AC1=0.864, but low correctness paired with high agreement limits potential gains from combining LLM decisions.Per-criterion agreement was include AC1=0.778 and exclude AC1=0.868.Source: §6.3 · pp. 41–42

6.4 Inclusion and exclusion criteria refinement (RQ4)

  • Mean F1 rose from 0.16 to 0.19 after criteria refinement, while some criteria became easier to interpret and compound rules sometimes obscured individual conditions.The authors recommend singular, clear criteria with explained acronyms, and caution against criteria relying on unavailable paper metadata.Source: §6.4 · p. 42

6.5 Limitations and Threats to Validity

  • Results cover only software engineering secondary studies and may not generalize to fields with different criteria conventions or abstract structures.Source: §6.5 · p. 42
  • Only zero-shot prompting was tested; prior work found OS, FS, and FS-CoT improved performance, while low-effort results are described as a lower bound.Six of eight models used low reasoning effort, and the evidence motivating that choice predates the evaluated models.Source: §6.5 · pp. 42–43
  • Some benchmark ground-truth labels may be wrong, and manual criteria refinements may not match original authors’ intent despite review by two authors.The authors also note human error cannot be ruled out in the laborious refinement process.Source: §6.5 · pp. 43–44

7 Conclusions and future work

The extended benchmark found only marginal LLM screening gains, while differences between secondary studies remained larger than differences between models. The authors conclude that LLMs cannot yet replace human screeners and outline agent-based screening and further prompting as future work.

7 Conclusions and future work

  • The paper extended SESR-Eval by benchmarking new LLMs, creating SESR-Eval-Mini, and examining prompts, output structuring, model agreement, and criteria refinement.Source: §7 · p. 44
  • Despite an 18-month gap, LLM screening performance improved at most marginally, leading the authors to conclude that reported LLM advances have largely stalled for software engineering screening.Source: §7 · p. 44
  • Differences in screening performance remained larger between secondary studies than between LLMs, so the authors recommend testing a broad selection of studies.Source: §7 · p. 44
  • Refining screening rules for poorly performing studies produced only modest improvements; the authors suggest study differences may have other sources, including dataset difficulty.Source: §7 · p. 44
  • LLMs are not yet reliable enough to replace human screeners, though the authors see them as useful companions and identify context and prompt design as bottlenecks.Source: §7 · p. 45
  • Future work includes distributing criteria across small-model agents, using probability thresholds to maximize recall, and testing prompting techniques with multi-agent systems.Source: §7 · p. 45
Loading 2610.10633v1…