PaperlayerView exact revision on arXiv

Source-linked AI summary

Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews

Arya Rahgozar, Pouria Mortezaagha

arXiv:2608.14551v1cs.CLcs.DLcs.IRcs.LG

TL;DR

LLM title-abstract screening lacks structured uncertainty, limiting reviewers’ ability to identify ambiguous decisions. This paper evaluates a BERT+GCN uncertainty signal and prompt-delivery strategies, finding that full-context delivery improves screening efficiency while MAYBE-only routing offers the best cost-benefit trade-off.

  • Problem

    LLM screeners lack calibrated, structured uncertainty, making it difficult for reviewers to identify decisions needing human verification.

  • Method

    Across eight benchmark reviews and 600 fold-level runs, the study combines a BERT+GCN uncertainty classifier with five LLM prompt-delivery strategies.

  • Results

    +0.050 WSS@95 improvement followed full-context delivery over baseline at a 1.28× token-cost premium, with recall preserved.

  • Takeaways & Limitations

    MAYBE-only routing is the recommended cost-sensitive configuration, while two-pass self-triage is ruled out by its 0% decision-flip rate.

  • Takeaways & Limitations

    Generalisation beyond drug-class reviews, including social and environmental science reviews, remains untested.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for title-abstract screening in systematic reviews, but their decisions lack calibrated uncertainty. We show that an auxiliary BERT+GCN classifier supplies a structured uncertainty signal that improves LLM screening efficiency, and we identify the prompt-delivery strategy that maximises the benefit-to-cost ratio. We evaluate five LLM prompt-delivery conditions on eight drug-class datasets from the Cohen (2006) benchmark using 3 seeds x 5-fold stratified cross-validation (600 fold-level results). A BERT+GCN model trained per fold classifies each test paper as INCLUDE, EXCLUDE, or MAYBE via two spectral tests (algebraic radical and categorical paradox). Conditions vary information content (none / label / full scores), selectivity (all papers vs. MAYBE only), and timing (proactive vs. reactive two-pass). A cross-model pilot against gpt-4.1-mini on three datasets tests cross-generation transfer. Three findings: (i) Full-context delivery yields significant gains in F1 (+0.011, paired Wilcoxon p=0.008) and WSS@95 (+0.050, p=0.039) at a 1.28x token-cost premium, while preserving recall. (ii) MAYBE-only routing is Pareto-optimal: highest mean recall (0.92) and AUC-ROC (0.54) at only 1.05x baseline cost -- one sixth of full-context overhead. (iii) The two-pass design escalates 22.2% +/- 8.8% of records yet never revises its decision (0% flip rate across all datasets and folds), giving decisive evidence that current instruction-tuned LLMs cannot self-triage. The cross-model pilot shows an identical +0.8% recall uplift for both LLM generations. A per-paper ablation across 20,796 observations shows the dual paradox test reduces empirically to a one-line logit-gap criterion. We release the full pipeline; the 600-run experiment replays in under one hour from cached LLM responses.

1 Introduction

This study addresses the screening bottleneck in systematic reviews by supplying LLMs with structured uncertainty from a BERT+GCN spectral pipeline. It evaluates delivery strategies that trade off screening performance, uncertainty routing, and API cost.

  • Motivation: Systematic-review screening can require thousands to tens of thousands of records and hundreds of expert hours.The bottleneck is deciding from titles and abstracts which papers merit full-text review.
  • Motivation: LLMs achieve competitive recall but lack calibrated confidence, making uncertain and certain decisions indistinguishable for downstream action.This absence of structured uncertainty limits reviewers’ ability to identify cases requiring attention.
  • Approach: The BERT+GCN pipeline pre-classifies papers as INCLUDE, EXCLUDE, or MAYBE using algebraic-radical and categorical-paradox tests.MAYBE denotes classifier uncertainty from a low include–exclude logit gap or disagreement between textual and relational embeddings.
  • Contributions: MAYBE-only routing achieves mean recall 0.92 and AUC-ROC 0.54 at 1.05× baseline API cost, capturing the spectral benefit at one sixth of full-context overhead.It is identified as the Pareto-optimal configuration for cost-sensitive deployments.
  • Contributions: Full spectral context improves F1 (+0.011, paired Wilcoxon p = 0.008) and WSS@95 (+0.050, p = 0.039) at a 1.28× cost premium.The contribution reports statistically significant gains when all spectral scores are delivered to the LLM.
  • Contributions: Two-pass self-triage escalates 22% of papers yet flips 0.0% of decisions across all datasets, seeds, and folds.This clean null result rules out a class of architectures for LLM self-triage.

2 Related Work

Prior work finds that LLM screening can achieve competitive recall but often has low specificity and lacks principled uncertainty handling. This paper instead supplies uncertainty from a separate BERT+GCN model, framing spectral context as a structured second opinion for LLM screening.

  • LLM screening: LLM screening studies report competitive recall, but frequently low specificity and limited principled uncertainty handling.Prompt engineering substantially affects performance in systematic assessments of LLM screening.
  • Uncertainty calibration: Existing confidence-calibration methods use verbalized uncertainty, token probabilities, or ensembles, usually requiring white-box model access.This work complements them by providing uncertainty estimates from a separate, purpose-built classification model.
  • Tool augmentation: Tool-augmented LLMs use external tools, retrieval, or chain-of-thought prompting to address model limitations and support structured reasoning.The paper treats spectral context as a specialised tool providing a domain-specific model’s structured second opinion.
  • Spectral graph analysis: Spectral gaps for ambiguity detection build on graph-based text classification, graph neural networks, BERT+GCN architectures, and algebraic graph theory.The contribution is an empirical evaluation of deploying such a classifier as an auxiliary signal for an LLM screener, rather than a new spectral method.

3 Background: Spectral Paradox Detection

The BERT+GCN spectral pipeline combines textual and graph-based representations to generate structured uncertainty signals for LLM screening. It flags ambiguity when classifier confidence or textual-relational agreement is insufficient, assigning each paper to INCLUDE, EXCLUDE, or MAYBE.

  • Representations and training: Each paper combines a PubMedBERT title-abstract embedding with a GCN embedding learned from a similarity graph and optional citation edges.Relational graph convolution learns separate parameters for similarity and citation edge types, while MAYBE and unlabelled papers participate in message passing but not loss computation.
  • Paradox tests: τ = |p_include − p_exclude| flags algebraic ambiguity when τ < ϵ, with default ϵ = 0.15.The test identifies papers whose include and exclude probabilities are too close for confident classification.
  • Paradox tests: cos(b, g) < δ flags a categorical paradox when textual and relational embeddings disagree, with default δ = 0.0.Here, b is the BERT embedding and g is the GCN embedding; low cosine similarity indicates contradictory signals.
  • Output states: The engine assigns INCLUDE, EXCLUDE, or MAYBE, with MAYBE indicating that one or both paradox tests triggered and additional scrutiny is required.INCLUDE and EXCLUDE represent high-confidence relevance or irrelevance when no paradox is detected.

4 Method

The method trains a per-split BERT+GCN spectral classifier and injects its uncertainty context into LLM screening prompts through five condition-dependent delivery strategies. Conditions vary spectral information, paper selectivity, and two-pass escalation, with performance assessed using screening and escalation metrics.

  • Pipeline: A BERT+GCN pipeline encodes papers with frozen PubMedBERT embeddings, applies a 2-layer RGCNConv to a kNN graph, and classifies papers as INCLUDE, EXCLUDE, or MAYBE.The spectral engine uses algebraic radical and categorical paradox tests.
  • Experimental conditions: Five conditions vary spectral information content, affected papers, and delivery strategy, including no context, label-only context, full scores, MAYBE-only routing, and two-pass escalation.The conditions are evaluated as alternative ways to provide spectral context to the LLM screener.
  • Prompt design: The baseline prompt contains only the title and abstract, whereas the full spectral prompt adds the model decision, confidence gap, overall confidence, and detected paradox type.A disclaimer allows the LLM to agree or disagree with the spectral assessment.
  • Selective delivery: MAYBE-only routing applies full spectral context only to papers classified as MAYBE, while two-pass self-triage escalates baseline responses showing low or medium confidence or hedging language.Escalated papers receive the previous confidence level and full spectral context in a second prompt.
  • Evaluation: Five primary metrics evaluate screening quality, including recall, AUC-ROC, F1, and WSS@95, alongside override rate, second-pass rate, and second-pass flip rate.Recall and F1 use the binary INCLUDE/EXCLUDE subset, while AUC-ROC uses mapped LLM confidence scores.

5 Experimental Setup

The experiment evaluates LLM-assisted screening across eight Cohen (2006) drug-class benchmarks spanning varied dataset sizes and inclusion rates. A controlled 5-fold, 3-seed design produces 600 runs using a BERT+GCN spectral classifier to provide consistent context across five LLM conditions.

  • Datasets: 8 Cohen (2006) benchmark datasets range from 310 to 2,544 records, with inclusion rates spanning 1.6% to 27.0%.This variation tests robustness across dataset scale and class imbalance.
  • Experimental design: 600 experimental runs combine 8 datasets, 5-fold stratified cross-validation, 3 random seeds, and 5 LLM screening conditions.The seeds are 42, 123, and 456; PubMedBERT embeddings are pre-computed and cached, leaving GCN training to vary across folds and seeds.
  • Experimental design: Within each fold, the spectral engine labels test papers INCLUDE, EXCLUDE, or MAYBE before applying all five LLM conditions to the same test set.Using the same test set enables controlled comparison across prompt-delivery conditions.
  • Model implementation: The spectral model uses frozen 768-dimensional PubMedBERT embeddings and a 2-layer RGCNConv with 256 hidden dimensions, dropout 0.3, and k = 10 neighbours.Training runs for up to 100 epochs with early stopping at patience 10; the similarity graph also uses cosine threshold 0.5.
  • Model implementation: Default spectral thresholds remain fixed across datasets at ϵ = 0.15 and δ = 0.0, with per-dataset tuning deferred to future work.The thresholds govern the algebraic radical and categorical paradox tests, respectively.

6 Results

Across 600 main-matrix runs, full spectral context significantly improved F1 and WSS@95, while MAYBE-only delivery provided the strongest cost–quality trade-off. Two-pass escalation was cost-neutral but never changed decisions, and ablation reduced the dual test to a logit-gap criterion.

  • Aggregate performance: ∆=+0.011 F1 and ∆=+0.050 WSS@95 were significant for full-context delivery, with paired Wilcoxon p=0.008 and p=0.039, respectively.The main matrix comprised 600 runs across 8 datasets, 5 modes, 3 seeds, and 5 folds.
  • Cost–quality trade-off: 0.92 mean recall and 0.54 mean AUC-ROC made MAYBE-only the cost–quality frontier leader at 1.05× baseline cost.This was one sixth of full-context delivery’s 1.28× API-cost premium; no metric differed significantly from baseline or full-context delivery.
  • Numerical context: +0.018 recall, +0.004 F1, +0.063 WSS@95, and +0.033 AUC-ROC favored full spectral over decision-only across 7 datasets.Override rates were nearly identical: 55.0% with full spectral versus 53.8% with decision-only.
  • Two-pass screening: 22.2%±8.8% of papers triggered a second pass, but the flip rate was exactly 0.0% across all datasets and folds.Two-pass recall was 0.90 ± 0.07 versus 0.91 ± 0.06 for baseline, with a cost ratio of 1.00× baseline.
  • Ablation: 0.0 cosine-paradox firings at δ=0.0 reduced the dual paradox test empirically to the algebraic radical logit-gap criterion.The BERT–GCN cosine similarity was positive for every labelled paper, with minimum 0.05, median 0.91, and maximum 1.00.

7 Discussion

The discussion finds that selective MAYBE-only spectral augmentation captures the main screening benefit at near-baseline cost, while two-pass self-triage fails to revise decisions. Results support spectral context as an orthogonal auxiliary signal, but conclusions remain limited by model, domain, and evaluation scope.

  • Spectral-context benefit: +0.05 recall on ADHD and +0.04 on OralHypoglycemics are the largest full-spectral improvements, while NSAIDs gains AUC-ROC (+0.03) despite baseline recall of 1.00.On ACEInhibitors, baseline recall is 0.98 versus 0.95 for full spectral, motivating adaptive thresholds for severe class imbalance.
  • Spectral-context benefit: +0.018 recall and +0.063 WSS@95 are the paired gains of full spectral over decision-only across eight datasets.The findings indicate that numerical uncertainty estimates provide additional signal beyond categorical labels, with WSS@95 improving screening efficiency.
  • Deployment recommendation: 0.92 mean recall and 0.54 mean AUC-ROC make MAYBE-only the best-performing condition at 1.05× baseline cost versus 1.28× for full spectral.The benefit concentrates on papers flagged as ambiguous, making MAYBE-only the recommended default for cost-sensitive screening.
  • Self-triage limitation: 22% average escalation with a zero flip rate shows that two-pass self-triage identifies hedged responses but does not produce decision revisions.The LLM may anchor on its initial decision, so the discussion recommends human-in-the-loop escalation instead.
  • Cross-model interpretation: +0.8% average recall uplift from full spectral context is identical for gpt-5.4-mini and gpt-4.1-mini across the three pilot datasets.This pattern is most consistent with spectral context providing an orthogonal auxiliary signal rather than compensating for a reasoning gap.
  • Scope conditions: The conclusions are scoped to gpt-5.4-mini across eight datasets, a three-dataset gpt-4.1-mini pilot, binary LLM outputs, untested domains, and no human-in-the-loop evaluation.Temperature-0 determinism also prevents analysis of inter-run variance, and larger models may exploit continuous spectral scores differently.

8 Conclusion

Across eight Cohen (2006) benchmark datasets and 600 fold-level runs, the study identifies operationally effective delivery strategies for auxiliary BERT+GCN uncertainty signals in LLM screening. Full-context delivery improves efficiency, MAYBE-only routing minimizes cost, and two-pass self-triage fails to revise decisions.

  • Scope: The evaluation covers eight Cohen (2006) benchmark datasets, five prompt-delivery conditions, 600 fold-level runs, and a three-dataset cross-model pilot.These findings have direct operational implications for systematic-review teams.
  • Conclusion: Full-context delivery improves WSS@95 by +0.050 (paired Wilcoxon p = 0.039) and F1 by +0.011 (p = 0.008) over baseline at 1.28× token cost.Recall is preserved across all five conditions (p ≥0.12).
  • Conclusion: MAYBE-only routing achieves the highest mean recall (0.92) and AUC-ROC (0.54) at 1.05× baseline cost.The configuration uses auxiliary context only for papers flagged as uncertain, with one sixth of blanket full-context API overhead.
  • Conclusion: 0% flip rate across all datasets, seeds, and folds shows that second-pass auxiliary context never revises LLM decisions.The LLM signals uncertainty through hedging language on roughly 22% of records, but two-pass self-triage is not viable for current instruction-tuned models.
  • Future work: Future work includes adaptive spectral thresholds, direct MAYBE outputs, broader cross-model validation, larger models, human-in-the-loop studies, and non-biomedical domains.The proposed extensions target uncertainty coverage, cross-generation invariance, numerical reasoning, real-workflow time savings, and generalization.
  • Availability: All code, prompt templates, experiment configurations, cached LLM responses, and a deterministic table-generating analyser are publicly available.The implementation is released at the project’s GitHub repository.

Code and Data Availability

The study releases its implementation and raw fold-level results in a permissively licensed GitHub repository. Benchmark datasets are redistributed through ASReview Synergy and fetched at runtime via the OpenAlex API.

  • Code and data: The GitHub repository provides source code, prompt templates, experiment configurations, and raw fold-level CSV results under a permissive open-source licence.Repository: https://github.com/rahgoar/LR-Spectral-BERTGCN-Topological-Undecidability-in-Clinical-AI
  • Code and data: The Cohen (2006) benchmark datasets are redistributed by the ASReview Synergy project and fetched at runtime through the OpenAlex API.The passage also mentions preprocessed benchmark data, but the provided text is truncated.

A Prompt Templates

The prompt templates define a fixed screening role, dataset-specific criteria, and strict decision formatting, then vary how BERT+GCN spectral analysis is delivered—from full context to decision-only and two-pass follow-up prompts.

  • A.1 System Prompt: The system prompt casts the LLM as a clinical-literature screening assistant deciding whether each paper is INCLUDED or EXCLUDED.It also supplies per-dataset screening criteria, such as the ACEInhibitors review context.
  • A.1 System Prompt: The system prompt requires exactly formatted outputs containing a decision, HIGH/MEDIUM/LOW confidence, and a 1–2 sentence reason.
  • A.2 Baseline User Prompt (Condition 1): The baseline user prompt presents only the paper abstract for screening.
  • A.3 Full Spectral Context Block (Condition 2): The full spectral-context prompt adds the abstract, BERT+GCN model decision, confidence gap, model confidence, threshold=0.15, and paradox type.It identifies algebraic_radical, categorical_paradox, or none, while allowing the LLM to agree or disagree with the model.
  • A.4 Decision Only Prompt (Condition 3): The decision-only prompt gives the abstract plus only the separate BERT+GCN suggestion of INCLUDE, EXCLUDE, or MAYBE.The LLM is told that it may agree or disagree based on its reading.
  • A.5 Two-Pass Follow-Up Prompt (Condition 5, Pass 2): The two-pass follow-up states the prior confidence, repeats the abstract and spectral analysis, and asks the LLM to reconsider its decision.The follow-up preserves the decision, confidence, and reason output format while providing model decision, confidence gap, model confidence, threshold=0.15, and paradox type.

B Hedging Language Patterns

The two-pass uncertainty detector identifies hedging language with a regex pattern, and either a hedging match or explicit LOW/MEDIUM confidence triggers escalation to a second pass.

  • Hedging detection: The detector uses a regex to identify hedging terms such as “might,” “unclear,” “borderline,” and “uncertain” in LLM responses.The pattern also includes phrases such as “not sure,” “possibly,” “could be,” and “difficult to determine.”
  • Escalation trigger: Both LOW or MEDIUM explicit confidence and a hedging-pattern match trigger escalation to a second pass.

C Full Per-Dataset Results

Table 10 reports full per-dataset results, averaging the five experimental conditions across three seeds and five folds using the primary metrics.

  • Full per-dataset results: Table 10 covers all five conditions and reports primary metrics for each dataset.The results are presented as full per-dataset outcomes.
  • Full per-dataset results: Each reported value is the mean over 3 seeds × 5 folds.This aggregation defines the table’s fold-level summary.
  • Full per-dataset results: The table provides the complete per-dataset comparison across the experimental conditions.Its scope is dataset-level reporting rather than a single pooled result.
Loading 2608.14551v1…