PaperlayerView exact revision on arXiv

Source-linked AI summary

Your Retriever Already Knows: Distribution-Shape QPP for RAG Retrieval Sufficiency

Matyáš Veselý, Michal Průšek, Jiří Franc

arXiv:2609.11646v1cs.IR

TL;DR

RAG systems need an inference-time signal of whether retrieved context is sufficient, especially in sensitive deployments where third-party LLM APIs are unavailable. This paper compares score-based, content-based, and hybrid QPP for retrieval sufficiency. GeneralQPP offers a strong latency–quality trade-off, while adding an LLM judgment as a feature helps in some settings.

  • Problem

    The paper examines how to predict retrieval sufficiency before generation, including whether score distributions suffice or content-aware judging adds essential information in a sensitive deployment.

    Source: §1 · pp. 1–2

  • Method

    The authors compare score-only features, a multimodal LLM judge that reads retrieved pages, and hybrids that add the judge’s score to trained predictors.

    Source: §1 · p. 2, §5 · pp. 10–11

  • Results

    GeneralQPP reaches 0.856 weighted-average AUROC on ViDoRe, beating the local judge by 0.207; H2 leads most evaluated settings, with 0.863 on ViDoRe and 0.911/0.954 on SÚJB.

    Source: §5 · pp. 10–11, §6 · p. 11

  • Takeaways & Limitations

    Score-distribution features provide the stronger latency–quality trade-off in the tested setting, while the local judge is more useful as an auxiliary feature than alone.

    Source: §5 · p. 11

    SÚJB uses synthetic queries, and retriever-agnosticism remains untested with one retriever and one local judge.

    Source: §3.1 · p. 6, §5 · p. 11

Abstract

from arXiv · show

Standard Retrieval-Augmented Generation (RAG) pipelines often provide no reliable inference-time signal of whether retrieval succeeded; on ambiguous or out-of-scope queries, generation may then hallucinate. Motivated by a Czech nuclear-regulator deployment where data sensitivity precludes third-party LLM APIs, we compare three Query Performance Prediction (QPP) paradigms for retrieval sufficiency in RAG: score-based features, a content-based LLM judge, and a hybrid. On the eight ViDoRe vision domains (14,514 queries), our 24 non-lexical features (GeneralQPP; 15 distribution-shape, 5 query-surface, 4 global) reach a weighted-average AUROC of 0.856 at 2 ms per query, ahead of a classic-QPP literature pool (Classic Full, 0.835) and well above a local Qwen3.5 LLM judge (0.649, +0.207 gap; $\sim$3000$\times$ faster and cheaper per query). Adding the LLM judgment as one feature (hybrid) matches S1 on ViDoRe (0.863) but gains a statistically significant edge on S\'{U}JB (AUROC 0.911 at Hit@5, adversarial-detection 0.954; 1,510 queries, 500 synthetic adversarial), at LLM latency. Rankings agree across datasets (Spearman $\rho = 0.90$). Under Leave-One-Domain-Out, S1 drops to 0.706; a 13-feature LODO-stepwise subset (S1-Lean) recovers to 0.719 (+0.032 over the literature pool).

1 Introduction

This work asks whether retrieval-score distributions can signal sufficiency without reading document content, or whether content-aware judging adds essential information. It compares score-based, content-based, and hybrid QPP for RAG, motivated by a sensitive deployment that requires fast, cheap, local confidence estimation.

1 Introduction

  • A motivating Czech nuclear-regulator deployment forbids third-party LLM APIs for data-sensitivity reasons and requires confidence estimation to be fast, cheap, and local.The research question is whether retrieval-score distribution shape is sufficient or content-aware judging adds essential information.Source: §1 · pp. 1–2
  • Prior RAG QPP studies use retrieval-side features but do not compare them against an LLM that reads retrieved documents.The cited lines include supervised prediction of retrieval-related perplexity gain and a framework combining score-only features with BERT-QPP.Source: §1 · p. 2
  • Vision-document retrieval-sufficiency QPP is relatively unexplored, despite ViDoRe V3 benchmarking vision RAG across eight enterprise domains.The broader motivation is to predict whether top-k context likely contains the answer before generation.Source: §1 · pp. 1–2
  • The paper compares score-based, content-based, and hybrid predictors of retrieval sufficiency for vision-document RAG, where this QPP setting remains relatively unexplored.Its contributions include GeneralQPP, a 24-feature predictor that reads no document content, and a uniform comparison across ViDoRe and SÚJB.Source: §1 · p. 2
  • Score-distribution features offer the best latency–quality trade-off in the authors’ setting.Their comparison frames a local multimodal LLM judge as more useful as an auxiliary feature than as a standalone predictor.Source: §1 · pp. 2–3

2 Methods

The methods compare score-only, content-reading, and hybrid predictors under a shared model-selection protocol. GeneralQPP represents retrieval-score shape and related non-lexical signals, while hybrids add an LLM judgment as a feature.

2 Methods

  • The study evaluates score-based S0–S4, content-based C1, and hybrid H1–H3 methods; trained methods share the architecture-selection protocol.C1 is not trained and has no architecture to select.Source: §2 · p. 3, §2.1 · p. 3

2.1 Architecture Selection

  • Trained methods select architectures by three-fold stratified CV on training data, standardize features, then refit on the full training fold.The candidate pool includes Logistic Regression, Ridge, and MLPs; preprocessing and selection stay within each training fold.Source: §2.1 · p. 3

2.2 Score-Based Methods

  • GeneralQPP uses 24 non-lexical features across distribution shape, query-surface statistics, and global similarity; S0 uses only maximum similarity.S1-Lean retains 8 distribution and 5 query features, with no global features; S2, S3, and S4 represent prior feature pools.Source: §2.2 · pp. 3–4, Table 1 · p. 4

2.3 Content-Based Method

  • C1 reads retrieved pages and emits a sufficiency score; the selected three-shot English prompt achieved zero parse failures and best overall AUROC among four tested variants.Unscorable outputs default to 0.5, and the same pipeline is used on ViDoRe and SÚJB.Source: §2.3 · pp. 4–5, Table 2 · p. 5

2.4 Hybrid Methods

  • H2 augments GeneralQPP with C1’s score as a 25th feature, while H3 adds that score to Classic Full’s 18 features.H1 adapts Tian et al.’s hybrid by replacing fine-tuned BERT-QPP with a zero-shot multilingual cross-encoder because Czech QPP-labeled training data are unavailable.Source: §2.4 · p. 5

3 Experimental Setup

The evaluation uses a Czech regulatory case study and an eight-domain vision-document benchmark, with retrieval sufficiency defined by whether at least one self-sufficient page appears in the top-k. AUROC is the primary comparison metric, alongside calibration and precision–recall diagnostics.

3.1 Datasets

  • SÚJB comprises 517 pages from five Czech nuclear-regulatory documents and 1,510 synthetic queries; ViDoRe covers eight domains and 14,514 queries.SÚJB uses a fixed 80/20 split, while ViDoRe uses stratified 80/20 splits within each domain and trains per domain.Source: §3.1 · pp. 5–6

3.2 Ground Truth

  • Retrieval success is Hit@k: at least one self-sufficient page must appear in the top five or ten; negative classes differ between ViDoRe and SÚJB.ViDoRe negatives have no fully relevant page in the corpus, whereas SÚJB adversarial queries concern topics absent from the corpus entirely.Source: §3.2 · p. 6

3.3 Evaluation Metrics

  • AUROC is the primary metric, while AUPRC, Brier score, and ECE provide complementary diagnostics; results average five random seeds.Table 4 reports per-domain test AUROC and the weighted average across ViDoRe’s eight domains.Source: §3.3 · pp. 6–7, Table 4 · p. 7

4 Results

On ViDoRe, S1 is the strongest standalone score-based predictor, while H2 has a small, nonsignificant AUROC advantage; on SÚJB, score-based and hybrid methods show useful distinctions for adversarial queries. Rankings broadly agree across datasets, and C1’s overconfidence on retrieval failures remains a practical concern.

4.1 Main Results: ViDoRe Benchmark

  • 0.856 weighted-average AUROC: S1 leads score-only methods; C1 reaches 0.649, 0.207 below S1, while H2’s 0.863 uplift is not significant on ViDoRe.S1 beats S4, S2, and S3; H2’s uplift over S1 is +0.007.Source: §4.1 · pp. 7–8

4.2 Diagnostic: Failure-Case Calibration

  • 0.394 positive-minus-negative confidence separation for S1 contrasts with C1’s 0.047, as C1 assigns failed-retrieval queries mean confidence 0.560 after calibration.H2 has the lowest mean confidence on failed-retrieval queries, 0.355; Fig. 2 shows S1 separating answerable from unanswerable queries more cleanly than C1.Source: §4.2 · p. 8, Figure 2 · p. 8

4.3 Deployment Case Study: SÚJB

  • 0.934 adversarial-detection AUROC for S1 exceeds C1’s 0.864, while H2 would answer only 6–7% of adversarial queries at threshold 0.5.S1 leads the score-only tier on SÚJB with AUROC 0.893; the test set has 302 queries.Source: §4.3 · p. 9, Table 6 · p. 10

4.4 Robustness Checks

  • 0.90 Spearman ρ (Kendall τ = 0.78, p < 0.005) indicates broad ranking agreement between ViDoRe and SÚJB; calibration reduces C1’s ECE without changing AUROC.C1’s ECE falls 6× on ViDoRe and halves on SÚJB Hit@5; reported train–test AUROC gaps remain below 0.023.Source: §4.4 · pp. 9–10

5 Discussion

Score-based S1 offers the stronger ViDoRe latency–quality trade-off, while the hybrid can improve results in some SÚJB settings when an LLM judge is already in the inference path. Under leave-one-domain-out evaluation, S1-Lean improves transfer over S1, with conclusions bounded by the tested data and system setup.

Score Features vs. LLM Judges

  • +0.207 weighted-average AUROC separates S1 from C1 on ViDoRe, where S1 wins all eight domains; S1 also leads by +0.061 on SÚJB Hit@5.S1 reads no document content at 2 ms per query, versus C1’s ∼6.3 s LLM pass; H2’s ViDoRe gain over S1 is small and statistically non-significant at LLM-level latency.Source: §5 · pp. 10–11

LLM as Feature, Not Predictor

  • On SÚJB, H2 improves AUROC over S1 and C1 at both retrieval depths, but its uplift over S1 is setting-dependent and pays off primarily when the judge is already in the inference path.The uplift is +0.018 on SÚJB Hit@5 versus +0.007 on ViDoRe (p = 0.261); on ViDoRe, H2 beats the standalone judge by +0.214 (p = 0.001).Source: §5 · p. 11

OOD Robustness: S1-Lean

  • 0.719 LODO AUROC for 13-feature S1-Lean exceeds full S1’s 0.706 and the classical baseline S4 by +0.032, while trading away 0.074 in-domain ViDoRe AUROC.S1-Lean also has the best SÚJB Hit@10 calibration (ECE 0.043 versus S4’s 0.048 and H2’s 0.076).Source: §5 · p. 11

Limitations

  • The evidence is bounded by one retriever and one local judge, and retriever-agnosticism remains untested; live, non-synthetic query validation is future work.C1 is constrained by JSON parsing, and H1 uses a conservative zero-shot cross-encoder adaptation instead of fine-tuned BERT-QPP.Source: §5 · p. 11

6 Conclusion

The paper compares score-based, content-based, and hybrid QPP for retrieval sufficiency across ViDoRe and SÚJB, finding that a hybrid can perform well while score-based GeneralQPP offers a strong low-latency result. The conclusions are strongest for local multimodal retrieval-sufficiency prediction with modest target-domain supervision, under one tested retriever/judge pairing.

6 Conclusion

  • 0.856 weighted-average AUROC on ViDoRe for GeneralQPP is +0.207 over a local Qwen3.5 judge, with 2 ms per query; SÚJB adversarial-detection AUROC is 0.934.The evaluation covers nine methods and two datasets: ViDoRe (14,514 queries) and SÚJB (1,510 queries).Source: §6 · p. 11
  • H2 wins the most evaluated settings, scoring 0.863 on ViDoRe (n.s. vs. S1) and 0.911/0.954 on SÚJB; H3 is essentially tied and slightly higher on SÚJB Hit@10 AUROC.Source: §6 · p. 11
  • Spearman ρ = 0.90 indicates that method rankings agree across the two datasets; the paper presents this as one of the first QPP-style retrieval-sufficiency applications to vision-document retrieval.The conclusions are strongest for local multimodal retrieval-sufficiency prediction with modest target-domain supervision.Source: §6 · pp. 11–12
  • The ranking is limited to one tested retriever/judge pairing; stronger judges and broader answer-quality predictors may alter it.Future work includes live-query validation, cross-retriever generalization, cross-lingual evaluation, and selective RAG with abstention.Source: §6 · p. 12
Loading 2609.11646v1…