PaperlayerView exact revision on arXiv

Source-linked AI summary

Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection

Max Nelson, Hanoz Bhathena, Aviral Joshi, Saket Sharma

arXiv:2609.02745v1cs.IRcs.CL

TL;DR

Production RAG teams need reliable, repeatable retrieval-model comparisons, but relevance judgments are expensive and difficult to extend to new candidates. The paper incrementally pools LLM judgments over retrieved documents and reuses them across systems. Across benchmarks and a production financial QA deployment, rankings agree strongly with gold evaluation while reducing judgment cost through document overlap.

  • Problem

    RAG retrieval-model selection requires comparative relevance judgments, but gold-standard human judgments are expensive and difficult to repeat as new systems arrive.

  • Method

    The method judges the union of documents retrieved by candidate systems, reuses shared judgments, and judges only newly contributed documents when systems are added.

  • Results

    Pooled LLM rankings correlate strongly with gold-standard evaluation across four benchmarks, with nDCG@10 ρ = 0.69–0.95 and 74% of pairwise disagreements within qrels bootstrap noise.

  • Takeaways & Limitations

    The workflow supports ongoing retrieval model selection by reusing judgments and comparing new candidates without re-judging previously assessed documents.

  • Takeaways & Limitations

    The evaluation relies primarily on GPT-4.1, with cross-judge validation covering only GPT-4.1 and Claude Sonnet 4.6 on one dataset.

Abstract

from arXiv · show

Selecting a retrieval model for a production RAG system requires reliable comparative evaluation, but obtaining relevance judgments at scale is expensive and difficult to repeat as new candidate systems arrive. We study pooled LLM evaluation, in which an LLM judges the union of documents retrieved by the current set of candidate systems, and the pool is then expanded incrementally as new systems are introduced by judging only the new documents they contribute. These judgments are reused to evaluate all systems on a common basis. We validate this approach on four retrieval benchmarks with 11 systems spanning dense, sparse, and hybrid configurations, and deploy it to compare 62 retrieval configurations for a financial news QA system. Pooled LLM rankings correlate strongly with gold-standard evaluation across datasets, and 97% of pairwise system orderings are preserved once bootstrap uncertainty in the qrels is taken into account. In production, document overlap yields 65-80% judgment reuse and up to 4.9x lower evaluation cost, allowing teams to benchmark new retrieval candidates without re-judging previously assessed documents. These results suggest pooled LLM evaluation is a practical and cost-effective workflow for incremental retrieval model selection in deployed systems.

1 Introduction

Retrieval evaluation must support repeated comparison of new systems without the expense and redundancy of independent human or LLM judgments. The paper proposes incrementally pooled LLM evaluation and reports strong ranking agreement with gold judgments.

  • RAG practitioners need to select retrieval models as new embedding models arrive, but gold-standard human judgments are expensive, slow, and difficult to repeat.
  • O(N · Q · K) judgments are required for independent LLM evaluation, with substantial redundancy from overlapping retrieval results.
  • Pooled LLM evaluation judges the union of documents retrieved by all systems and reuses those judgments to compute standard IR metrics.
  • Cost amortization: Shared documents are judged once, so each added system incurs only marginal cost for uniquely retrieved documents.
  • Ranking stability: System-level P@k and DCG@k remain invariant under pool expansion, while empirical macro-averaged rankings remain highly stable for additional systems.
  • Reduced pool bias: Teams can start with two or three systems and add candidates without systematic penalties for novel retrieval behavior caused by unjudged documents.
  • Accuracy: nDCG@10 rank correlation with gold human judgments is 0.69–0.95 across four public IR benchmarks, while 74% of pairwise disagreements fall within qrels bootstrap noise.

2 Related Work

Prior work establishes that LLM judgments can support system-level retrieval evaluation, while classical pooling can disadvantage novel systems through unjudged documents. This work focuses on reducing cost and characterizing ranking stability for incremental industry evaluation.

  • Prior studies report LLM relevance assessments and support labels that agree with human annotators or yield correlated system-level rankings.
  • The paper targets efficiency, bias, and ranking stability as practical concerns in LLM-based retrieval evaluation.
  • Test Collection Reusability: Classical TREC pooling can treat documents absent from the original pool as non-relevant, disadvantaging systems with novel retrieval behavior.
  • Test Collection Reusability: The proposed approach judges all documents retrieved by any system, providing a structural safeguard against unassessed relevant documents.
  • Cost-Effective Evaluation: Unlike active selection or sampling methods, pooled evaluation judges all pooled documents and reduces cost by reusing judgments across systems with overlapping results.

3 Method

The method retrieves top-k documents for each system, pools unique query-document pairs, judges them once on a graded scale, and evaluates every system using shared judgments. New systems are added incrementally by judging only documents absent from the existing pool.

  • Retrieve: For each system and query, the procedure retrieves the top-k ranked documents.
  • Judge: Each unique pooled query-document pair receives an LLM relevance judgment on a 3-point scale.
  • Evaluate: Standard IR metrics are computed for each system using shared pool judgments restricted to that system’s retrieved documents.
  • Incremental evaluation: A new system contributes only documents outside the existing pool, while all previous judgments are reused exactly for equal-footing comparison.
  • Stability properties: P@k and DCG@k are exactly invariant under pool expansion when labels remain frozen and ranked lists remain fixed.
  • Cost Analysis: Independent evaluation requires up to N·|Q|·K judgments, whereas pooling judges only unique query-document pairs and measures reuse through 1 − Cpool / (N·|Q|·K).
  • Cost Analysis: The marginal cost of adding a system is limited to its uniquely retrieved documents outside the existing pool.

4 Experimental Setup

Experiments cover four retrieval benchmarks and dense, sparse, and hybrid systems, using structured document processing and pooled LLM judgments. Judge consistency and cross-judge analyses assess whether these judgments support reliable system comparisons.

  • Datasets: The evaluation uses FiQA, TREC-COVID, Natural Questions, and FinRAGBench-V, covering BEIR and financial-domain retrieval settings.
  • Datasets: Documents are parsed into text, tables, and figures, with multimodal descriptions interleaved in natural reading order while preserving modality provenance.
  • Retrieval systems: Systems span five embedding families across dense, sparse, and hybrid configurations, yielding five dense, five hybrid, and one BM25 system per dataset.
  • LLM judge: GPT-4.1 judges each query-document pair independently at temperature 0.0, using a single-turn prompt without in-context examples or system-level calibration.
  • LLM judge: The judge assigns graded relevance scores of 0, 1, or 2 to distinguish non-relevant, partially relevant, and fully relevant documents.
  • Judge consistency: Exact 3-point re-judging agreement reaches 93–96% across BEIR datasets, with κ = 0.84–0.90 and at least 99.9% agreement within ±1 grade.
  • Cross-judge agreement: Cross-judge system rankings remain highly correlated despite item-level differences, with nDCG@10 ρ = 0.89, recall@100 ρ = 0.92, and MAP ρ = 0.88.
  • Evaluation protocol: All systems retrieve top-100 documents per query, and metrics are computed against pooled LLM judgments and published human qrels.

5 Results

Pooled LLM rankings correlate significantly with gold-standard qrels, and most disagreements fall within qrels’ own bootstrap uncertainty. Pool growth is empirically stable, with reversals confined to statistically indistinguishable near-ties.

  • 5.2 Pairwise Rank Agreement: Figure 1 shows most pseudolabel ranks inside the qrels’ 95% bootstrap rank confidence intervals.Blue circles represent qrels ranks, while orange diamonds represent pseudolabel ranks.
  • 5.2 Pairwise Rank Agreement: 74% of nDCG@10 disagreements fall within qrels’ bootstrap noise, leaving only 7 of 27 as genuine ranking differences.Bootstrap uncertainty is estimated from 10,000 query-resampling iterations.
  • 5.2 Pairwise Rank Agreement: 66% of disagreements across all four metrics on the three BEIR benchmarks fall within qrels noise.FinRAGBench-V shows the same pattern, with 4 of 6 nDCG@10 disagreements within qrels uncertainty.
  • 5.3 Cost Amortization: Pooling reduces marginal judgment cost as the pool grows because successive systems contribute fewer novel documents.Table 4 compares independent evaluation with pooled unique (q, d) judgments and reports approximately 1.26M redundant judgments eliminated.
  • 5.1 System-Level Rank Correlation: nDCG@10 Spearman correlations range from 0.691 to 0.945 across FiQA, TREC-COVID, NQ, and FinRAGBench-V.Correlations are strongest on FiQA and NQ, while TREC-COVID is lower but still significant.
  • 5.4 Ranking Stability Under Permuted Order: No pair with a meaningful score gap (∆> 0.001) reverses under any tested system-addition ordering.NQ and TREC-COVID show zero reversals across 500 orderings; observed reversals elsewhere involve near-tied systems.

6 Analysis

Pseudolabels differ from qrels in absolute metric values, but their approximately uniform bias preserves ordinal rankings. Ranking disagreements concentrate among near-tied systems and are reduced relative to classical pool bias when newly contributed documents are judged.

  • Metric Bias: nDCG is inflated by +0.1% to +51% and recall is deflated by −17% to −39%, with approximately uniform bias across systems.Within-column standard deviations are ≤12.8 percentage points for nDCG and ≤11.1 percentage points for recall.
  • Metric Bias: Uniform calibration offsets preserve ordinal system rankings despite differences in absolute pseudolabel and qrels scores.The passage states that the calibration offset does not vary meaningfully between retrieval configurations.
  • Disagreement Patterns: Pairwise disagreements are concentrated among systems with very small score differences, especially in TREC-COVID’s specialist biomedical setting.On FinRAGBench-V, remaining meaningful differences are concentrated in the emb3-large family.
  • Pool Bias: Classical pooling produced 19 pairwise ranking errors across three affected datasets, whereas judging newly contributed documents removed all of them.Classical pooling treats documents absent from the original pool as unjudged and effectively non-relevant.

7 Production Application

The production deployment used pooled evaluation to compare 62 retrieval configurations for a financial news QA system. It achieved substantial judgment reuse and selected a hybrid emb3-large configuration while exposing retrieval and query-design trade-offs.

  • Deployment Setup: 62 retrieval configurations were benchmarked across five embedding models, dense, hybrid, and sparse modes, multiple dimensions, query formulations, and query-expansion variants.The benchmark covered original, keyword-reduced, and instruction-framed queries.
  • Cost Savings: 79.6% judgment reuse reduced evaluation cost 4.9×, requiring 766,350 pooled judgments instead of 3,765,643 independent judgments.The pool was built incrementally over four weeks without rerunning previously computed judgments.
  • Production Rankings: Hybrid-emb3-large-256 was selected for production after hybrid configurations occupied all five top MAP@100 positions.The selected system ranked second with MAP 0.459, behind hybrid-emb3-large-768 at 0.461.
  • Additional Findings: Mean ∆MAP from increasing dense-retrieval dimensionality is +0.029, narrowing to +0.012 under hybrid retrieval.The results suggest BM25 fusion compensates for reduced embedding capacity under hybrid retrieval.
  • Additional Findings: Instruction-framed queries degrade all systems by ∆MAP = −0.039, while query expansion improves robustness for verbose formulations.Query expansion changes the framing-induced recall@100 penalty from −4.8pp to −1.1pp but lowers MAP on well-formed queries from 0.459 to 0.424.
  • Incremental Pool Growth: Early production conclusions such as “hybrid > dense” remained valid as the pool expanded from 5 to 62 systems over four weeks.The paper contrasts this incremental process with rerunning the entire benchmark for each addition.
  • Cost Savings: The 766,000-plus GPT-4.1 judgments cost approximately $800 at the reported pricing.The stated pricing was $2 per million input tokens and $8 per million output tokens.
  • Beyond Model Selection: Reusable judgments supported additional analyses about dimensionality, keyword queries, and query expansion with zero additional LLM cost.These analyses were not part of the team’s original plans.

8 Discussion

Pooled LLM evaluation is presented as a reusable incremental workflow for production retrieval teams. It is most valuable when relevance judgments are unavailable or expensive, candidates change regularly, and stable historical benchmarking matters.

  • Practical Implications: Production teams can start with 3–5 systems, judge only newly contributed documents for later systems, and compare every system using reused scores.Previously judged documents are reused exactly as the pool expands.
  • When This Method is Most Valuable: The method is particularly suited to domains lacking affordable gold-standard judgments, regularly evaluating new embedding models, or requiring stable historical benchmarks.The paper gives quarterly vendor updates as an example of recurring candidate evaluation.

9 Conclusion

Pooled LLM evaluation is presented as a practical framework for ongoing retrieval model selection, combining strong agreement with human judgments and substantial judgment reuse. A production deployment evaluated 62 configurations at $800 total annotation cost.

  • $800 total annotation cost evaluated 62 retrieval configurations over four weeks in a financial news QA production deployment.The deployment directly informed model selection.
  • nDCG@10 rankings correlated strongly with human judgments across four benchmarks, with ρ = 0.69–0.95.The benchmarks contained 11 systems each.
  • 74% of pairwise disagreements fell within qrels’ sampling uncertainty, leaving only 7 meaningful nDCG@10 differences across 220 system pairs.Bootstrap analysis was used to distinguish meaningful ranking differences from qrels uncertainty.
  • 65–80% document overlap produced judgment reuse, making incremental evaluation economically viable as new embedding models arrive.Previously assessed documents can be reused when new retrieval candidates are introduced.

10 Limitations •

The evaluation has several scope and stability boundaries: it does not establish universal macro-ranking invariance, cover diverse judges or reranking architectures, or reliably separate nearly tied systems. Bootstrap stress tests provide empirical support within these boundaries.

  • Tight clusters: Systems separated by <0.001 nDCG cannot be reliably distinguished, with bootstrap swap probabilities exceeding 40% for both qrels and pseudolabels.The method is intended for comparative model selection rather than resolving effectively tied systems.
  • Limited judge diversity: The primary rankings use a single GPT-4.1 judge, while validation with Sonnet 4.6 covers only one dataset and two frontier-class LLMs.Performance and bias characteristics may differ for smaller, open-weight, or domain-specialized judges.
  • System architecture scope: The evaluated systems are single-stage dense, sparse, or hybrid retrievers; reranking and cross-encoder architectures are not tested.Their document-overlap patterns and score distributions may differ substantially.
  • Benchmark-level stability: Macro-averaged rankings can shift when pool expansion changes the relevance denominator, although stress tests confined observed instability to pairs separated by <0.001.Per-query invariance does not provide a universal benchmark-level guarantee.

B Calibration Bias Details

Pooled LLM judgments show dataset-dependent calibration bias relative to human qrels: they generally inflate nDCG and deflate recall, with an exception for TREC-COVID recall. The bias is approximately uniform across retrieval configurations, preserving ordinal rankings.

  • Pooled LLM judgments assign graded relevance to more documents than human annotators, inflating nDCG while lowering recall-based metric values.Broader relevance labels increase nDCG gain contributions and expand recall denominators.
  • Dataset-specific bias: FiQA nDCG@10 pseudolabels score 29–74% higher than qrels, with mean +51%, while recall@100 scores 33–45% lower.
  • Dataset-specific bias: TREC-COVID nDCG@10 bias is near zero, with mean +0.1%, while recall@100 is 145–211% higher under pseudolabels.Its exhaustive qrels identify approximately 1,327 relevant documents per query, beyond the pool’s 256 documents.
  • Dataset-specific bias: NQ nDCG@10 pseudolabels score 2–46% higher, with mean +17%, while recall@100 scores 31–43% lower.
  • Dataset-specific bias: FinRAGBench-V nDCG@10 pseudolabels score 29–76% higher, with mean +43%, while recall@100 scores 13–23% lower.Sparse qrels of approximately 1.1 relevant documents per query allow the LLM to identify many additional relevant documents.
  • Cross-system calibration: Low within-column standard deviations indicate calibration bias is approximately uniform across retrieval configurations, preserving ordinal system rankings.The bias differs by dataset and metric, but not meaningfully between configurations.
Loading 2609.02745v1…