PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory

Fabian David Schmidt, Donato Crisostomi, Carlos Lassance, Nils Reimers

arXiv:2609.35739v1cs.IR

TL;DR

Human qrels can be sparse, noisy, and too coarse to distinguish rerankers, while LLM preferences lack cross-query comparability when judged relatively or absolutely. RCP combines tournament rankings with rubric criteria and IRT calibration; its gains align more closely with human judgments, and RCP-nDCG agrees with NIST grades on every pair they separate significantly.

  • Problem

    Relative LLM judgments distinguish documents within a query but lack a common cross-query scale, while absolute grades support comparison across queries but can leave documents tied.

    Source: §1 · p. 2

  • Method

    RCP combines a listwise Bradley–Terry tournament and binary rubric criteria, using IRT to place tournament scores from all queries on one scale.

    Source: §1 · p. 2

  • Results

    RCP gains reach AUC 0.910 versus 0.651 for qrels against 46 annotators, and RCP-nDCG agrees with NIST on every reranker pair their grades separate significantly.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    RCP-nDCG provides dense calibrated gains that can score rerankers and aligns with human judgments across the reported annotator and TREC-DL comparisons.

    Source: §1 · pp. 2–3, §6 · p. 9

    RCP gains and annotator grades assess each document alone, ignoring redundancy, and measure perceived relevance rather than factual correctness.

    Source: §6 · p. 9

Abstract

from arXiv · show

Rerankers decide which documents users and LLMs see, yet their standard metric, nDCG, relies on human relevance labels that are costly, sparse, noisy, and discretely graded. As rerankers approach each other in quality, nDCG on these labels therefore increasingly fails to separate them. LLM judges could supply dense labels. Relative judgments within one query tell even close candidates apart, yet their scores share no scale across queries. Absolute grades share one scale but are too coarse to distinguish documents of similar relevance. We propose Rubric-Calibrated Preferences (RCP), which combine both kinds of judgment. A listwise Bradley-Terry tournament orders each query's documents, and a rubric of yes/no criteria of increasing stringency provides an absolute standard. Item Response Theory (IRT), which scores test-takers based on their answers to common questions, then uses the shared criteria to put all queries' tournament scores on one scale. RCP's retrieval metric, RCP-nDCG, replaces nDCG's discrete labels with the resulting calibrated relevance probabilities. Against blind grades from 46 external annotators, calibration raises the correlation between a query's mean score and its mean human grade from 0.538 to 0.795. The probabilities rank a useful document above a non-useful one with probability 0.910 (AUC, chance 0.5), versus 0.651 for the benchmark labels. When the annotators' grades prefer one of two rerankers and exactly one metric agrees, that metric is RCP-nDCG in 72.4% of 185 comparisons (chance about 53%). On TREC-DL, RCP-nDCG sides with NIST assessors' grades on every reranker pair that these grades separate significantly. RCP-nDCG also resolves many of nDCG's ties and separates 1.9 times as many reranker pairs on NanoBEIR. Rubric calibration thus turns relative LLM judgments into dense relevance labels that are comparable across queries and agree with human judgment.

1 INTRODUCTION

Human qrels can leave rerankers indistinguishable, while LLM preferences lack cross-query comparability when relative and absolute judgments are used separately. RCP combines within-query rankings with rubric criteria and IRT calibration, and its retrieval metric is evaluated against human judgments and benchmark grades.

1 INTRODUCTION

  • 45.6% of NanoBEIR and 30.7% of BRIGHT queries show nDCG saturation, flooring, or compression for the 14 evaluated rerankers.The paper attributes these issues to qrels that are costly, sparse, noisy, and discretely graded.Source: §1 · p. 1
  • Relative LLM judgments distinguish candidates within queries but lack a shared scale, while absolute grades support cross-query comparison but leave many documents tied.The paper asks whether fixed yes/no rubric criteria can put within-query LLM preferences on a scale shared across queries.Source: §1 · p. 2
  • RCP combines a listwise Bradley–Terry tournament, rubric-based binary criteria, and IRT to calibrate within-query preferences across queries.RCP-nDCG uses the resulting dense calibrated gains instead of discrete labels.Source: §1 · p. 2
  • 72.4% of 185 contests favor RCP-nDCG when exactly one of it and nDCG agrees with annotators’ preferred reranker, versus about 53% chance.Against annotators’ grades, RCP gains have AUC 0.910 versus 0.651 for qrels.Source: §1 · p. 2
  • RCP-nDCG agrees with NIST assessors on every reranker pair their grades separate significantly, with either of two LLM judges.The paper also reports comparisons against 46 external annotators and benchmark qrels.Source: §1 · p. 3

2 BACKGROUND

The background describes how nDCG uses benchmark relevance labels and why common comparison and calibration models address different scoring needs. Bradley–Terry scores are query-relative, while IRT can share a scale across queries but may not order documents finely.

2 BACKGROUND

  • nDCG computes a ranking score from discounted gains, with benchmark qrels supplying gains and unlabeled documents assigned zero.The ideal ranking sorts documents by decreasing gain, and scores are averaged over queries.Source: §2 · p. 3
  • Bradley–Terry models pairwise preference as σ(θi − θi′), but scores from separate comparison sets are not comparable.Within each set, scores are identifiable only up to an additive constant, and their spread may vary.Source: §2 · p. 3
  • IRT gives documents a shared ability scale through common criterion parameters, but a limited set of yes/no answers can leave many documents tied.In the 2PL model, criterion difficulty sets the half-probability ability and discrimination controls how sharply pass probability rises.Source: §2 · p. 3

3 METHODOLOGY

RCP combines a tournament-based within-query ordering with rubric responses calibrated through a shared 2PL IRT model. Its continuous relevance gains preserve the tournament order while supporting cross-query comparison.

3.1 STAGE A: LISTWISE BRADLEY–TERRY TOURNAMENT

  • Stage A compares windows of 10 documents, converts the judge’s scores into pairwise preferences, and fits Bradley–Terry scores to rank documents.A three-phase window schedule focuses on uncertain ranks and avoids allowing dominant documents to lower mid-ranked documents’ scores.Source: §3.1 · p. 3

3.2 STAGE B: POINTWISE CRITERION EVALUATION

  • Stage B evaluates documents against five binary criteria of increasing stringency, recording criterion pass responses across windows.The criteria target general-purpose retrieval; other applications may need different criteria.Source: §3.2 · p. 3, Table 1 · p. 4

3.3 CALIBRATION: 2PL IRT MERGE

  • The 2PL calibration maps each query’s Bradley–Terry scores affinely to abilities using a query-specific offset and positive scale factor, with criterion parameters shared across queries.Sorting calibrated abilities preserves the Stage A order, and shared calibration makes criterion pass probabilities comparable across queries.Source: §3.3 · p. 4

3.4 RCP-nDCG: A CONTINUOUS RELEVANCE METRIC

  • RCP-nDCG replaces qrel gains with a continuous gain formed from calibrated ability, weighting criterion pass probabilities by discrimination.The gain rises most where criteria measure most precisely; RCP-nDCG’s ideal ranking is the Stage A order. Count-nDCG instead uses criterion pass share, so equal pass shares tie.Source: §3.4 · p. 4

4 EXPERIMENTS

The experiments evaluate RCP’s fit and calibration, its relevance labels, and RCP-nDCG’s reranker comparisons across retrieval benchmarks. Against blind human verdicts and NIST grades, RCP-nDCG agrees more often than qrel-nDCG and separates more reranker pairs.

4 EXPERIMENTS

  • The evaluation covered 13 NanoBEIR datasets, 12 BRIGHT tasks, TREC-DL 2019 and 2020, and eight ViDoRe v3 corpora, using 14 rerankers from six model families.For the human study, 46 external annotators graded 2,268 documents across 311 contests; the experiments examined calibration, label quality, and reranker comparisons.Source: §4 · pp. 4–5

4.1 2PL DIAGNOSTICS

  • The 2PL fit the judge’s criterion answers: conditional criterion-answer correlations averaged 0.123 versus 0.60 for raw answers, and transferred parameters cut held-out cross-entropy by 60.6%.Criterion difficulty increased from C1 to C5, while C1–C4 had near-uniform discrimination; the per-query scale varied about threefold from the 5th to 95th percentiles.Source: §4.1 · pp. 5–6, Figure 3 · p. 5

4.2 CALIBRATION QUALITY

  • Calibration raised the correlation between query-mean scores and human grades from 0.538 for raw BT to 0.795, while pooled document-score correlations rose from 0.641–0.693 to 0.753–0.830.RCP’s gains had ECE 0.054 against 7,080 human grades; on TREC-DL, calibrated-score correlations with NIST grades were 0.649 and 0.595 for 2019 and 2020.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 BLIND EVALUATION OF THE LABELS

  • RCP gains ranked a useful document above a non-useful one with probability 0.910, versus 0.651 for binary qrels, which disagreed with annotators in both directions.Blind LLM judges preferred RCP’s document set over the positive-qrel set on 77.9% of decided NanoBEIR queries and 87.8% of decided BRIGHT queries.Source: §4.3 · p. 7

4.4 RCP-nDCG RANKING RESULTS

  • On NanoBEIR, zerank-2 and zerank-1 rose from ranks 14 and 13 under qrel-nDCG@10 to ranks 1 and 2 under RCP-nDCG@10.The former qrel-nDCG leaders fell to ranks 9 and 6; on TREC-DL, annotators preferred zerank in 78.8% of 66 contests where RCP-nDCG and qrel-nDCG disagreed.Source: §4.4 · p. 7, Figure 5 · p. 7
  • RCP-nDCG separated 63.5% of reranker pairs on NanoBEIR versus 34.1% for qrel-nDCG, and judges preferred its ideal top-10 over Count-nDCG’s in 80.6–98.8% of queries with a preference.Count-nDCG separates about as many pairs as RCP-nDCG, but the tournament resolves within-query differences that the rubric alone ties.Source: §4.4 · pp. 7–8, Figure 6 · p. 8

4.5 BLIND EXTERNAL PREFERENCE VALIDATION

  • 72.4% of 185 contests where exactly one metric agreed with annotators favored RCP-nDCG, and it agreed with NIST on every NIST-decisive pair.With QWEN3.6-27B, RCP-nDCG separated 60 pairs in 2019 and 70 in 2020.Source: §4.5 · pp. 8–9, Figure 7 · p. 8

5 RELATED WORK

Prior IRT work builds scales by treating models as respondents and benchmark questions as items; RCP instead treats documents as respondents and rubric criteria as items. In retrieval, RCP also differs from prior approaches by using shared criteria to set each query’s scale and offset while preserving tournament order.

5 RELATED WORK

  • RCP treats documents as IRT respondents and rubric criteria as items, linking queries on a shared latent scale.Earlier NLP uses treated models as respondents and benchmark questions as items.Source: §5 · p. 9
  • Unlike query-wise BT rescaling, RCP uses shared criteria to set each query’s scale and offset while keeping its tournament order.Yan et al. (2024) adjust pointwise labels to respect pairwise preferences; concurrent work rescales each query’s BT scores to a fixed range (Georgiev et al., 2026).Source: §5 · p. 9

6 CONCLUSION

RCP combines relative LLM preferences and rubric judgments through IRT, and RCP-nDCG agrees with human comparisons while separating rerankers more finely. The conclusion also identifies limits of perceived-relevance judgments and outlines possible extensions beyond retrieval.

6 CONCLUSION

  • RCP-nDCG reaches AUC 0.910 versus 0.651 for qrels, and agrees with annotators in 72.4% of 185 contests where exactly one metric agrees.With either TREC-DL judge, it favors NIST’s winner on every NIST-decisive pair.Source: §6 · p. 9
  • The rubric criteria provide an interpretable relevance scale and most of the added reranker-pair separation, while IRT raises mean-score/mean-grade correlation from 0.538 to 0.795.The tournament resolves distinctions too fine for the rubric alone; blind LLM judges prefer RCP-nDCG’s ideal top-10 over Count-nDCG’s in 80.6–98.8% of queries with a preference.Source: §6 · p. 9
  • RCP measures perceived relevance rather than factual correctness and ignores redundancy between documents; annotator grades are within one step in 77.8% of cases.The annotators’ questions resemble the rubric by design, and LLM judges may share rerankers’ preferences.Source: §6 · p. 9
  • Shared rubrics could calibrate pooled judgments beyond retrieval, while multidimensional IRT or calibrated gains could support Pareto comparisons or continuous rewards.The proposed extensions include correctness and safety dimensions and rewards that rise fastest where the rubric measures most precisely.Source: §6 · p. 9

AI USE STATEMENT

The authors used generative AI across implementation, data preparation, analysis, mathematical writing, and paper editing, while stating it did not devise the framework or hypotheses. They reviewed the AI-assisted work and take responsibility for the final content and artifacts.

AI USE STATEMENT

  • Generative AI supported implementation, dataset cleaning and reformatting, data analysis, mathematical claim formulation and proof writing, and readability edits.It also helped create or edit code and figures and identify and validate references.Source: AI USE STATEMENT · p. 10
  • The authors state that AI did not devise the conceptual framework, generate synthetic datasets, propose or refine hypotheses, or interpret results.A coding agent edited rubric wording during rubric development.Source: AI USE STATEMENT · p. 10
  • No LLM annotation replaced human annotation, and the authors reviewed the AI-assisted work and accept responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The human study used paid external contractors, and the authors report no formal institutional review board review. The released data anonymizes annotators and omits their free-text comments.

ETHICS STATEMENT

  • The study used 46 paid external contractors at $35 per hour, with informed consent and no formal institutional review board review.Source: ETHICS STATEMENT · p. 10
  • Released data replaces annotator identities with one-way hashes and omits free-text comments.Source: ETHICS STATEMENT · p. 10
  • The datasets are public and used under their stated licenses or terms, and LLM judges may share biases with other LLM-based evaluation systems.Source: ETHICS STATEMENT · p. 10

Appendices

The appendices provide protocol, calibration, human-study, and robustness details for the main text. They map each main-text claim to supporting analyses and checks.

Appendices

  • Appendix D details the human study, including its design, annotator experience, and grading reliability.The guide points to Appendix J for what annotators saw and Appendix I for reliability.Source: Appendices · p. 17, Appendix E.1 · p. 17
  • Appendix C documents the Section 3 protocol, judging budgets, prompts, and proofs.It also covers calibration fitting and judge setup.Source: Appendices · p. 17

Appendix D

  • Appendices E, F, G, and H expand Section 4’s calibration, human-label comparisons, failure modes, robustness checks, and verdict analyses.Appendix D covers criterion parameters, query-level calibration, transfer, and label-free checks.Source: Appendix D · p. 17, Appendix E.1 · p. 17

A LIMITATIONS

The appendices qualify RCP’s scope and define its evaluation procedures. They specify what the metric measures, how scores and uncertainty are assessed, and how judgments are collected and calibrated.

A LIMITATIONS

  • RCP measures perceived relevance, not factual correctness, and pointwise gains do not credit redundancy or document complementarity.Differences below 0.02 in RCP-nDCG@5 are read as ties because annotators side with RCP-nDCG only at chance in that range. Calibration and human evidence also have limits: annotations overlap across human analyses, annotator agreement is imperfect, and external LLM judges may share systematic preferences with the primary judge.Source: A LIMITATIONS · p. 18
  • RCP is validated as an evaluation method; models trained on its feedback are best tested on other labels, and new pool documents require new judgments and refitting.Scores are comparable only within one calibration fit, and the fit ignores Stage A score uncertainty.Source: A LIMITATIONS · p. 18

B.1 METRICS AND TIES

  • The four compared metrics use nDCG with different gains, and tie handling varies by suite and can affect sensitivity by at most 0.4 points.The gain variants are qrel grades, criterion pass share, centered raw BT scores, and calibrated RCP probabilities.Source: §B.1 · p. 18

B.2 STATISTICS

  • Sensitivity counts reranker pairs separated by a paired t-test; the appendix also defines concordance, calibration scores, reliability statistics, and contest verdict measures.Intervals generally use query-clustered bootstraps, while human-contest rates use analytic intervals and margin AUC uses a delete-one-query jackknife.Source: §B.2 · pp. 18–19, §B.3 · p. 19

C.1 PROTOCOL, POOLS, AND RERANKERS

  • The full protocol uses top-150 candidate pools built from three fused retrievers, with relevant qrel documents added when missed.The reranker evaluation includes 14 systems; the human study compares pairs from 10, with results also reported without preference-trained ZeroEntropy rerankers.Source: §C.1 · pp. 19–20, Table 2 · p. 19

C.2 STAGE A AND STAGE B SCHEDULES

  • Stage A samples random, stratified, then adaptive document windows, with adaptive calls targeting uncertain, highly ranked, and rarely compared boundaries.Stage B asks about each document independently and uses balanced appearances before grouping documents by preliminary estimates.Source: §C.2 · pp. 20–21

C.3 CALIBRATION FIT

  • Calibration maps each query’s BT scores to an ability scale using a positive scale and offset, while shared criterion parameters place queries on a common scale.The fit is a maximum a posteriori estimate; priors keep estimates finite when all documents pass or fail every criterion. Stage A score uncertainty is not propagated.Source: §C.3 · p. 21

C.4 JUDGES, BUDGETS, AND COST

The appendices describe judge costs, dataset-specific evaluation settings, and the human-study design. They also report that reduced judgment budgets preserve the internal RCP-nDCG leaderboard closely on TREC-DL.

C.4 JUDGES, BUDGETS, AND COST

  • With 50.2% of judgments, the largest mean RCP-nDCG@10 change is 0.009 and leaderboard Kendall correlation is 0.987; at 35.0%, they are 0.026 and 0.969.Neither reduced budget reverses a system pair separated by the full budget; this check compares RCP-nDCG with itself, not with NIST.Source: §C.4 · p. 22

C.5 NANOBEIR AND BRIGHT

  • NanoBEIR uses 649 queries across 13 datasets and judged pools of 150 documents; BRIGHT evaluation uses 1,366 shared queries and removes officially excluded documents from scored rankings.NanoBEIR pool construction includes special handling for missing qrel positives and query-identical documents; BRIGHT pools retained excluded documents during calibration.Source: §C.5 · p. 22

C.6 VIDORE V3

  • On ViDoRe v3, six language versions share labels, so analysis uses 2,419 native questions rather than counting each version as a separate query.The 14,514 question rows were fitted in one calibration; each question has a 150-page pool, and assessors graded pages on a three-point scale.Source: §C.6 · pp. 22–23

C.7 TREC-DL 2019 AND 2020

  • TREC-DL evaluates 2019 and 2020 separately against NIST grades, using both full-pool and top-150 panels and paired tests to identify NIST-decisive reranker pairs.The main fit pools both years with ungraded MS MARCO development queries; the reduced judging protocol uses fewer adaptive windows and omits reversed Stage A copies.Source: §C.7 · p. 23

D HUMAN STUDY PROTOCOL

  • The human study has a document-grade layer and a list-verdict layer, but both use the same annotations and are not independent evidence.The study asks whether annotators side with RCP-nDCG when exactly one of it and qrel-nDCG names their preferred list.Source: D HUMAN STUDY PROTOCOL · p. 23

D.1 DESIGN AND SAMPLE

The appendices detail how the human comparisons and calibration checks were conducted. They report panel-grade reliability and cross-dataset transfer results, while noting rubric and task-specific constraints.

D.1 DESIGN AND SAMPLE

  • The 311 contests oversample metric disagreements, and annotators grade the union of each reranker’s top five documents without seeing rankings or metric scores.The sample covers 292 queries; analysis depth is five for verdicts although contest draws compare metrics at depth ten.Source: §D.1 · p. 24, §D.2 · p. 24, Table 5 · p. 24

D.2 WHAT ANNOTATORS SAW

  • Annotator grades take the most stringent yes as the grade, and list picks use nDCG@5 under a shared ideal ranking, with near-equal margins treated as ties.A Copeland score can break ties using annotators’ pairwise rankings of equally graded documents.Source: §D.2 · pp. 24–25, Table 6 · p. 25

D.3 ANNOTATORS AND TERMS

  • The 46 external annotators each held at least a subject-relevant BSc, and three graded each contest; admitting one excluded annotator barely changes the main measure.Re-admission moves the measure from 72.4% to 72.3%.Source: §D.3 · p. 25

D.4 RELIABILITY

  • Panel-mean grade reliability is ICC(1,k)=0.802, while single-grade Krippendorff’s α is 0.571.Annotator pick agreement is 72.1%; leaving out any one annotator keeps verdict agreement between 70.2% and 73.8%.Source: §D.4 · p. 25

E DIAGNOSTICS AND CALIBRATION

E.1 FITTED PARAMETERS AND TRANSFER ACROSS DATASETS

  • 60.6%: held-out binary cross-entropy falls against the pass-rate baseline on NanoBEIR with QWEN3.5-397B.Reductions are 64.1% with GPT-OSS-120B on NanoBEIR, and 46.5% and 49.6% on BRIGHT with QWEN3.5-397B and GPT-OSS-120B, respectively. Setting all discriminations to 1 raises held-out BCE by 1.1–1.3% on NanoBEIR and 2.0–2.4% on BRIGHT.Source: §E.1 · p. 26

E.2 LABEL-FREE DIAGNOSTICS

  • On TREC-DL, criterion difficulties do not follow rubric order, and the rubric is most informative below the documents filling the top ten.C3 is easier to pass than C2; C5 is much harder, and removing C2 raises top-10 information efficiency. A rubric can pass diagnostics yet miss relevance aspects.Source: §E.2 · pp. 26–27, Table 8 · p. 27

E.3 CALIBRATION AGAINST HUMAN GRADES AND SIMPLER MAPS

Calibration improves cross-query agreement with human grades and tests placement against external qrels, while alternative gain shapes largely preserve leaderboards. The section also checks how reliably the tournament order reproduces across translations.

E.3 CALIBRATION AGAINST HUMAN GRADES AND SIMPLER MAPS

  • The gain’s ECE is 0.054 for Useful-or-better grades, versus 0.167 for Topical-or-better and 0.187 for Answers-or-better.No human grade enters the fit, making this correspondence an empirical result.Source: §E.3 · p. 27
  • On ViDoRe v3, the 2PL has a Brier score of 0.0225 versus 0.0426 for logistic regression, and wins on 94.9% of queries.Pooled ECEs are 0.013 for the 2PL and 0.012 for logistic regression; on NanoBEIR, the 2PL has lower Brier scores on 97.8% of queries.Source: §E.3 · pp. 27–28
  • Calibration raises Spearman correlation with mean grade over raw BT by 0.190 on NanoBEIR, 0.072 on BRIGHT, and 0.067 on ViDoRe v3.Per-query z-scores lower Spearman correlation and AUC below raw BT in every suite; calibration also improves ordering of same-suite, cross-query pairs graded differently by the same annotator.Source: §E.3 · p. 28, Table 9 · p. 28

E.4 PLACEMENT AGAINST NIST GRADES AND VIDORE QRELS

  • On TREC-DL, pooled score-to-NIST-grade correlations are 0.649 in 2019 and 0.595 in 2020, below ceilings of 0.768 and 0.743.Calibration raises correlation over raw BT by +0.031 in 2019, with an interval including zero, and +0.054 in 2020.Source: §E.4 · p. 28
  • On ViDoRe v3, the RCP gain has AUC 0.917 for qrel grade at least 1, versus 0.892 for raw BT, an improvement of +0.025.Within-question placement of grade-1 pages above unjudged pages is 0.871; for grade-2 pages, it is 0.941.Source: §E.4 · p. 28
  • Across six ViDoRe v3 language versions, tournament orders reproduce at Spearman 0.962, versus 0.815 for Count and 0.888 for pooled Count.The pooled Count uses Stage B placements from three versions and has about as many judgments per page as RCP; reproducible orders can still reproduce a judge’s error.Source: §E.4 · pp. 28–29

E.5 GAIN SHAPE

  • Alternative gains yield leaderboard correlations of at least 0.859, while NanoBEIR min-max gain separates 68.6% of pairs versus 63.5% for the paper’s gain.The authors interpret separated-pair share as resolution, not validity, and retain the discrimination-weighted gain for its probability reading and test-information slope.Source: §E.5 · p. 29

F.1 THE LABEL LAYER

The human-label comparisons favor the RCP gain over qrels on document-level usefulness and in blind set-selection studies, while HotpotQA exposes a boundary for documentwise judgments. The ranking analyses report nDCG failure modes, pair separation, and agreement across judges.

F.1 THE LABEL LAYER

  • Pooled AUC at Useful is 0.910 for the RCP gain versus 0.651 for qrels; at a matched 28.7% flag rate, RCP recovers 64.1% of useful documents versus 47.5% for qrels.The false-positive rates are 7.3% for RCP and 17.3% for qrels.Source: §F.1 · p. 29, Figure 9 · p. 30, Table 10 · p. 30
  • The calibrated score’s Spearman correlation with the panel’s mean grade is 0.792, near the estimated 0.773 reliability of a second panel of the same size.The reliability estimate uses random half-panels and the Spearman–Brown formula.Source: §F.1 · p. 29

F.2 STUDY 1: QREL SETS AGAINST RCP SETS

  • External judges preferred RCP-selected sets in 77.9% of 222 decided NanoBEIR cases and 87.8% of 843 decided BRIGHT cases.Study 1 conditions on queries whose qrel and RCP sets differ; external grades cover all 2,268 study documents.Source: §F.2 · p. 31

F.3 MULTI-HOP QUERIES: HOTPOTQA

  • NanoHotpotQA exposes a boundary: documentwise relevance can favor a redundant primary-fact page over the secondary-fact document required by the qrel set.Only six HotpotQA contests enter the external human study, too few for a claim; RCP has no multi-document criteria and does not model diversity or individual user needs.Source: §F.3 · p. 31

G.1 FAILURE MODES OF NDCG@10

  • At least one nDCG@10 failure mode occurs on 45.6% of NanoBEIR queries and 30.7% of BRIGHT queries, versus 7.0% and 5.6% on TREC-DL in 2019 and 2020.The modes are saturation, floor, or compression; Figure 10 shows their share across suites.Source: §G.1 · p. 31, Figure 10 · p. 32

G.2 SENSITIVITY PER DATASET

  • RCP-nDCG separates more reranker pairs than qrel-nDCG on 11 of 13 NanoBEIR datasets and all 12 BRIGHT tasks, except ArguAna and HotpotQA.On the exceptions, RCP-nDCG separates fewer pairs.Source: §G.2 · p. 32

G.3 RERANKER SCORES PER DATASET

  • Tables 11–14 report reranker scores across NanoBEIR, BRIGHT, and ViDoRe v3; only reranker ordering is comparable because the metrics differ in level.Table 14 also gives NanoBEIR and BRIGHT suite means under both judges.Source: §G.3 · p. 32

G.4 SECOND JUDGES

  • Across datasets, two judges’ RCP-nDCG leaderboards correlate at mean Spearman 0.959 on NanoBEIR and 0.988 on BRIGHT; TREC-DL systems correlate at Kendall 0.987.No NanoBEIR or BRIGHT dataset falls below 0.916; the two judges’ calibrated TREC-DL passage scores correlate at Spearman 0.934.Source: §G.4 · p. 32

G.5 THE ZERANK RERANKERS

Across reranker analyses, RCP-nDCG often favors zerank systems and resolves ties that Count-nDCG leaves unresolved, with human and NIST comparisons providing checks. Relevance rubrics generally preserve separated pair decisions, while deliberately poor rubrics can reverse them.

G.5 THE ZERANK RERANKERS

  • The three zerank rerankers' mean RCP-nDCG@10 exceeded that of the other 11 rerankers by 5.3 pp overall and on 11 of 13 NanoBEIR datasets.Their calibrated-score correlation was higher than that of the other rerankers, but the analysis could not distinguish better rankings from shared LLM preferences.Source: §G.5 · p. 33
  • 78.8% of 66 human contests favored the zerank list when RCP-nDCG and qrel-nDCG disagreed, and 85.3% favored it when RCP-nDCG's margin was at least 0.02.The zerank lists also won 69.3% of 153 decided contests; on ViDoRe v3, zerank-2 and zerank-1 rose from qrel ranks 4 and 5 to ranks 1 and 2.Source: §G.5 · p. 34

G.6 RUBRIC, PROMPT, AND SIMULATION ROBUSTNESS

  • Of 6,004 rubric-prompt test opportunities, all 20 reversals came from rubrics designed to fail; the 59 relevance rubrics reversed none of the 76 reference-separated pairs.With the shipped rubric, the four alternate prompts also reversed no separated pair; the full five-criterion rubric ranked third in information efficiency at 0.407.Source: §G.6 · p. 35
  • In simulation, RCP-nDCG detected more true differences than nDCG with an accurate judge, but sometimes fewer with noisier judges, depending on label type.Both metrics' false-positive rates on truly equal pairs stayed near the nominal 5% level.Source: §G.6 · p. 36

G.7 RCP-nDCG AND COUNT-NDCG

  • External judges favored the tournament's order in 86.8% of decided ideal-list comparisons and 84.0% of minimal pairs; Count-tied pairs aligned with NIST 0.671 of the time.Count-nDCG tied up to 66.8% of the five best rerankers' pairs on TREC-DL, while RCP-nDCG tied at most 1.0%.Source: §G.7 · pp. 37–38

H.1 NIST-DECISIVE RERANKER PAIRS

  • On TREC-DL pool-panel comparisons with linear gain, RCP-nDCG agreed with NIST on every NIST-decisive reranker pair in both years.Across both years, RCP-nDCG agreed on 36 of 39 decisive pairs and separated only one pair in the opposite direction.Source: §H.1 · p. 38
  • On DL19, NIST and independent reassessors separated none of the ten pairs among the five rerankers ranked highest by RCP-nDCG.The leaderboard comparison therefore had no human-reference ordering for those leading rerankers.Source: §H.1 · p. 39

H.2 NIST POSITIVES THAT RCP SCORES LOWEST

  • All 14 rerankers ranked these passages below other positives within their queries (mean AUC 0.879, 95% CI 0.865–0.894); on DL19, reassessors rated 2.7% highly relevant or better, versus 40.1% of other positives.Source: §H.2 · p. 39

H.3 HUMAN VERDICTS AND CONTROLS

Blind human verdicts and external LLM comparisons favor RCP-nDCG in many cases where it disagrees with qrel- or Count-nDCG. The section also reports controls for verdict artifacts and outlines the theory results.

H.3 HUMAN VERDICTS AND CONTROLS

  • The margin AUC was 0.810 for RCP-nDCG versus 0.521 for qrel-nDCG, and RCP-nDCG led in every suite and without zerank rerankers.The main human-verdict measure remained between 70.6% and 74.4% when each annotator was left out in turn.Source: §H.3 · p. 40

H.4 LLM JUDGE PANEL AND STUDY 2

  • External judges preferred RCP-nDCG's list in 78.4% of decided NanoBEIR sign-flips and in 74.4% of decided BRIGHT sign-flips.The majority favored RCP-nDCG on 12 of 13 NanoBEIR datasets; display order and list length did not explain the preference.Source: §H.4 · pp. 41–42

H.5 EXTERNAL LLM JUDGES

  • Across external-judge comparisons, the majority favored the RCP side in 63.1% of per-query disagreements, 86.8% of ideal-list cases, and 84.0% of minimal pairs.When both metrics were beyond their tie zones, judges favored RCP-nDCG in 66.1% of decided cases; the split was not independent evidence.Source: §H.5 · pp. 42–43

I THEORY

  • The theory section proves stated properties of the gain and calibration, and relates RCP-nDCG to Count-nDCG as its rubric-only special case.Source: I THEORY · p. 43

I.1 THE GAIN, ITS SLOPE, AND ORDER PRESERVATION

This appendix establishes properties of RCP’s gain and calibration, explains tie preservation and the role of tournament ordering, and documents the prompts, study procedures, released assets, and annotation privacy limits. It also situates RCP among related calibration methods and records author contributions.

I.1 THE GAIN, ITS SLOPE, AND ORDER PRESERVATION

  • RCP gains are smooth, bounded between 0 and 1, strictly increasing, and have slope equal to test information divided by total discrimination.Under the constraint that discriminations sum to C, the slope is test information divided by C.Source: §I.1 · p. 43

I.2 IDENTIFICATION OF THE CALIBRATION

  • The calibration likelihood permits a shared affine transformation of discriminations and difficulties, while hard constraints uniquely fix scale and difficulty location.The constraints identify the parameters; priors do not determine identification.Source: §I.2 · p. 44

I.3 THE RUBRIC-ONLY SPECIAL CASE AND THE RESOLUTION OF DISCRETE GAINS

  • RCP-nDCG separates 63.5% of NanoBEIR reranker pairs, versus 60.7% for Count-nDCG.Count-nDCG cannot distinguish rerankers that order documents with equal pass shares, while RCP gains inherit the Stage A order. With an ordered rubric ladder, at least ⌈K/(C + 1)⌉ documents share a response pattern.Source: §I.3 · p. 45
  • For documents that fail every rubric criterion, RCP assigns a mean gain of 2.9%, close to the 3.2% of external grades rated Useful or better.These documents comprise 45.8% of TREC-DL pool documents; RCP’s difference from the human rate is -0.2 pp with interval [-1.9, +1.3] pp, while Count’s zero gain differs by -3.2 pp with interval [-5.0, -1.7] pp. Setting RCP gains to zero changes paired-test decisions in 4 of 182 TREC-DL cells, 54 of 1,183 NanoBEIR cells, and 12 of 1,092 BRIGHT cells, reversing no separated pair.Source: §I.3 · p. 45

J PROMPTS

  • The prompts were refined using 300 NanoBEIR documents, about 0.3% of its pools, without external-annotator or NIST grades; BRIGHT, ViDoRe v3, and TREC-DL were held out.Criteria wording was edited to increase agreement between QWEN3.5-397B and two authors.Source: J PROMPTS · p. 45

J.1 STAGE A PROMPT

  • The printed Stage A prompt was used unchanged for TREC-DL, but stored NanoBEIR, BRIGHT, and ViDoRe v3 runs do not establish that it produced their scores.The prompt asks for a strictly descending ranking and calibrated logit-scale scores, with score gaps reflecting confidence.Source: §J.1 · pp. 46–47

J.2 STAGE B PROMPT

  • Stage B asks five standalone yes/no diagnostic questions per document; the printed prompt was used on ViDoRe v3 and TREC-DL, while NanoBEIR and BRIGHT used an earlier version with the same C1–C5 wording.The criteria cover topical relevance, information utility, entity/detail match, direct answer, and thorough treatment.Source: §J.2 · pp. 47–48

J.3 META-JUDGE AND EXTERNAL-JUDGE PROMPTS

  • Studies 1 and 2 compare candidate annotations or reranker top-10 lists using external and meta-judges, with Study 2 also including author judgments on shortened documents.The studies used three judges at maximum reasoning effort, no output cap, and provider-default temperature; Study 2 documents were shortened to 1,500 characters.Source: §J.3 · p. 49
  • Each external-judge case compares two ranked lists using task-specific relevance descriptions, blinded document identifiers and metric hints, and swapped labels and display order.Three judges assess which list better serves the query, weighting top ranks more heavily; documents over 2,000 characters are truncated at a sentence boundary.Source: §J.3 · p. 49

K ASSETS AND RELEASE

  • Released human grades use one-way hashes instead of annotator names and emails; free-text comments are withheld because they were not screened for personal information beyond email replacement.Source: K ASSETS AND RELEASE · p. 50
  • The release includes both pipeline stages, calibration, metric, prompts, calibrated query-document scores, gain parameters, and 7,080 human grades on 2,268 documents.BRIGHT and NanoArguAna labels also cover documents removed from some rankings, which are excluded from rankings and tables except in specified follow-up analyses.Source: K ASSETS AND RELEASE · p. 50

L EXTENDED RELATED WORK

  • RCP preserves tournament order and fits per-query scale and offset from shared binary criteria, unlike Yan et al. (2024)’s minimally perturbed pointwise labels.The passage relates shared criteria to IRT linking and noisy-annotator label models.Source: L EXTENDED RELATED WORK · p. 50
  • Unlike SABER-Math’s fixed-range rescaling and zELO’s per-query centering, RCP’s shared rubric supplies score units and zero points comparable across queries.Source: L EXTENDED RELATED WORK · p. 50
  • Count-nDCG is RCP’s rubric-only special case, while RCP calibrates tournament scores using its rubric without human labels.The passage contrasts rubric and checklist aggregation methods and other work that calibrates LLM judges against human labels.Source: L EXTENDED RELATED WORK · pp. 50–51

M AUTHOR CONTRIBUTIONS

  • Schmidt conceived the contributions, led most analyses and the annotation study, ran TREC-DL experiments, and wrote the final paper version.Crisostomi ran initial experiments and drafted an earlier manuscript; Lassance provided regular feedback, and Reimers approved and supported the project.Source: M AUTHOR CONTRIBUTIONS · p. 51
Loading 2609.35739v1…