Source-linked AI summary
Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
Abhishek Divekar
TL;DR
Human evaluation is costly and LLM judges can bias ranking metrics. PRECISE extends PPI to correct that bias using small human-labeled and large LLM-judged sets, including hierarchical Precision@K. On ESCI, Sonnet reduced Precision@4 standard error from 4.45 to 3.50, and production A/B testing confirmed the ranking of three variants with +407 bps in daily sales.
Problem
Human evaluation is expensive, while small labeled sets are noisy and LLM judges can systematically bias evaluation metrics.
Method
PRECISE extends PPI by combining a small human-labeled set with a large LLM-judged set and sparsely reformulating hierarchical Precision@K.
Results
4.45 to 3.50: Claude 3 Sonnet reduced ESCI Precision@4 standard error by 21% relative, and production A/B testing confirmed the predicted ranking with +407 bps in daily sales.
Takeaways & Limitations
The framework provides statistically bias-corrected ranking evaluation and supported reliable discrimination among production system variants.
Takeaways & Limitations
The hierarchical extension has been validated only on Precision@K, assumes conditional independence across documents, and requires gold and unlabeled data from the same distribution.
Abstract
from arXiv · showhide
With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.
1 Introduction
PRECISE addresses costly human evaluation and biased LLM judges by statistically correcting judge bias with a small human-labeled set and a large LLM-annotated set, while extending PPI to hierarchical ranking metrics.
- Human evaluation is expensive, and small labeled sets yield confidence intervals too wide to distinguish genuine system improvements from noise.
- LLM-as-a-Judge methods reduce reliance on human annotation but can systematically bias evaluation metrics when used as substitutes.
- PRECISE accepts LLM judge bias and corrects it statistically rather than attempting to build a better judge.
- The framework extends Prediction-Powered Inference by combining a small human-labeled gold set with a large LLM-annotated set.
- PPI produces a provably unbiased estimate, while additional LLM-judged examples reduce metric-estimate variance without introducing new bias.
- For Precision@K, sparse output-space reformulation resolves the mismatch between per-document annotations and per-query metric computation.
2 Method
The method combines human-grounded and LLM-based metric estimates through PPI, then makes hierarchical Precision@K tractable by representing only the top-K relevance outcomes.
- PPI estimator: PPI++ combines a small gold set with a much larger LLM-annotated set to estimate the target metric.
- PPI estimator: The estimator combines an LLM-based estimate with a bias-correction term based on human-grounded metric values.
- PPI estimator: The parameter λ is tuned to minimize variance, while the estimator remains unbiased for any λ > 0.
- Bias correction: The correction measures the judge’s deviation from human ground truth and subtracts that deviation from the LLM-only estimate.
- Hierarchical metrics: Naive PPI for Precision@K has output space {0, 1}^|C|, which is intractable when the corpus contains millions of documents.
- Hierarchical metrics: Because Precision@K depends only on the top-K retrieved documents, the metric becomes a scaled dot product over sparse vectors with output space {0, 1}^K.
- Hierarchical metrics: The LLM supplies per-document relevance probabilities, which are combined into a conditionally independent distribution over K-length binary vectors and marginalized over 2^K outcomes.
3 Results
Experiments show that PPI reduces estimation uncertainty and supports reliable production ranking: it improves Precision@4 estimation on ESCI and its predicted variant ordering matches A/B-test outcomes.
- ESCI benchmark: n=30 gold annotations and N=60,000 LLM-judged queries were used to evaluate Claude 3 Sonnet and Haiku on ESCI retrieval.
- ESCI benchmark: 4.45 to 3.50: Sonnet reduced Precision@4 standard error by 21% relative while maintaining low bias, from 1.04 to 0.70.
- ESCI benchmark: Haiku achieved standard error 3.86 and bias 0.29 at 12× lower inference cost than Sonnet.
- Production A/B test: T1 > T2 > Control: using n=100 human labels and N=8,400 LLM judgments, the framework predicted the production ranking in two hours of expert annotation.
- Production A/B test: +407 bps in daily sales and +571 bps in click-through rate: A/B testing confirmed the predicted production ranking.
- Scope: The hierarchical formulation was validated on Precision@K and applies in principle to other metrics aggregating fine-grained judgments.
4 Future Work
Future work targets reducing human-label dependence, extending bias-corrected inference, and improving evaluation beyond a single all-purpose judge. Proposed directions include synthetic covariates, doubly robust estimation, multi-judge or multi-objective approaches, and calibrated agentic critics.
- Synthetic covariates: Synthetic covariates could reduce PRECISE’s reliance on human labels by supplying textual covariates alongside fixed gold labels.The paper identifies human-label over-reliance as a major drawback, while noting that synthetic covariates may remain usable for estimation.
- Doubly robust estimation: Doubly robust estimation could provide a pathway toward real-time, bias-corrected metric inference and online evaluation.
- Multiple Judges: Aggregating multiple LLM judges may better match human ratings, while jointly tuning several rubrics in one prompt is brittle in practice.
- Agentic critics: Extending PRECISE to agentic critics could produce calibrated critic signals from minimal human labels for more reliable agent optimization.
Ethics Statement
The production evaluation protected privacy by anonymizing user queries and used domain experts during normal working hours rather than crowdworkers. The framework reduces, but does not eliminate, human annotation needs.
- User queries were anonymized before processing by LLM judges or human annotators, removing personally identifiable information.
- Domain experts annotated during normal working hours as part of their regular responsibilities, with no crowdworker labor used.
- The framework reduces but does not eliminate human annotation and is not intended to replace human evaluation entirely.
Limitations
The framework’s validated scope, document-independence assumption, and dependence on matched gold and unlabeled distributions define important boundaries for applying hierarchical PPI.
- Hierarchical PPI has been validated only for retrieval Precision@K, leaving other hierarchical metrics untested.
- The conditional independence assumption across documents may fail when document relevance depends on other retrieved documents, such as in diversity-sensitive ranking.
- The gold set must come from the same distribution as the unlabeled set; distribution shift could degrade bias correction.