PaperlayerView exact revision on arXiv

Source-linked AI summary

Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges

Chen Feng, Minghe Shen, Ananth Balashankar, Carsten Gerner-Beuerle, Miguel R. D. Rodrigues

arXiv:2601.20913v1cs.LGcs.AIcs.CV

TL;DR

Reliable LLM certification is difficult because scalable judges are imperfect and human evaluation is costly, while naive judge use can invalidate statistical guarantees. The paper introduces a calibrated noisy hypothesis test using human labels to estimate judge error rates and a variance-corrected threshold, obtaining finite-sample Type-I error control and characterizing power regimes and the Oracle Gap. Its experiments support the theory and its limitations include restricted pass/fail evaluation and unresolved challenges with subjective, noisy-ground-truth tasks.

  • Problem

    Human evaluation is costly to scale, while LLM judges can introduce noise and bias that undermine statistically reliable certification.

  • Method

    The framework estimates judge TPR and FPR from a small human-labelled calibration set and uses a variance-corrected test on a large judge-labelled dataset.

  • Results

    The framework guarantees finite-sample Type-I error control despite judge-parameter uncertainty, characterizes when noisy testing outperforms direct testing, and finds an Oracle Gap between practical and idealized procedures.

  • Takeaways & Limitations

    The method provides interpretable diagnostics for judge reliability, judge selection, sample-size planning, and evaluation-protocol design.

  • Takeaways & Limitations

    The analysis is restricted to pass/fail evaluation, while granular quality assessment and subjective tasks with noisy human labels remain open challenges.

Abstract

from arXiv · show

Reliable certification of Large Language Models (LLMs)-verifying that failure rates are below a safety threshold-is critical yet challenging. While "LLM-as-a-Judge" offers scalability, judge imperfections, noise, and bias can invalidate statistical guarantees. We introduce a "Noisy but Valid" hypothesis testing framework to address this. By leveraging a small human-labelled calibration set to estimate the judge's True Positive and False Positive Rates (TPR/FPR), we derive a variance-corrected critical threshold applied to a large judge-labelled dataset. Crucially, our framework theoretically guarantees finite-sample Type-I error control (validity) despite calibration uncertainty. This distinguishes our work from Prediction-Powered Inference (PPI), positioning our method as a diagnostic tool that explicitly models judge behavior rather than a black-box estimator. Our contributions include: (1) Theoretical Guarantees: We derive the exact conditions under which noisy testing yields higher statistical power than direct evaluation; (2) Empirical Validation: Experiments on Jigsaw Comment, Hate Speech and SafeRLHF confirm our theory; (3) The Oracle Gap: We reveal a significant performance gap between practical methods and the theoretical "Oracle" (perfectly known judge parameters), quantifying the cost of estimation. Specifically, we provide the first systematic treatment of the imperfect-judge setting, yielding interpretable diagnostics of judge reliability and clarifying how evaluation power depends on judge quality, dataset size, and certification levels. Together, these results sharpen understanding of statistical evaluation with LLM judges, and highlight trade-offs among competing inferential tools.

1 INTRODUCTION

The paper addresses statistically reliable LLM certification when human evaluation is costly and LLM judges are noisy or biased. It introduces a calibrated, variance-corrected test with finite-sample validity, characterizes when noisy testing has greater power than direct evaluation, and validates the framework empirically.

  • Human evaluation is costly and difficult to scale, while LLM judges can introduce noise, bias, prompt sensitivity, and domain-dependent errors.
  • Unlike PPI’s black-box control-variate treatment, the framework explicitly models judge error profiles to support certification and diagnostic interpretation.
  • The framework calibrates judge TPR and FPR on a small human-labelled set and uses them to construct a variance-corrected hypothesis test on a large judge-labelled set.
  • Theoretical analysis derives conditions under which noisy testing has higher statistical power than direct human evaluation and quantifies the Oracle Gap caused by estimating judge parameters.
  • Experiments on Jigsaw, Hate Speech, and SafeRLHF with multiple LLM-judge pairs align with the theoretical predictions and identify regimes where the method outperforms direct testing.
  • The framework provides interpretable diagnostics for judge selection, sample-size planning, and evaluation-protocol design.

2 RELATED WORK

The related work spans automatic and human LLM evaluation, LLM-as-a-Judge methods, and statistical inference with imperfect labels. The paper positions its contribution as hypothesis testing that uses a small human holdout to calibrate scalable judge labels while controlling Type-I error.

  • Evaluation paradigms for LLMs: Automatic benchmarks scale broadly but can be affected by contamination, while human evaluation remains important for complex tasks but is costly.
  • LLM as a judge: LLM-based judging scales across code, dialogue, and multimodal tasks but exhibits biases, prompt sensitivity, and attack vulnerabilities.
  • Statistical foundations: The paper uses a small human holdout for calibration and incorporates its estimates into a test that controls Type-I error.
  • Statistical foundations: Related statistical foundations include classical proportion testing, conformal guarantees under exchangeability, and PPI methods combining limited clean labels with many imperfect labels.

3 CERTIFICATION SETTING

The certification setting treats model reliability as a hypothesis test about whether the failure rate exceeds a user-specified threshold. It contrasts a small human-labelled dataset with a much larger judge-labelled dataset and defines judge quality through TPR and FPR.

  • The binary ground-truth and judge variables encode whether a response is incorrect or judged incorrect, respectively.
  • TPR measures how often the judge flags genuinely unreliable responses, whereas FPR measures how often it incorrectly flags reliable responses.
  • The analysis assumes a useful judge has TPR greater than FPR and excludes equality because the judge then carries no information about ground truth.
  • Certification tests whether the model failure rate RM exceeds the user-specified threshold α, with Type-I error controlled at significance level ζ.
  • The setup uses a small human-labelled dataset DM and a large judge-labelled dataset DJ with nJ much greater than nM.
  • Direct HT uses only the small ground-truth dataset, remains statistically valid, and has power constrained by its limited sample size.

4 NOISY HYPOTHESIS TESTING: PROCEDURE

Noisy HT reformulates the true-failure-rate test in terms of judge-labelled failures, calibrates the judge using human-labelled data, and applies a variance-corrected threshold to the large judge-labelled dataset. With known judge parameters, the procedure approaches an oracle test.

  • Noisy HT converts the target threshold α into α′ = FPR + (TPR − FPR) · α, linking the noisy test to the original failure-rate hypothesis.
  • The procedure has two operations: judge modelling on the human-labelled dataset and judge-based hypothesis testing on the large judge-labelled dataset.
  • Judge modelling augments human-labelled samples with judge predictions and estimates the judge’s TPR and FPR.
  • The estimated threshold is α̂′ = FPR̂ + (TPR̂ − FPR̂) · α.
  • Testing computes the noisy failure rate on DJ, constructs a variance-corrected critical threshold, and rejects the null when the observed rate is below that threshold.
  • The critical threshold combines the estimated null-boundary mean with variance contributions from judge labels and finite-sample TPR and FPR estimates to control Type-I error.
  • As nM approaches infinity, the procedure defaults to the oracle noisy test with known judge parameters.

5 NOISY HYPOTHESIS TESTING: GUARANTEES

The framework provides finite-sample Type-I error control despite uncertainty in estimated judge parameters, then characterizes its Type-II error and comparative power. Its guarantees identify when noisy testing can outperform direct evaluation and quantify the performance cost of calibration uncertainty.

  • Type-I and Type-II Error Guarantees: Theorem 5.1 establishes Type-I error control at the desired level despite uncertainty in the judge’s performance parameters.The variance-corrected critical threshold incorporates calibration-set variance, preserving rigorous safety claims when human annotations are limited.
  • Type-I and Type-II Error Guarantees: Judge quality and model reliability determine certification power: higher TPR, lower FPR, and lower model failure rates improve statistical power.A highly reliable model may be certifiable with a moderately imperfect judge, while superior judges require smaller samples for the same power.
  • Noisy Hypothesis Testing vs Oracle Noisy Hypothesis Testing: For large nJ, the practical noisy test has higher Type-II error than the oracle test with known judge parameters.The resulting Oracle Gap quantifies the statistical cost of estimating TPR and FPR; larger calibration sets or prior bounds can reduce it.
  • Noisy Hypothesis Testing vs Direct Hypothesis Testing: For large nJ and nM, noisy testing has lower Type-II error than direct testing exactly under the condition in Theorem 5.4.A powerful judge always satisfies the adoption condition, while stricter certification requirements demand higher TPR or lower FPR.

6 EXPERIMENTS

Experiments across classification and generation broadly support the theory: noisy hypothesis testing controls Type-I error and can improve Type-II error over direct testing when judges are reliable, while calibration and prompt choices affect performance.

  • Across Jigsaw, Hate Speech, and SafeRLHF, experiments broadly align with theoretical predictions and maintain Type-I error control at the 5% significance level.
  • PPI often outperforms noisy hypothesis testing with poor judges, while a significant gap remains between PPI-based procedures and oracle noisy testing.
  • Noisy hypothesis testing can considerably outperform direct testing when judges have high TPR and low FPR.
  • TPR/FPR estimators remain unbiased as calibration size increases from 25 to 100, while their standard deviation decreases at the expected O(1/√nM) rate.
  • Federated prompts improve judge quality through higher TPR and lower FPR, producing the lowest Type-II error and the most favorable certification-power trade-off.

7 CONCLUSION

The paper concludes that its noisy hypothesis-testing framework enables statistically rigorous LLM certification by explicitly modeling judge behavior, while recognizing important scope and approximation limits.

  • The framework captures interactions among judge ability, model capability, dataset sizes, and certification requirements while providing interpretable reliability diagnostics.
  • Noisy hypothesis testing can considerably outperform conventional hypothesis testing in certain regimes, but practical methods remain separated from the idealized oracle setting.
  • The current analysis is restricted to pass/fail evaluation, leaving granular response-quality certification and subjective tasks for future work.
  • Normal approximations may lose precision for extremely small calibration sets such as nM < 5 or for rare failure events.
  • Separate validation data are required for judge selection to avoid peeking and Type-I error inflation; reusing DM requires corrections such as Bonferroni.

A.2 TYPE-I ERROR PROBABILITY

The section characterizes Type-I error for baseline and PPI-related testing procedures, using boundary reductions and Berry–Esseen arguments to establish finite-sample control.

  • Type-I error under the null is bounded by evaluating rejection probability at the boundary RM = α.
  • The baseline procedure analyzes Type-I error through a random-variable construction followed by the Berry–Esseen inequality.
  • PPI comparison: PPI methods use a shared difference-correction estimator with variant-specific scalar weights for variance reduction.
  • The procedures return reject or fail-to-reject decisions while targeting Type-I error control at the specified significance level.
  • Oracle procedure: The oracle noisy test uses judge-labelled data, known TPR/FPR, and the transformed threshold α′ = FPR + (TPR − FPR) · α.

C.2 TYPE-I ERROR PROBABILITY

The oracle and estimated-parameter analyses establish Type-I error bounds by modeling judge noise, calibration uncertainty, and evaluation-sample randomness explicitly.

  • Oracle analysis: The oracle procedure’s Type-I error is bounded at the transformed boundary RJ = α′ using a boundary argument and Berry–Esseen inequality.
  • Proxy risk: The noisy proxy risk satisfies RJ = RM · TPR + (1 − RM) · FPR under TPR > FPR.
  • Estimated parameters: Estimated-parameter Type-I analysis averages over calibration-set and judge-labelled evaluation-set randomness.
  • Critical threshold: The estimated critical threshold uses α̂′ = FPR̂ + (TPR̂ − FPR̂) · α and expands its uncertainty through TPR and FPR estimation errors.
  • Error bounds: Parallel Berry–Esseen analyses characterize Type-I and Type-II probabilities for the estimated-parameter procedure.

D.4 BEHAVIOUR OF TYPE-II ERROR PROBABILITY OF HYPOTHESIS TESTING PROCEDURE

The analysis links Type-II error to judge quality, model failure rate, calibration composition, and the gap between practical noisy testing and oracle or direct testing.

  • Judge quality: Type-II error decreases as TPR increases, provided TPR > 1/2, and increases as FPR increases, provided FPR < 1/2.
  • Model reliability: Type-II error increases with the model failure rate RM when TPR > FPR.
  • Noisy versus direct testing: The noisy test can outperform direct testing only when judge quality and calibration composition satisfy the finite-sample superiority condition.
  • Calibration effects: Calibration imbalance, such as few positive examples, increases variance and requires a larger TPR–FPR gap for noisy testing to remain superior.
  • Oracle Gap: Estimating judge parameters creates an Oracle Gap, while bounded estimation can reduce variance and narrow that gap when valid parameter ranges are available.

E.3 INITIAL EXPERIMENTS

The experiments study bounded estimation and noisy-testing behavior across datasets, showing power improvements from valid bounds while illustrating divergence between noisy and direct decisions.

  • Experimental setup: The experiments compare unbounded estimation with loose and tighter TPR/FPR bounds using synthetic settings and Jigsaw, Hate Speech, and SafeRLHF evaluations.
  • Bounded estimation: Bounded estimation consistently reduces Type-II error relative to the unbounded case and shifts performance toward the Oracle baseline.
  • Bounded estimation: The largest gains occur for small calibration sets, such as nM < 50, where unconstrained MLE variance is highest.
  • Validity risks: When bounds contain the true parameters, Type-I error control is maintained asymptotically; incorrect bounds may inflate Type-I error.
  • Divergent decisions: A representative HSO case shows Noisy HT rejecting H0 and incorrectly certifying safety while Direct HT correctly accepts H0.

F.3 CASE 2: NOISY HT CORRECTLY REJECTS, DIRECT HT COMMITS TYPE-I ERROR

In the representative HSO case, the noisy test correctly identifies an unsafe classifier while direct testing incorrectly certifies it as safe. The example illustrates how judge calibration can prevent a Type-I error.

  • Noisy HT fails to reject H0 and correctly identifies the classifier as unsafe, whereas Direct HT rejects H0 and incorrectly certifies it as safe.
  • The case uses Qwen2.5-0.5B-Instruct as classifier and LLaMA-3.1-8B-Instruct as judge on HSO, with RM = 0.41 and α = 0.3.
  • The calibration example reports estimated TPR = 1.000 and FPR = 0.000.
  • The experiments compare direct, noisy, oracle noisy, and prediction-powered hypothesis testing using human-labelled and judge-labelled datasets.
  • The evaluation uses toxicity and hate-speech datasets with multiple classifier–judge configurations and repeated trials to estimate Type-I and Type-II error probabilities.

G.4.1 SYNTHETIC SETTING

Synthetic and cross-dataset experiments show that noisy hypothesis testing controls error rates and outperforms direct testing only when judge reliability is sufficiently high. The reported trends remain consistent across alternative classifier–judge pairs and SafeRLHF.

  • Figure 11 compares Type-I and Type-II errors against LLM failure rate and maps outperforming regions on the TPR–FPR plane.
  • Noisy hypothesis testing controls Type-I error, while its Type-II error depends on judge reliability.
  • Noisy HT outperforms direct HT only in the high-TPR/low-FPR regime, whereas oracle testing outperforms the baselines.
  • As RM increases, the boundary curves shift downward, enlarging the region where noisy testing outperforms direct testing.
  • Alternative classifier–judge pairs on Jigsaw and Hate Speech, plus experiments on SafeRLHF, show trends consistent with the main results.

H ANALYSIS OF HIGH FALSE POSITIVE RATE (FPR) SCENARIOS

Increasing FPR preserves finite-sample Type-I error control but reduces statistical power by shrinking the judge’s discriminative gap. At sufficiently high noise, noisy testing matches or underperforms direct testing.

  • Figure 15 fixes TPR = 0.95 and varies FPR across 0.05, 0.25, 0.50, and 0.75 to compare certification procedures.
  • Across FPR regimes, Noisy HT controls Type-I error below ζ = 0.05, including when FPR = 0.70.
  • When FPR = 0.05, the discriminative gap is TPR −FPR = 0.90 and Noisy HT has lower Type-II error than Direct HT.
  • As FPR increases, the discriminative gap shrinks to 0.45 and 0.25, shifting Noisy HT rightward and reducing power.
  • At FPR = 0.50, Noisy HT matches or underperforms Direct HT because judge noise outweighs the benefit of the larger nJ.
Loading 2601.20913v1…