PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

Ian Arawjo

arXiv:2609.35815v1cs.CLcs.AIcs.HCstat.ME

TL;DR

Small-sample AI evaluations and studies using LLM judges lack well-calibrated statistical guidance, complicating inference from their results. The paper develops simulation-based recommendations and PPI-corrected tests, packages them in evalstats, and reports that high judge agreement can still accompany false-positive risk.

  • Problem

    Small-sample AI evaluations and LLM-judge studies lack well-calibrated recommendations for confidence intervals, hypothesis tests, and judge-bias correction.

    Source: §1 · p. 2

  • Method

    The authors use Monte Carlo simulations to recommend small-sample methods and implement PPI-corrected hypothesis tests for mixed human-AI judge designs in evalstats.

    Source: §1 · p. 2, §6.1.2 · p. 8, §9 · p. 15

  • Results

    For numeric metrics, false-positive risk peaks around “almost perfect” agreement, and alignment ρ² should be at least 0.4 to be worthwhile, while below 0.2 the judge is too poor to use.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Researchers analyzing mixed human-AI judge studies should correct for judge bias rather than run statistical tests directly over raw judge scores.

    Source: §1 · p. 2

    The work focuses on means, leaves pairwise confidence intervals for unpaired data and validation of multi-run methods unreported, and shrinkage can cost strongly aligned judges a few points of power.

    Source: §9.6 · p. 16

Abstract

from arXiv · show

Researchers across academia increasingly base significance claims on LLM judge scores and small-sample AI evaluations. Yet without well-calibrated confidence intervals (CIs), hypothesis tests, and judge-bias corrections, such claims are unreliable. We address these issues in several contributions. First, we find that running statistics over raw LLM judge scores leads to inflated false positives: counterintuitively, for many inter-rater agreement metrics, false positive risk peaks at "almost perfect" human-LLM agreement. To help researchers understand how to run statistics over LLM judges responsibly, we present guidance and tooling for the statistical analysis of mixed human-AI judge designs, and implement nine hypothesis tests via prediction-powered inference (PPI), including the first known PPI corrections for four rank-based tests (Wilcoxon signed-rank, Mann-Whitney U, and omnibus variants). To keep PPI++ stable with small human-labeled calibration sets, we introduce bootstrap-adaptive power tuning, which shrinks the estimated weight toward a target estimated from the labeled data, and accounts for that weight's own sampling variance. Second, through Monte Carlo simulations, we derive recommendations for what CI, p-value, and FWER correction methods to use for small-sample AI evaluations (N<100), and warn researchers against bootstrap CIs. We package these recommendations into evalstats, an open-source Python package that selects calibrated methods automatically, and demonstrate it in three scenarios, including one where a real LLM judge validated at "substantial agreement" would have led a researcher to publish a spurious finding. evalstats is publicly available at https://github.com/ianarawjo/evalstats.

1 Introduction

The paper addresses weak statistical guidance for small AI evaluations and LLM-judge studies, where practitioners lack reliable methods for uncertainty and inference. It introduces calibrated methods and evalstats, and shows that high human–LLM agreement does not make raw-score testing safe.

1 Introduction

  • Only 16% of over 400 published benchmark papers report any statistics, while small-evaluation guidance often leaves method selection to practitioners.Existing recommendations often target large benchmarks, leaving small-sample and LLM-judge settings under-addressed.Source: §1 · p. 2
  • For numeric metrics, false-positive risk peaks at high inter-rater reliability, while Cohen’s κ on binary scores does not eliminate that risk.A real LLM judge validated at “substantial agreement” can still produce a very significant but entirely false finding.Source: §1 · p. 2
  • The authors implement and validate PPI-corrected versions of nine common hypothesis tests for mixed human-AI judge designs.These include the first known PPI corrections for Wilcoxon signed-rank, Mann–Whitney U, and omnibus rank-based tests.Source: §1 · p. 2
  • Monte Carlo simulations provide recommendations for small-sample confidence intervals, p-values, and family-wise error corrections.The paper warns that commonly chosen methods, including bootstrap confidence intervals, can be miscalibrated in these settings.Source: §1 · p. 2
  • evalstats automatically selects statistical methods and demonstrates its utility in three scenarios, including preventing a spurious LLM-judge finding from entering peer review.The authors validate the compare() workflow end to end because methods that work in isolation can fail in composition.Source: §1 · p. 2

2 Background and Related Work

Prior evaluation tools focus mainly on running or comparing evaluations, while statistical analysis and judge calibration remain limited. This paper builds on mixed human-AI inference and statistical-methodology work to address those gaps.

2 Background and Related Work

  • Many LLM evaluation tools support comparison or rubric alignment, but provide no validated inference beyond point estimates or uncertainty quantification.The passage notes that some tools display intervals without describing how they are computed.Source: §2 · pp. 2–3
  • LLM judges are often validated on a smaller human-scored sample and then treated as human surrogates, although their systematic biases complicate that assumption.Reported biases include preferences for longer or confident responses, position, or the judge itself.Source: §2 · p. 3
  • Prediction-powered inference estimates judge bias from a small human-labeled sample and uses that estimate to correct downstream inference.The reference implementation lacks two-sample, paired, rank-based, and omnibus tests, multiple-comparison correction, and small-sample power-weight calibration.Source: §2 · p. 3
  • Existing statistical toolkits generally target complex experiments at scale, while general-purpose libraries leave method choice to analysts.AI evaluations bring distinct needs, including judge-score calibration and small samples that can be difficult to fit with GLMMs.Source: §2 · p. 3
  • Prior HCI methodology work introduced nonparametric tools and statistical guidance; this paper’s use of PPI-corrected rank tests reflects their common use in experiment analysis.evalstats also adopts gradient plots, which prior work found produce better-calibrated inferential judgments than bar charts with error bars.Source: §2 · p. 3

3 Gaps, Scope, and Commitments

The paper identifies gaps in small-sample inference, judge alignment, stability testing, and uncertainty visualization, then states its scope and methodological commitments. It focuses on downstream analysis of collected evaluation results and uses simulation-informed defaults within a frequentist, estimation-oriented framework.

3 Gaps, Scope, and Commitments

  • The paper identifies limited guidance for small-sample eval reporting, carrying judge alignment into inference, stability across runs or variations, and uncertainty visualization.The examples include choosing among models and prompts, comparing agents using judge scores, and assessing repeated evaluation runs.Source: §3 · p. 4
  • Judge-bias correction estimated from a small human-labeled subset inherits the small-sample behavior of its underlying confidence-interval and p-value methods.The authors note that these regimes are coupled because LLM judges are widely used in evaluation.Source: §3 · p. 4
  • The work covers downstream analysis of spreadsheet-form evaluation results, not eval-set creation, construct validity, prompt optimization, or judge creation.The judges considered score artifacts produced by people or models; LLMs are not treated as stand-ins for human participants.Source: §3 · p. 4

3.1 Philosophy and Approach

  • evalstats foregrounds estimation with confidence intervals and effect sizes, while retaining p-values and tests such as Wilcoxon for users and venues that expect them.Defaults follow simulation evidence and a frequentist framing; intervals drawing on Bayesian methods are technically credible intervals.Source: §3.1 · p. 4
  • The design favors slightly conservative coverage, confidence intervals over standard errors, pairwise differences, and nonparametric methods for commonly bounded or non-normal evaluation scores.It sets a minimum sample floor, reports no statistics below N=15, and allows users to override defaults or opt out of p-values and NHST output.Source: §3.1 · p. 4

4 The evalstats Toolkit

evalstats organizes AI evaluation analysis around compare(), which selects methods and reports uncertainty for conditions and pairwise comparisons. It also supports judge-bias correction and additional workflows, with stated validation and scope boundaries.

4 The evalstats Toolkit

  • compare() takes long-format evaluation results and returns uncertainty estimates, including means with 95% confidence intervals and pairwise differences with simultaneous intervals and corrected p-values.Its output can include effect sizes, rank bands, and a plain-language summary; users can override method choices or suppress p-values.Source: §4 · pp. 4–5
  • Table 1 contrasts work developers would otherwise perform with evalstats calls; its judge-bias correction features are described as not straightforward to achieve without this contribution.The table is presented as a summary of features for statistical analysis of AI evaluation results.Source: §4 · p. 5, Table 1 · p. 5
  • The package also provides multi-run confidence intervals and uncertainty-aware joint metric analysis, but validation of its multi-run methods is deferred.Current limitations include unvalidated between-subjects pairwise intervals, a focus on means, and restrictions on combining PPI with multi-run data.Source: §4 · p. 5
  • With alignment=, compare() treats LLM-judge metrics as untrusted and corrects means, confidence intervals, pairwise differences, and p-values using a human-labeled calibration set.judge_alignment() reports agreement metrics and checks whether the labeled subset appears random and representative.Source: §4 · p. 5
  • Figure 2 presents decision trees for selecting confidence-interval and p-value or FWER methods for paired bounded-score evaluations with N=15–100.The recommendations are selected automatically by compare() and are backed by simulations.Source: §4 · p. 5, Figure 2 · p. 6

5 What are the best-performing CI, p-value, and FWER-correction methods for small-sample AI evaluation regimes (𝑁<100)?

Monte Carlo simulations compare uncertainty and multiple-testing methods for small-sample AI evaluations. The results identify recommended methods and show that bootstrap confidence intervals are often overconfident.

Simulation design

  • Simulations assess binary, continuous [0,1], and Likert-like scores using synthetic distributions and subsets of benchmark outputs, measuring calibration, efficiency, worst-case performance, and runtime.CI simulations include adversarial distributions such as zero-inflated results; the study also reports interval score and worst-case coverage.Source: §5 · p. 6

Methods compared

  • The tested candidates span recommended intervals, hypothesis tests, multiple-comparison corrections, and several bootstrap CI variants.Sources include prior recommendations and methods used in NLP, HCI, economics, and bioinformatics.Source: §5 · p. 6

Recommendations

  • The recommendations are summarized as decision trees for confidence intervals and p-value/FWER correction methods.Figure 3 compares recommended methods with runners-up and other tested methods across calibration, efficiency, and worst-case performance.Source: §5 · p. 6, Figure 3 · p. 7

Results

  • Romano-Wolf step-down performs well for FWER correction at n≥30; for paired binary CIs, Bonett and Price outperforms Bowyer et al. on worst-case coverage and Type-I error while running faster.Bootstrap CI methods were overconfident nearly everywhere; on paired binary data, none reached nominal coverage even at N=100.Source: §5 · p. 6

6 How to run statistics over LLM judge scores? Mixed human-AI versions of standard hypothesis tests

The section examines whether high human–LLM agreement makes judge scores safe for statistical testing and develops PPI-corrected tests and guidance for mixed human-AI designs. Its simulations show that raw judge scores can inflate false positives even at high agreement, while corrected tests can retain power over human-only subsets.

6.1.1 Monte Carlo simulations of PPI-corrected statistical tests.

  • The first known PPI corrections for four rank-based tests use subsampling-stable estimands; bootstrap-adaptive power tuning helps keep PPI++ stable with small calibration sets.The rank-test corrections reformulate Mann–Whitney, Wilcoxon, and Kruskal–Wallis estimands, while Friedman uses within-subject rank means.Source: §6.1.1 · p. 8
  • PPI-corrected tests showed good calibration, with expected underperformance when the required MCAR assumption was violated; paired tests appeared most robust to mild deviations.The simulations validated most tests for numeric data; for binary data, only Welch’s and paired t-tests were validated.Source: §6.1.1 · p. 8

6.1.2 Summary of simulation sweep (§B.2.1) for LLM judge bias sce-

  • The simulations crossed data, labeling, and judge factors to test for failures that emerge only in combination, including documented judge-bias modes.The factorial sweep complements one-factor-at-a-time experiments.Source: §6.1.2 · p. 8

6.2 Does high inter-rater reliability certify judge trustworthiness?

  • For Likert data, false-positive risk was worst around 0.80–0.90 agreement and remained high through 0.90; continuous-data risk rose with Pearson’s r, while binary Cohen’s κ reduced but did not eliminate it.For Likert data, risk returned to nominal only above 0.99 agreement.Source: §6.2 · p. 9

6.3 How much power do LLM judges buy you? Quantifying effective human labels

  • The effective-label curve compares a mixed design’s power with the human-only sample size needed to match it, with the y=x diagonal marking no judge benefit.The comparison pools rejection rates across specified tests and data types.Source: §6.3 · p. 9
  • At ρ²=0.4, the estimated efficiency multiplier’s lower bound was 1.36× and its average was 1.57×; the authors recommend targeting at least this alignment.At ρ²≈0.2, the worst measured average multiplier was 1.06×, so meaningful gains were not assured.Source: §6.3 · pp. 9–10

6.4 Guidance for running mixed human-AI judge designs and analyzing results with PPI-corrected hypothesis tests

  • Guidance recommends at least 30 human labels per condition (15 as a floor), random sampling, and PPI correction regardless of apparent IRR.Researchers should target test-appropriate ρ²≥0.4; below approximately 0.2, corrected tests may not outperform the human-only subset.Source: §6.4 · pp. 10–11

7 End-to-end validation of compare()

The authors tested evalstats’ compare() with method selection set to auto across simulated data types, sample sizes, condition counts, and labeling budgets. Coverage and Type-I error stayed near nominal, while FWER-corrected PPI tests retained power gains over tests on the human-labeled subset.

7 End-to-end validation of compare()

  • Across 717,600 calls, estimates were near-nominal across data types and from per-arm through FWER-corrected simultaneous CIs, while p-value Type-I error remained controlled.The sweep varied N=15 to 1000, conditions k=2 to 10, and labeling budget.Source: §7 · p. 12
  • With FWER correction at k=3, PPI tests recovered up to ≈14 percentage points of power, averaging ≈9 points over tests on the human-labeled subset alone.Source: §7 · p. 12
  • The end-to-end sweep used evalstats’ compare() with method choices set to auto across binary, continuous, and Likert data.Source: §7 · p. 12

8 Demonstrations

Three demonstrations show how evalstats handles prompt selection, LLM-judge bias, and mixed human-AI evaluation. They illustrate calibrated uncertainty and judge correction in practical analysis, including a case where correction removes a spurious finding.

8 Demonstrations

  • The demonstrations follow practical evaluation workflows, from choosing dialogue prompts to comparing feedback styles and analyzing mixed human-AI judge data for publication.The reported compare() results use real data or, where noted, synthetic data.Source: §8 · p. 12

8.1 Choosing a model and prompt for the dialogue component

  • 43%: Eyeballing the 20-item prompt-evaluation bar chart would select a sub-optimal prompt in this example.The result comes from 5,000 draws of 20 items from the full 120-ticket set.Source: §8.1 · p. 13
  • 90%: Across 28 pairwise comparisons, at least one bootstrap interval missed its true mean difference about this often, even with Šidák correction.The percentile bootstrap intervals covered the true difference on 88% of draws; BCa covered it on 82%.Source: §8.1 · p. 13
  • 97%: With 80 tickets, evalstats would rule out the worst prompt, while across 1,500 draws it never discarded a truly winning prompt.At 20 tickets, it rules out the worst prompt on only 3% of draws; the intervals are conservative but well-calibrated for this binary-data setting.Source: §8.1 · p. 13

8.2 What if the metric is an LLM judge?

  • 1.43 points (95% CI [0.53, 2.32]): Correcting for the judge’s condition-dependent length bias reversed the apparent tie and indicated that plain-writing mode may help.The example uses synthetic evaluation data with a simulated response-length confound.Source: §8.2 · pp. 13–14

8.3 Preventing a spurious finding from reaching publication: Analysis of mixed human-AI judge designs

  • p=1.9 × 10^-4: Raw judge scores produced a significant Kruskal–Wallis result despite no real condition difference (p=0.47); agreement was κ=0.66.Three of six Holm-corrected pairwise comparisons were significant, illustrating how reassuring agreement and conventional testing still led to a spurious finding.Source: §8.3 · p. 14
  • p=0.31: PPI-corrected Kruskal–Wallis found no significant difference across four conditions, and no post-hoc comparison survived correction.All four conditions formed one statistically indistinguishable band, matching the ground truth in the example.Source: §8.3 · p. 15
  • 3.7%: Across 300 resamples, the corrected analysis produced a false finding only when both omnibus and pairwise tests were significant; the human-only baseline also yielded 3.7%.The corpus was close to, but not exactly at, the null, so these are rates of findings surviving relabeling rather than exact Type-I error rates.Source: §8.3 · p. 15

9 Discussion

The discussion presents guidance and validated tests for mixed human–AI judge designs, alongside recommendations and tooling for small-sample evaluations. It also identifies assumptions, application boundaries, and open limitations.

9 Discussion

  • The paper addresses statistical-know-how gaps with guidance and validated tests for mixed human–AI judge designs, plus recommendations and tooling for small-sample evaluation.These are presented as related but distinct regimes.Source: §9 · p. 15

9.1 Implications for researchers

  • Corrected inference is presented as an expected practice for LLM judges; the authors caution against bootstrap CIs and evalstats refuses statistics below N=15.The authors also note that corrected methods can use alternatives to PPI, including design-based supervised learning.Source: §9.1 · pp. 15–16

9.2 Making uncertainty-aware interfaces for AI evaluation possible

  • Existing AI-evaluation interfaces lack statistical inference, while evalstats could add calibrated comparisons and gradient plots to tools such as MetricMate, ChainForge, and DocWrangler.The authors identify a graphical interface for planning, running, analyzing, and reporting mixed human–AI judge designs as a next step.Source: §9.2 · p. 16

9.3 Responsible setup and statistical analysis of mixed human-AI judge designs

  • PPI requires trustworthy human labels; its guarantees require random, independent label selection, and alignment ρ2≈0.4 adds about 17 effective human samples from 30 labels and 300 judge scores.The authors caution that PPI cannot create agreement where none exists, and that mixed designs may require a highly aligned judge or a large dataset.Source: §9.3 · p. 16

9.4 PPI tests outside of LLM judge scenarios

  • PPI can adjust inference for noisy predictors such as crowdworkers, but synthetic-user scores gain nothing without unit-level covariation between judge and human scores.Applying PPI to synthetic subjects conditioned on real participant profiles or traces needs separate validation.Source: §9.4 · p. 16

9.5 Why not Bayesian?

  • The paper uses a frequentist framing despite identifying potential Bayesian benefits, including prior information and posterior probability statements.The authors cite prior complexity, conservative behavior of some tested Bayesian intervals, computational demands, and audience expectations as difficulties.Source: §9.5 · p. 16

9.6 Limitations

  • Limitations include no unpaired pairwise-CI recommendations, no reported validation of multi-run CIs, a focus on means, and no user study.The authors also identify possible extensions to additional metrics and tests, and note continuing debate over CI defaults and p-value reporting.Source: §9.6 · pp. 16–17

10 AI Usage Acknowledgment

The appendix describes AI assistance in the work and outlines the data and simulation resources used to evaluate statistical methods. Its real-data corpora draw on OpenEval and manually run Inspect AI benchmarks.

10 AI Usage Acknowledgment

  • The PPI-corrected methods and rank-test appendix received AI assistance, while the author takes responsibility for errors.The acknowledgment distinguishes AI contributions to methods, plots, code, and drafting from the author's other writing.Source: §10 · p. 17

A Monte Carlo simulations comparing statistical methods in LLM evals-like data settings

  • Simulation tests are available through the repository harness, run in parallel, and may take many hours; appendix tables color-code under-coverage, conservative results, and near-nominal results.The reported runs used a 2024 MacBook Pro with 64 GB RAM and an M4 Max chip.Source: A Monte Carlo simulations comparing… · p. 20

A.1.1 Synthetic data generation.

  • The synthetic suite sweeps distribution shapes, reliability, and effect sizes, with effect sizes scaled by each shape's standard deviation at the specified ICC.The same suite supports CI, p-value, FWER, and end-to-end compare() simulations; PPI scenarios use a separate judge-bias generator.Source: §A.1.1 · p. 20, Table 2 · p. 21

A.1.2 Real evaluation data sourcing.

  • Real-data simulations sample from larger benchmarks and use their mean estimates as ground truth, drawing on OpenEval and manually run Inspect AI corpora.The corpora comprise 15 OpenEval model–benchmark pairs and six models across nine Inspect AI benchmarks; real-data results cover binary and continuous data.Source: §A.1.2 · pp. 20–21

A.2 Which confidence interval (CI) methods perform optimally in small-samples eval settings?

The section recommends confidence-interval methods for mean estimates and pairwise differences, then reports preferred pairwise tests by data type. The recommendations reflect coverage and interval-score comparisons across small-sample settings.

A.2 Which confidence interval (CI) methods perform optimally in small-samples eval settings?

  • The CI comparisons cover 95% intervals for mean estimates and mean pairwise differences, using coverage, interval width, and interval-score measures.The authors caution that interval scores can obscure poor coverage tails, so they also consider minimum coverage and penalty.Source: §A.2 · p. 22
  • Binary mean CIs: Wilson; numeric mean CIs: logit-transformed t; binary pairwise CIs: Bonett-Price; continuous pairwise CIs: logit-transformed t; Likert pairwise CIs: Bayesian NIG.The logit-transformed t method uses a Clopper-Pearson fallback in zero-width cases; Bonett-Price never falls below .90 coverage in any tested cell.Source: §A.2.1 · p. 22, §A.2.2 · p. 22, Table 4 · p. 23

A.3 Which p-value methods for pairwise comparisons perform optimally in small-samples eval settings?

  • For numeric data, the section recommends Wilcoxon signed-ranks; for binary data, it recommends McNemar’s mid-p.Wilcoxon was slightly more conservative at low n and made no parametric assumptions; for binary data, the choice also avoided frequent disagreement with the Bonett-Price CI.Source: §A.3 · p. 22

A.4 Which FWER correction methods perform optimally for CIs and p-values?

This section recommends family-wise error correction methods for small-sample evaluations, introduces bootstrap-adaptive power tuning for PPI++, and describes the synthetic and real judge data used to evaluate the methods.

A.4 Which FWER correction methods perform optimally for CIs and p-values?

  • 0.95 average coverage was maintained by Šidák for simultaneous-CI FWER correction at N<100, making it the recommended small-sample method.Its worst cell was 0.897; the joint bootstrap had lower worst-case coverage and collapsed on sparse or lopsided binary data.Source: §A.4 · p. 23
  • For p-value FWER, use Romano–Wolf at n≥30; use Shaffer below n=30, when pairwise p-values are required, or for extremely lopsided binary data.In the last case, the stated trigger is fewer than 5 observations of the rarer outcome in any compared group.Source: §A.4 · p. 24

B.1 Bootstrap-adaptive power tuning

  • Bootstrap-adaptive power tuning shrinks the estimated PPI++ weight toward a target selected by labeled-data resampling and adds the weight’s sampling variance to intervals and p-values.The target is the share of 800 resamples in which the judge earns at least half the weight; the shrinkage weight is Nlab/(Nlab+20).Source: §B.1 · pp. 26–27

B.2.1 Synthetic judge-bias generation.

  • Synthetic judge-bias simulations varied individual and combined bias mechanisms, noise, effect size, sample size, label count, and labeling mechanism across evaluation data types.Factorial magnitudes were standardized by each evaluation type’s population standard deviation; pathological truth shapes received a smaller set of dedicated scenarios.Source: §B.2.1 · p. 27

B.2.2 Real LLM judge data sourcing.

  • Real-world validation used five human-label/judge-score corpora spanning binary, continuous, and Likert-style data, with randomly revealed human labels for PPI.Four corpora had multiple judges; WMT DA also supplied a paired variant, while ICLR Metareview had a single judge.Source: §B.2.2 · p. 27

B.3 Type I error rates of bias-corrected tests versus running statistics over uncorrected judge scores

PPI correction controlled false positives across MCAR synthetic and real-judge checks, while MNAR labeling exposed less reliable tests; the section also reports alignment, CI, and omnibus-power findings.

B.3 Type I error rates of bias-corrected tests versus running statistics over uncorrected judge scores

  • On four real-judge corpora, PPI-corrected tests retained calibrated Type-I error control, whereas tests without PPI showed spiking false-positive rates; under synthetic MCAR labeling, uncorrected statistics also spiked false positives.Source: §B.3 · pp. 27–28

B.4 Type I error by judge-human alignment, binary and continuous data

  • PPI correction remained well-calibrated across judge-human alignment levels for binary, continuous, and Likert data, including settings where common inter-rater metrics miss bias.At IRR near zero, the passage notes that no discernible human–judge correlation means noise dominates any effect to detect.Source: §B.4 · pp. 28–29

B.3 Type I error rates of bias-corrected tests versus running statistics over uncorrected judge scores

  • Under MNAR labeling, paired t, Wilcoxon, and RM-ANOVA stayed near nominal on average, while Friedman and independent-groups tests inflated, especially under strong MNAR.The figure summarizes 4,092 null scenarios across nine tests and identifies Friedman, Mann–Whitney U, and Kruskal–Wallis as least reliable when MCAR is violated.Source: §B.3 · p. 28, Figure 10 · p. 29

B.5 PPI correction of main-path CI methods

  • PPI-corrected marginal and pairwise confidence intervals had coverage near or above nominal across synthetic and real judge data.The reported main-path methods include Wilson and Bonett-Price for binary data and logit-t intervals for bounded numeric data; NIG is not corrected.Source: §B.5 · p. 29

B.6 Do PPI omnibus tests preserve the ability to detect real differences?

  • PPI power for the four omnibus tests tracked close behind the all-human oracle and exceeded both the human-only subset and uncorrected LLM baselines across panels.The comparison covers continuous and Likert data, varying effect size at a fixed labeled-item budget and varying labeled count at Cohen’s d=0.3.Source: §B.6 · p. 29

C Extending PPI correction to rank-based nonparametric tests

The paper extends prediction-powered inference to rank-based tests by correcting subsample-defined estimands, then evaluates calibration under stated sampling and null assumptions.

C Extending PPI correction to rank-based nonparametric tests

  • Four rank-based PPI tests correct population estimands defined on subsamples rather than applying the average-based itemwise correction to literal rank statistics.The tests assume i.i.d. items and a uniformly sampled labeled subset; several use a sharper null than the weak null, so simulations assess finite-sample calibration.Source: C Extending PPI correction to… · pp. 29–30

C.1 PPI-corrected Friedman test

  • Friedman needs no rank-statistic reformulation because each subject’s condition ranks depend only on that subject’s row, so mean ranks remain usable on labeled subsets.The corrected means are no longer literal sample mean ranks, so the test uses a modified Iman–Davenport form and relies on simulations for calibration.Source: §C.1 · pp. 30–31

C.2 PPI-corrected Mann-Whitney 𝑈test

  • Mann–Whitney PPI corrects the two-sample dominance estimand 2P(A=B)-1/2, defined on any subsample and equal to zero when the groups share a distribution, ties included.Its reported effect size is the corrected rank-biserial correlation.Source: §C.2 · p. 31

C.3 PPI-corrected Kruskal-Wallis test

  • Kruskal–Wallis PPI corrects pairwise dominance estimands underlying group mean ranks, but, like classical Kruskal–Wallis, has no power against cyclic dominance.At k=2, its estimand reduces to Mann–Whitney’s.Source: §C.3 · pp. 31–32

C.4 PPI-corrected Wilcoxon signed-rank test

  • Wilcoxon uses a Walsh-average dominance estimand whose zero-null is weaker than symmetry and whose mid-probability form retains information under zeros and ties.Under symmetry, the estimand’s null is the usual Wilcoxon null.Source: §C.4 · p. 32

C Extending PPI correction to rank-based nonparametric tests

  • Omnibus PPI power tracked close behind the all-human oracle and ahead of human-only and uncorrected LLM baselines across the reported panels.The omnibus comparison pools ANOVA-independent, ANOVA-repeated, Friedman, and Kruskal–Wallis tests.Source: Figure 13 · p. 33

D Label efficiency and robustness of the 𝜌2 ≥0.4 rule of thumb

The reported label-efficiency rule links judge-human agreement to expected savings, and the end-to-end validation describes how evalstats’ automatic routing was tested.

E End-to-end validation of compare()

  • End-to-end compare() validation swept 23 distribution shapes, three data types, multiple sample sizes and arm counts, and left method selection to method="auto".For FWER correction on the PPI path, the validation used Shaffer to keep the selected pairwise PPI tests consistent with those being validated.Source: E End-to-end validation of compare() · p. 34

D Label efficiency and robustness of the 𝜌2 ≥0.4 rule of thumb

  • 1.25× label savings were cleared by every data type at ρ²≈0.4, while below ρ²≈0.2 savings stayed under 1.25× for all data types.The authors recommend ρ²≈0.4 as the threshold to aim for; the multiplier was averaged over parametric and nonparametric tests.Source: D Label efficiency and robustness of… · p. 34, Figure 15 · p. 35

F Full-size plots of IRR metrics under differential judge bias, per data type

The figures sweep judge bias against judge noise across Likert, continuous, and binary data, comparing uncorrected false-positive rates across inter-rater-reliability metrics. PPI-corrected two-sample tests held near nominal error across the sweep.

F Full-size plots of IRR metrics under differential judge bias, per data type

  • 0.047 average Type I error across 5,420 cells for PPI-corrected two-sample tests.The sweep covers 5- and 7-point Likert, continuous, and binary data, with up to eleven judge variants per cell.Source: F Full-size plots of IRR metrics under… · p. 36
  • For 5-point Likert data, PPI-corrected rates hold at nominal levels in every cell.The figure plots uncorrected rates in red and PPI-corrected rates in blue across IRR metrics and judge-bias levels.Source: Figure 17 · p. 36
  • 0.92–0.95 false-positive-rate peaks appear for 7-point Likert data because half a scale point is a smaller fraction of the scale’s SD.The rounding threshold is half a scale point.Source: Figure 18 · p. 37
  • For continuous data, false-positive risk rises monotonically across IRR metrics.Source: Figure 19 · p. 38
  • For binary data, false-positive risk falls as agreement rises.Source: Figure 20 · p. 39

Related papers

Loading 2609.35815v1…