Source-linked AI summary
Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation
Ancil Crayton
TL;DR
Overall scorer performance can mask errors that distort comparisons between prompts, and audit samples may not represent the study population. The paper derives how these errors affect contrasts and tests them with multiple audits. On identical cases, one comparison’s interval included zero, while two other comparisons kept their direction under both independent LLM raters.
Problem
Interactive LLM evaluations use prompt differences to assess pressure, persuasion, or robustness, but pooled scorer performance does not establish that errors preserve those comparisons.
Source: §1 · p. 1Method
Results
On identical cases, one automatic comparison shrank to an interval including zero, while two others kept their direction under both independent LLM raters.
Source: §6 · pp. 7–8Takeaways & Limitations
Prompt comparisons should be tested on matched cases, with evidence tied to the target population and uncertainty, rather than validated by pooled agreement alone.
Source: §6 · p. 8The labels are uncertain LLM ratings rather than human judgments or ground truth, and the representative target is limited to an equal-model mixture of clean nonreferents in six arms.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
Many LLM studies compare how models respond to different follow-up prompts. For example, one prompt may neutrally ask a model to reconsider, while another insists that an invented film really exists. The difference is used to measure the effect of conversational pressure. An automatic scorer then labels whether each response makes an unsupported factual claim. Researchers commonly validate the scorer with one overall accuracy or $F_1$ score. However, high overall performance can hide prompt-specific mistakes: if the scorer makes more errors after one prompt, the measured difference can be too large, too small, or point in the wrong direction. An audit containing mostly difficult responses can also misrepresent performance in the full study. We derive how such errors alter a prompt comparison and examine the problem using six models, 180 invented-entity questions, and six prompt conditions. In a balanced audit of 288 responses, overall $F_1$ was $.916$. Yet, depending on the prompt, the scorer flagged 2.6%-33.3% of responses that the primary LLM rater judged not to contain an unsupported claim; these prompt-specific estimates were uncertain. In a second audit, the scorer and two separate LLM raters evaluated the same 300 model-question cases under each prompt. An apparent 9.3-point increase from asserting certainty rather than neutrally asking the model to reconsider fell to 2.3 points under both raters. The primary-rater interval included zero, so it did not establish an increase. Two other prompt comparisons kept their direction under both raters, while an attempted prompt-specific correction remained too uncertain for a firm conclusion. Researchers should therefore validate the exact difference they plan to report, not just the scorer as a whole: audit both prompts on matching cases and include scorer and rater uncertainty in the final result.
1 Introduction
A scorer can appear reliable overall while prompt-specific errors distort the difference researchers use to evaluate pressure, persuasion, or robustness. The paper derives how scoring errors affect that comparison and examines the issue in a multi-model, multi-prompt case study.
1 Introduction
- A strong F1 can conceal condition-specific errors when a scorer handles hypothetical answers differently across prompts.The resulting measured difference combines model behavior with scoring error.Source: §1 · p. 1
- Interactive evaluations use prompt contrasts to support claims about pressure, persuasion, or robustness, so the difference—not average scorer accuracy—is the target of validation.A scorer can overstate, understate, or reverse the direction of the result.Source: §1 · p. 1
- Figure 1 schematically shows how unequal false positives across arms can leave pooled F1 strong while distorting the contrast.The figure distinguishes primary-rater labels from automatic-pipeline labels and illustrates unequal signed errors across the treatment and comparator arms.Source: §1 · p. 1, Figure 1 · p. 2
- The study examines six models, 180 constructed nonexistent-entity questions, and six prompt conditions, using stress, balanced, and paired audits.The audits include 550 stress-audit responses, 288 balanced-audit responses, and same-case comparisons against two independent LLM raters.Source: §1 · p. 2
- The paper derives arm-specific error effects, tests same-case comparisons, and proposes decisions based on construct, target population, audit sample, paired evidence, and uncertainty.The analyses are exploratory diagnostics, not a scorer leaderboard.Source: §1 · p. 2
2 How scoring errors distort a prompt comparison
Scoring error distorts a prompt comparison when the arms have different errors, even if pooled performance looks strong. Audit sampling and routing also affect observed performance, so validation must specify its target and test the comparison on matching cases.
2.1 Exact accounting of comparison bias
- False positives raise signed arm bias and false negatives lower it, so unequal arm biases add error to the reported contrast.Pooled F1 constrains a contrast only when it also implies similar arm biases for the target arms; even equal sensitivity and specificity can yield unequal bias when prevalence differs.Source: §2.1 · p. 2
2.2 Why the audit sample and fallback judge matter
- Oversampling predicted positives, difficult responses, selected arms, or routed cases changes audit prevalence and can alter errors within label groups.A hard-case audit cannot estimate target precision without a justified sampling model and weights; a fixed scorer version does not ensure transportability.Source: §2.2 · p. 3
- Hybrid scoring combines rule-scored and routed-judge cases, and either route-specific error or the routing share can change arm bias.Because harder cases enter the routed judge, route-specific behavior is descriptive and does not show the causal effect of using that judge.Source: §2.2 · p. 3
- Validation is tied to an outcome, prompt contrast, target population, sampling and weighting plan, and fixed automatic-pipeline and rater processes.The four gates assess whether arms share a construct, errors were observed by arm, the audit transports to the target, and same-key ratings preserve the contrast.Source: §2.2 · p. 3
- LLM judges can agree with independent ratings yet retain task- and model-dependent bias, while pooled benchmark agreement does not test whether signed error preserves a declared contrast.The paper also distinguishes factual support, abstention, and uncertainty as different outcomes [Min et al., 2023, Madhusudhan et al., 2025, Yona et al., 2024].Source: §2.2 · p. 3
- Sensitivity/specificity correction and prediction-powered inference can use LLM-rater labels, but neither repairs a changed construct or an audit that does not apply to target cases.The cited methods are Rogan and Gladen, 1978, and Angelopoulos et al., 2023.Source: §2.2 · p. 3
- A companion study shares the model–item panel but asks a different question and uses a different empirical analysis.This paper instead declares the target population and tests whether the pipeline preserves a difference between prompt conditions.Source: §2.2 · p. 3
3 Case study and audit design
The study evaluates six models on 180 constructed nonexistent-entity items across six prompt arms, using stress, representative, and paired audits. The balanced audit’s strong pooled F1 coexists with prompt-specific error variation, while sampling and construct differences limit what the comparisons establish.
3 Case study and audit design
- UUP-BENCH tests six models on 180 constructed nonexistent-entity items across six arms, with unsupported entity-specific assertions as the positive outcome.The 550-response stress audit deliberately included extra pressure-arm and routed-judge cases; the representative audit sampled 48 clean responses per arm and eight per model (n = 288).Source: §3 · pp. 3–4
- In a 96-row subset of the representative audit, the two LLM raters agreed on 94/96 = .979 (Cohen’s κ = .958); Spontaneous and Soft cue lacked secondary-rater labels.Neither rater is human, and neither supplies ground truth.Source: §3 · p. 4
3.1 Audit sampling changes the reported accuracy
- From hard to representative sampling, Neutral precision rose .667→.958, Authority .633→.966, and Spontaneous .684→.794, with wide, often overlapping intervals.These summaries use different sampling plans and do not establish a plan-by-arm interaction.Source: §3.1 · p. 4, Figure 2 · p. 4
3.2 Why errors must be measured by prompt condition
- Pooled F1 was .916, yet arm false-positive rates ranged from .026 to .333 in the balanced audit.The table reports arm-level accuracy for 48 rows per arm, with eight per model.Source: §3.2 · p. 5, Table 1 · p. 5
- The FPR analysis did not reject equal arm rates (χ2 5 = 9.10, p = .1052), and a 10-point difference near .20–.30 would require 292 negative rows per arm before clustering costs.The exploratory model uses 164 primary-rater-negative rows and clusters by item.Source: §3.2 · p. 5
- Unpaired bias diagnostics ranged from −.042 to +.146; the bias difference was +.083 for Confidence−Neutral and −.021 for Authority−Existence.These audit-slice diagnostics are not corrected panel effects; sensitivity was imprecise for Soft cue and Existence.Source: §3.2 · p. 5
- Existence permits hypothetical completion, changing the construct; therefore, the positive paired Authority−Existence contrast does not establish a neutral pressure contrast.The paired primary-rater estimate was +.150 (item-clustered 95% CI [+.061, +.279], n = 100 pairs), but H3 remains quarantined.Source: §3.2 · p. 5
- Claude Sonnet 5 reproduced the Neutral-over-Existence precision ordering, but its evidence was correlated and added no independent rater process or cross-family transport guarantee.It rescored the same 288 rows and is also an evaluated panel model.Source: §3.2 · p. 5
4 Does error change the comparison on the same cases?
On identical item–model keys, the raw Certainty-only−Neutral increase attenuates under both LLM raters, while two other contrasts retain their directions. An arm-aware correction moves one estimate toward the primary-rater value, but sparse calibration and rater uncertainty limit firm conclusions.
4 Does error change the comparison on the same cases?
- Certainty-only−Neutral attenuated from automatic +.093 to primary-rater +.023 ([−.059, +.053]); the secondary-rater estimate was also +.023.The primary-rater interval includes zero, so the raw headline is withheld.Source: §4 · p. 6, Table 2 · p. 6
- Confidence−Neutral remained positive (+.070 primary, +.107 secondary), while Confidence−Existence×Directness remained negative (−.167 primary, −.190 secondary).Neither contrast reversed sign; these decisions apply only to the labeled factorial contrasts.Source: §4 · p. 6
4.1 Correction does not remove calibration uncertainty
- Arm-aware correction changed automatic Confidence−Neutral RD from +.113 to +.032, near the primary-rater +.070, but its 95% interval was [−.146, +.197].The calculation carries uncertainty from target rates, sensitivity, and specificity; 48 calibration rows per arm do not support a precise corrected headline.Source: §4.1 · p. 6, Table 3 · p. 6
- The two LLM raters disagreed on 57/1,200 factorial labels (4.75%), with arm-specific disagreement from 1.7% to 6.7%.Their contrast estimates were close, but disagreement does not identify label error.Source: §4.1 · p. 6
- Under illustrative five-point per-arm prevalence-error budgets, only Confidence−Existence×Directness retained its sign; the budgets are breakpoints, not ground-truth error rates.Seven single-label changes among 600 arm labels could move Certainty-only−Neutral to zero.Source: §4.1 · p. 7
5 How to decide whether a comparison is supported
The protocol evaluates each intended prompt comparison through prespecified measurement, sampling, rater, and same-key checks. It assigns a disposition based on construct validity, audit applicability, and uncertainty, then requires revalidation when the record changes.
5 How to decide whether a comparison is supported
- Specify the positive label, unit, prompts, pairing key, target population and weights, pipeline and rater process, audit design, and meaningful contrast-bias tolerance before seeing effects.Prespecify whether success means preserving the sign or showing equivalence.Source: §5 · p. 7
- Use probability or explicitly balanced samples from both arms, preserve identical treatment–control keys, and justify any transport of hard-sample metrics to the target.Record inclusion probabilities or target weights; a separate hard sample can discover failures.Source: §5 · p. 7
- Blind independent LLM raters to hypotheses, expected directions, and automatic labels, and report counts, agreement, κ, and adjudication policy.Rater disagreement is outcome uncertainty, not a nuisance to omit.Source: §5 · p. 7
- Report TP/FP/FN/TN, intervals, prevalence, missingness, and routing shares by arm, including route-specific behavior for hybrid scorers.Pooled F1 cannot replace this disclosure.Source: §5 · p. 7
- Estimate automatic and reference contrasts and their difference on identical target keys and weights, using clustering that matches the design.An unpaired slice is only diagnostic; “ref” denotes the primary-rater contrast.Source: §5 · p. 7
- Apply construct, observation, transport, and same-key gates: retain, qualify, reject, or quarantine accordingly, and correct only after the first three gates with calibration uncertainty carried forward.Version and revalidate when prompts, labels, judges, routers, model panels, weights, sampling, or rater processes change; keep exploratory diagnostics distinct from prospective confirmation.Source: §5 · p. 7
6 Discussion and conclusion
On the same 300 keys, the factorial audit found that some prompt contrasts retained their direction across raters while another attenuated to an interval spanning zero. The authors recommend validating each defined comparison against its target population and matched-case evidence, with uncertainty guiding whether to retain, qualify, or quarantine it.
6 Discussion and conclusion
- Pooled F1 = .916 coexists with prompt-specific false-positive rates whose differences remain too imprecise to dismiss, and the prompt-specific correction remains too uncertain to replace the primary-rater estimate.The alternative automatic judge repeats one ordering but does not provide an independent check.Source: §6 · p. 7
- Validation should specify the outcome, two prompts, target cases, audit sample, and versioned automatic and LLM-rater processes.Error counts by prompt help explain a problem, while same-case ratings test whether the reported difference survives; prompts that measure different constructs or audits that do not apply to the target population cannot be rescued by those checks.Source: §6 · pp. 7–8
- In an equal-model mixture of six arms, observed FPRs span .026–.333, but the omnibus does not reject equal arm FPRs and modest variation remains unresolved.The hard audit finds failures under its documented enrichment and routing but does not estimate a population error rate.Source: §6 · p. 8
- On the same 300 keys, Certainty-only−Neutral attenuates to an interval spanning zero, while Confidence−Neutral stays positive under both raters and Confidence−Existence×Directness stays negative under the illustrative rater-error budget.The authors describe these as exploratory outcomes; arm correction cannot replace the fully rated estimate because calibration uncertainty is wide and sample dependence is only approximated.Source: §6 · p. 8
- For benchmark prevalence, estimate error in the benchmark population; for prompt differences, rate matched cases and test whether scoring error changes the contrast.Quarantine results when prompts measure different constructs, LLM ratings are unstable, or the audit does not apply to target cases.Source: §6 · p. 8
Limitations
The audits have limited precision and rely on uncertain LLM labels rather than human judgments or ground truth. Their target and correction assumptions also constrain how far findings can be generalized.
Limitations
- The representative audit has 48 responses per arm and eight per model, with wide precision, FPR, and especially sensitivity intervals.Its omnibus analysis is exploratory, treats six models as fixed, and lacks precision for modest arm differences; factorial intervals are also exploratory.Source: Limitations · p. 8
- Audit labels are uncertain LLM ratings, not human judgments or ground truth, and the representative target is only an equal-model mixture of clean nonreferents in six arms.That target does not represent provider traffic, future models, or arbitrary prompts; disagreements are not adjudicated.Source: Limitations · p. 8
- The alternative automatic judge is one correlated frontier model, and Rogan–Gladen correction assumes arm calibration transports to the target while handling overlap only approximately.Its interval rules it out as a replacement headline; the protocol gives no universal ϵ, and this retrospective study cannot demonstrate prospective preregistration.Source: Limitations · p. 8
Data and code availability
Reproduction materials are available online.
Data and code availability
- The reproduction package includes data, code, prompts, metadata, and seeds.The authors state that no proprietary scorer is claimed.Source: Data and code availability · p. 8
Ethical considerations
The study uses generated nonreferents and hosted LLM ratings, without private conversations or human participants.
Ethical considerations
- UUP-BENCH uses generated nonreferents and collision-screened names, with no private conversations or human subjects.Hosted LLM ratings are neither human labels nor ground truth.Source: Ethical considerations · pp. 8–9
- Hosted-call costs were not logged exhaustively, and endpoints are not redistributed.Source: Ethical considerations · p. 9
A Bias and routing derivations
The derivation shows why pooled agreement does not validate arm contrasts: signed error depends on each arm’s prevalence and error directions, while routing can shift pooled hybrid scores.
A Bias and routing derivations
- Equal F1 does not ensure equal signed arm bias: false positives raise bias and false negatives lower it, with prevalence affecting both.A pooled hybrid metric can also change with routing shares even when behavior within each route is fixed.Source: A Bias and routing derivations · p. 11
B Sampling and sensitivity diagnostics
- The 550 hard-audit labels are not inverse-weighted because tiering by pressure and routed-judge status targeted failure discovery, not population estimation.The displayed six arms account for 478 labels; 72 control-layer rows are omitted.Source: B Sampling and sensitivity diagnostics · p. 11
C Alternative automatic judge details
- On the same 288 representative rows, Claude's alternative judge gives Existence precision .842 and Neutral precision 1.000, but this evidence is correlated.Claude shares the rows, framing, and primary-rater labels, and scores some Claude outputs.Source: Table 6 · p. 12
D LLM-rater sensitivity details
- The primary-rater Certainty-only−Neutral RD is erased by seven flips, while two raters' RDs differ by at most 3.7 points across three contrasts.Neither rater supplies ground truth.Source: Table 7 · p. 12, Table 8 · p. 12
E Routing details
- Routed cases had F1 = .545 versus .762 for heuristic-only cases, but harder responses were more likely routed, so the difference is descriptive, not causal.A pooled hybrid metric can change with routing shares even when behavior within each route is fixed.Source: A Bias and routing derivations · p. 11, Table 9 · p. 13
F Existence error taxonomy
- All three Existence false positives were hypothetical abstentions or context requests for the pipeline, versus hypothetical or speculative completions for the alternative judge.Categories were single-coded post hoc.Source: Table 10 · p. 13, F Existence error taxonomy · p. 13
G Secondary transfer cases
Transfer audits support only bounded decisions for two non-original scorers, while the contrast card specifies evidence to archive for future headline claims.
G Secondary transfer cases
- Gaslight's heuristic has F1 = .512 on 300 primary-rater judgments and is quarantined; without same-key intervals, the audit cannot reject a particular contrast.The dual-rater κ = .573 does not repair its disagreement with the primary LLM rater.Source: G Secondary transfer cases · p. 13, Table 11 · p. 14
- On 60 SYCON trajectories, primary-rater everaccept was Existence 60/60, pressure 55/60, and Neutral 49/60; only a descriptive ordering is retained.The intervals are marginal rather than paired, and the analysis claims no reversal of the published conclusion [Hong et al., 2025].Source: G Secondary transfer cases · p. 13, Table 11 · p. 14
- Authority−Existence is quarantined because its prompt permits hypothetical completion; reviving the claim requires a construct-compatible comparator and/or target-matched audit, not pooled F1.The SYCON reversal claim is also withheld because its judge is non-original and Neutral recall is .355.Source: Table 12 · p. 14
H Suggested contrast card
- The contrast card archives the outcome and rubric, prompts and pairing key, target population, scorer and routing versions, sampling, error counts, paired effects, rater evidence, and decision boundaries.It also records prohibited extrapolations.Source: H Suggested contrast card · p. 14