PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions

Wenqi Li, Bin Liu, Mindi Ruan, Chuanbo Hu, Minglei Yin, Xin Li

arXiv:2610.09229v1cs.LGcs.AI

TL;DR

LLM-judge performance is often summarized by aggregate accuracy, despite the need to compare judges across deployment-relevant conditions. This paper introduces CAP, a post-hoc framework for profiling pairwise accuracy across eight conditions and evaluating it on seven judges and six benchmarks. The profiles reveal condition-specific weaknesses and adversarial degradation patterns that aggregate accuracy alone does not capture.

  • Problem

    Aggregate judge accuracy does not distinguish different failure modes, and existing per-category results lack a shared taxonomy and explicit benchmark-coverage accounting for cross-benchmark comparison.

    Source: §1 · p. 1

  • Method

    CAP reports pairwise judge accuracy across eight content-sensitivity, robustness, and rationale-quality conditions, using direct benchmark evidence, task-subset proxies, or controlled augmentation as available.

    Source: §1 · p. 1

  • Results

    Across seven judges and six benchmarks, CAP reveals condition-specific weaknesses and rank differences, while JudgeBench-Pro adversarial stress degrades robustness and content channels with the dominant channel varying by judge.

    Source: §1 · p. 2, §4.3 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    Condition-level profiles complement aggregate accuracy as a basis for selecting LLM judges for an intended application.

    Source: §6 · p. 9

    Rationale Quality relies on an independent LLM evaluator whose error and potential bias are a concern, and validation used 120 rationales and one unblinded annotator.

    Source: §7 · p. 10

Abstract

from arXiv · show

LLM-as-judge is now a standard tool for scalable evaluation, but judge performance is still often summarized by a single accuracy number. This aggregate view hides the deployment conditions under which a judge succeeds or fails. We introduce \textbf{Conditional Accuracy Profiling} (CAP), a post-hoc diagnostic framework that decomposes pairwise LLM-judge accuracy into eight conditions organized into content sensitivity, robustness, and rationale quality. CAP is benchmark-agnostic: it can be applied directly when a benchmark provides the required annotations, approximately through task-subset proxies, or through controlled augmentation when perturbation pairs can be generated. We instantiate CAP on seven LLM judges across six pairwise judging benchmarks, including \textsc{judgerEva-Standard}, a controlled testbed we created to support all eight conditions. CAP exposes profile differences hidden by aggregate accuracy: on \textsc{judgerEva}'s judge-independent Hard-Constructed subset, the two judges most sensitive to omitted qualifications rank in the bottom three of seven by overall accuracy, so omission sensitivity is not predicted by aggregate accuracy. Across benchmarks, Position Robustness shows the strongest rank stability (mean Spearman $\bar{\rho}{=}0.87$) but is itself fragile under JudgeBench-Pro adversarial stress, showing the largest mean accuracy drop among the shared conditions, though the dominant degradation channel varies by judge. Condition-level profiles provide a more actionable basis than aggregate accuracy for selecting LLM judges.

1 Introduction

LLM-judge evaluation commonly relies on aggregate accuracy, although judges with similar scores may fail under different conditions. CAP supplies a shared taxonomy for condition-level profiles, and evaluation across seven judges and six benchmarks finds substantial condition-specific variation.

1 Introduction

  • A shared taxonomy and explicit benchmark-coverage accounting are missing from otherwise available per-category evaluations, limiting comparability across judges and benchmarks.Aggregate accuracy can obscure different weaknesses, including sensitivity to response order, verbosity, or omissions.Source: §1 · p. 1
  • CAP is a post-hoc framework that reports pairwise judge accuracy across eight conditions spanning content sensitivity, robustness, and rationale quality.Measurable conditions receive bootstrap confidence intervals, and benchmark coverage is labeled direct, approximate, augmented, or unavailable.Source: §1 · p. 1
  • Across seven judges and six benchmarks, CAP reveals narrow condition-specific weaknesses, uneven within-family differences, and adversarial degradation in robustness channels as well as substantive judgment.The evaluation includes JUDGEREVA-STANDARD, designed to support all eight conditions, and five external benchmarks.Source: §1 · p. 2
  • The paper contributes a coverage matrix, JUDGEREVA-STANDARD, empirical profiles, and a reference implementation for condition-aware judge selection.The benchmark is designed to measure all eight conditions on the same items.Source: §1 · p. 2

2 Related Work

Prior benchmarks and studies investigate particular judge abilities or failure dimensions, while CAP organizes condition-specific evidence under a shared taxonomy with explicit benchmark coverage. The framework also incorporates uncertainty estimates and a controlled benchmark alongside naturalistic resources.

LLM-as-judge benchmarks

  • Existing benchmarks test distinct settings, from objectively labeled knowledge and reasoning pairs to reward-model tasks and instruction-following, but generally cover only selected dimensions.RM-Bench uses controlled content edits and style bias, but its independently scored responses do not support ordering robustness or rationale conditions.Source: §2 · p. 2

Bias discovery and robustness testing

  • CAP distinguishes prior bias-discovery and reliability work by fixing a condition taxonomy and reporting judges and benchmarks in a common layout.That layout supports attributing an accuracy gap to a specific failure dimension rather than merely identifying a bias or stress-testing one judge.Source: §2 · p. 2

Statistical foundations

  • CAP reports bootstrap confidence intervals and uses JUDGEREVA-STANDARD as a controlled-condition complement to naturalistic benchmarks for diagnosing deviations on shared conditions.The released outputs can also support condition-by-condition bias-corrected variants.Source: §2 · p. 2

3 Method

The method defines CAP as condition-specific accuracy over benchmark-supported item subsets, then builds a controlled benchmark to measure all eight conditions and maps external benchmarks to their available evidence. Seven judges are evaluated with common prompting, controlled perturbations where valid, and bootstrap and cross-benchmark analyses.

3.1 The CAP Framework

  • CAP defines each condition through an item subset and correctness indicator, then reports judge accuracy on that subset when the benchmark supports it.The eight conditions separate content sensitivity, presentation robustness, and rationale quality; robustness requires both judgments to be correct, not merely consistent.Source: §3.1 · p. 3

3.2 JUDGEREVA-STANDARD Benchmark

  • JUDGEREVA-STANDARD constructs flawed answers with four planted error types, creates three presentation variants, and evaluates both answer orderings.Its 2,092 plain items yield 6,276 pairwise items; Hard-Empirical selects errors made by evaluated judges, while Hard-Constructed is built independently of their errors.Source: §3.2 · p. 4, Figure 2 · p. 4
  • JUDGEREVA-STANDARD anchors the full eight-condition profile, while five external benchmarks provide external validation and adversarial stress.Position Robustness is the only condition supported uniformly across prior benchmarks under the strict both-correct criterion.Source: §3.2 · p. 5
  • External benchmarks support different CAP conditions through direct annotations, task-subset proxies, or controlled augmentation, with coverage recorded by benchmark.Augmentation is admissible only when it preserves the original label; heterogeneous-source pairs can make style or length controls confounded.Source: §3.2 · p. 5, Table 2 · p. 5

3.3 Judges, Augmentation, and Estimators

  • Seven judges receive the same pairwise prompt, and the protocol uses both orderings for Position Robustness and only label-preserving controlled variants for style and length tests.Rationale Quality is evaluated on correct-verdict items by an independent evaluator; estimates use 1,000 bootstrap resamples, with rank correlations and paired JudgeBench–JudgeBench-Pro differences for cross-benchmark analyses.Source: §3.3 · pp. 5–6

4 Experimental Results

CAP profiles show condition-specific differences that aggregate accuracy can miss. Position Robustness ranks transfer most consistently across benchmarks, while JudgeBench-Pro stress degradation varies by condition and judge.

4 Experimental Results

  • The experiments address profile differences, cross-benchmark ranking consistency, adversarial stress degradation, and within-family model differences; approximate and augmented cells use distinct protocols from direct measurements.Approximate cells use native task subsets as proxies, while augmented cells use generated controlled perturbations.Source: §4 · p. 6

4.1 Per-Benchmark CAP Profiles

  • Claude Sonnet 4.5 leads overall accuracy on Hard-Empirical at 89.7% versus Haiku’s 80.5%, yet scores 59.2% on Scope-Omission, 11.3 points below Haiku.Gemini Flash and Pro also diverge on Scope-Overclaim (72.4 vs. 90.8) and Rationale Quality (44.4 vs. 83.3).Source: §4.1 · p. 6, Table 3 · p. 7, Figure 3 · p. 7
  • On Hard-Empirical, GPT-4o-mini’s conditional C8 of 75.0 corresponds to an unconditional supported-correct rate of 20.3%, while Sonnet 4.5’s 85.7 corresponds to 76.9%.C8 is best read as a lower bound on true rationale quality because evaluator false negatives concentrate on critical-omission cases.Source: §4.1 · p. 7
  • 56.7% for Llama-3.3-70B and 51.7% for GPT-4o-mini lead Scope-Omission on HARD-CONSTRUCTED, although they rank third-weakest and weakest overall.The negative overall-accuracy/Scope-Omission rank correlation is −0.33 across seven judges.Source: Table 4 · p. 8, §4.1 · p. 8
  • On JudgeBench’s measurable conditions, Gemini 2.5 Pro is strongest, reversing JUDGEREVA’s ordering where Claude Sonnet 4.5 led.JudgeBench emphasizes reasoning, math, and coding, whereas JUDGEREVA balances planted error types.Source: §4.1 · p. 8, Table 5 · p. 9

4.2 Cross-Benchmark Consistency

  • 0.87 is the mean pairwise Spearman correlation for Position Robustness rankings across benchmarks; JudgeBench and JudgeBench-Pro have identical rankings at ρ=1.00.The broad ordering of strong versus weak judges persists across benchmarks, while fine-grained ranks shift.Source: §4.2 · p. 8

4.3 Adversarial Stress Degradation

  • 30.7 points is the largest mean stress drop for C7, but C7 is the largest per-judge drop for only two of seven judges.The dominant degradation channel varies by judge, so similar aggregate brittleness can reflect different condition-level vulnerabilities.Source: §4.3 · p. 8, Table 6 · p. 9

5 Discussion

CAP profiles are not benchmark-independent judge traits: transfer differs by condition, and judge selection depends on the application’s failure modes. Same-family models can also differ enough that family membership alone is not a sufficient substitution heuristic.

5 Discussion

  • Position Robustness transfers relatively well, but content-sensitivity rankings depend on benchmark error construction or task subsets.The authors recommend reading profiles jointly with benchmark coverage and the intended application setting.Source: §5 · p. 9
  • Scientific review emphasizes factual sensitivity and supported rationales, long-form evaluation is exposed to length bias, and legal or policy review is sensitive to omitted qualifiers.The highest-overall judge can be suboptimal for any one of these application-specific conditions.Source: §5 · p. 9
  • GPT-4o-mini matches GPT-4o at 60.7 on the JudgeBench Knowledge proxy but trails by 10.8 points on Logical Sensitivity and 15.4 on Position Robustness.The authors suggest profiles may inform cost-aware within-family judge selection, while leaving routing-policy design to future work.Source: §5 · p. 9

6 Conclusion

CAP decomposes pairwise LLM-judge accuracy into eight deployment-relevant conditions and reveals weaknesses and rank reversals hidden by aggregate accuracy. The authors conclude that evaluation should be condition-aware and identify downstream selection experiments and routing policies as future work.

6 Conclusion

  • CAP profiles eight deployment-relevant conditions across content sensitivity, robustness, and rationale quality, complementing aggregate accuracy.The framework is intended to make judge evaluation condition-aware.Source: §6 · p. 9
  • Across seven judges and six benchmarks, condition-specific weaknesses and rank reversals show that the strongest overall judge is not always strongest on Scope-Omission.Gemini models are also reported as substantially more robust under JudgeBench-Pro stress than Claude.Source: §6 · p. 9
  • Position Robustness has the most consistent measurement and the largest average adversarial-degradation channel, although the dominant channel varies by judge.Future work includes testing CAP-guided judge selection on real tasks and developing operational routing policies.Source: §6 · p. 9

7 Limitations

The authors qualify CAP’s scope by noting that its taxonomy is not exhaustive, its profiles depend on benchmark coverage and model versions, and several evaluations have specific measurement boundaries.

7 Limitations

  • CAP’s deployment-motivated taxonomy is not exhaustive, and extending the pairwise framework to scalar scoring or rubric-based grading requires new condition definitions.Position Robustness is specifically identified as needing new definitions for those settings.Source: §7 · p. 9
  • Individual judge profiles are point-in-time measurements and should be re-measured as models are updated.The framework, coverage matrix, and estimators are described as judge-set-independent.Source: §7 · pp. 9–10
  • Approximate content measurements use task subsets rather than direct error annotations, and scope sensitivity is directly measured only on JUDGEREVA-STANDARD.Naturalistic cross-validation of scope sensitivity remains open because existing naturalistic benchmarks do not annotate omission- or overclaim-type differences between paired responses.Source: §7 · p. 10
  • Rationale Quality relies on an independent LLM evaluator, whose error and bias remain possible despite aggregate human-validation agreement of κ=0.826.Validation used 120 stratified rationales and one unblinded annotator; the evaluator was systematically over-strict on critical-omission rationales.Source: §7 · p. 10
  • The adversarial stress analysis covers JudgeBench-Pro’s bias-injection mechanisms, so other attack settings may yield different CAP degradation patterns.Examples named include multilingual attacks, domain-specific prompt manipulation, and long-context distractions.Source: §7 · p. 10
  • JUDGEREVA-STANDARD uses flawed answers and presentation variants from a small set of generators, and absolute condition values remain generator-dependent.Judge rankings transfer across generators on four of five measurable conditions; the evaluated judges are also seven deployment-tier models, not the full ecosystem.Source: §7 · p. 10

A.1 Data Sources

The appendix describes JUDGEREVA-STANDARD’s source data, planted errors, response and presentation generation, hard subsets, quality controls, item release, and evaluation prompts.

A.1 Data Sources; A.2 Error Type Taxonomy

  • JUDGEREVA-STANDARD draws plain items from six public question sources and assigns each item one of four balanced planted error types.Sources cover six deployment-relevant domains; the error types are factual fabrication, logical reversal, scope-overclaim, and scope-omission.Source: §A.1 · p. 11, §A.2 · pp. 11–12, Table 7 · p. 11

A.3 Response Generation

  • Correct answers are extracted or expanded, while flawed counterparts are generated with error-specific prompts and filtered for subtlety.The average subtlety score is 3.82/5.0, and subtlety predicts judge difficulty across the seven judges (Spearman ρ=−0.115, p<0.001).Source: §A.3 · p. 12

A.4 Presentation Variants

  • Each base item has plain, polished, and length-padded presentation variants supporting A/B swaps, with padding roughly doubling the flawed answer.Length-padded variants are not directly comparable with JudgeBench length-pad values because their severity differs (∼2× here versus ∼1.5× for JudgeBench).Source: §A.4 · p. 12

A.5 Hard Subset Construction

  • The 384-item hard subset combines judge-error-selected Hard-Empirical items with judge-independent Hard-Constructed pairs containing imperfections in both answers.Hard-Constructed items are created by adding a minor nondisqualifying flaw to Answer A and factually correct supplementary statements to Answer B while preserving its planted error.Source: §A.5 · p. 12

A.6 Quality Control

  • Quality control filters and reviews low-subtlety items, checks variant lengths, and finds that excluding flagged items shifts no condition value by more than 2.4 points.Scope-Omission shifts at most 1.7 points and its judge ordering changes only among near-ties; the table reports condition changes after exclusion.Source: §A.6 · pp. 12–13, Table 10 · p. 13

A.7 Item Format and Data Release

  • Released item objects include source, domain, prompt, answers, gold preference, error type, presentation variant, and metadata, with all variants and subsets released.Hard-Constructed items also include annotations describing the added flaw and value.Source: §A.7 · p. 13

B.1 Pairwise Judging Prompt; B.2 Style Augmentation Prompt; B.3 Length Augmentation Prompt

  • The pairwise prompt asks judges for a verdict and brief rationale without a detailed rubric, while augmentation prompts seek style or length changes that preserve factual content.Style rewrites both responses; length padding expands the rejected response while leaving the chosen response unchanged.Source: §B.1 · p. 13, §B.2 · p. 13, §B.3 · p. 13

B.4 Rationale Evaluator Prompts

  • Rationale evaluators assess whether explanations identify a substantive distinction or the planted error, and use only rationales from correct original-order verdicts for JUDGEREVA-STANDARD.External-benchmark prompts assess whether the rationale gives a substantive reason for the chosen answer; the controlled-test prompts target the core planted error.Source: §B.4 · p. 14

C Evaluator Validation

The rationale evaluator shows substantial agreement with a human annotator, but its one-sided errors mean C8 estimates should be read as lower bounds. The appendix also specifies how coverage proxies and external-benchmark rationale scores are handled.

C Evaluator Validation

  • 91.7% raw agreement (110/120) and Cohen’s κ=0.826 accompany a one-sided pattern: all 10 disagreements are evaluator false negatives, concentrated on critical omissions (6/10).Per-error agreement ranges from 100% on factual fabrication to 85.7% on critical omission.Source: C Evaluator Validation · p. 15, Table 11 · p. 15
  • 98.6% implied deployment agreement reweights the validation sample by deployment prevalence, but relies on the enriched sample’s zero-false-positive result.The validation sample contains 78 evaluator-negative and 42 evaluator-positive rationales; the deployment estimate uses Pr[e=1]=0.889.Source: C Evaluator Validation · p. 15
  • Because evaluator errors are false negatives, reported C8 values under-count supporting rationales, especially for critical-omission items.The evaluator was not blinded to its verdict during annotation, which biases agreement toward agreement and makes κ an upper-bound estimate.Source: C Evaluator Validation · p. 15

D Coverage Operators

  • Approximate coverage uses an unchanged native task subset whose dominant error mode plausibly matches a condition, without control over the error mechanism.Subset names and sizes remain benchmark-specific, so approximate cells are not averaged with direct cells.Source: D Coverage Operators · p. 16
  • Ground-truth better-response labels vary by benchmark, including objective correctness, preference annotations, author labels, and planted errors.External-benchmark C8 uses the same correct-verdict evaluator protocol as JUDGEREVA.Source: D Coverage Operators · p. 16

E JudgeBench Supplementary

  • External C8 scores use an evaluator prompt without planted-error annotations and lack the human validation; Table 5 therefore reports n.r.Released per-item outputs permit re-aggregation, while augmented JudgeBench C5 and C6 outputs are not aggregated with direct measurements.Source: E JudgeBench Supplementary · p. 16

F Cross-Benchmark Spearman Matrix

The appendix reports cross-benchmark rank correlations for Position Robustness and examines whether CAP conditions produce redundant judge rankings. The condition rankings show distinct patterns, particularly for Scope-Omission.

F Cross-Benchmark Spearman Matrix

G Cross-Condition Non-Redundancy

  • Scope-Omission ranks negatively against every other condition (ρ from −0.14 to −0.86), while the scope-condition pair correlates at ρ=−0.32.The general-capability cluster C1, C6, C7, and C8 has ρ values from 0.83 to 0.96; with seven judges, these coefficients are descriptive of this set.Source: G Cross-Condition Non-Redundancy · p. 17, Table 14 · p. 17
  • Style Robustness correlates with no other condition above 0.57, and the matrix supports at least three distinct judge orderings rather than full orthogonality.On Hard-Empirical, C3–C4 has ρ=0.21; hard-subset C8 rank correlations are excluded because its cells are too small for stability.Source: G Cross-Condition Non-Redundancy · p. 17

H Strict vs. Pure-Consistency Robustness

The appendix contrasts strict usable robustness, which requires both verdicts to be correct, with pure consistency under perturbation. Stable-but-wrong behavior makes the distinction consequential for weaker judges.

H Strict vs. Pure-Consistency Robustness

I Multi-Generator Replication

Replication and supplementary analyses qualify how CAP profiles transfer across generators, model families, and calibration choices. They also document exclusions, estimators, and released materials.

I Multi-Generator Replication

  • Claude-generated items were uniformly harder than GPT-4o-generated items, but most judge rankings transferred; Scope-Omission rankings did not transfer at fine grain.Mean absolute condition change was 17.8 points, with a maximum of −40 on Factual Sensitivity; rank correlations were 1.00 for C1, 0.91 for C2, 0.84 for C3, and 0.93 for C7, but 0.25 for C4.Source: I Multi-Generator Replication · pp. 17–18, Table 16 · p. 18

J Within-Family L1 Distance

  • Within-family profile distance is smaller than cross-family distance wherever measured, yet expands on JUDGEREVA and under JudgeBench-Pro stress.Family similarity is therefore a coarse, not sufficient, substitution heuristic.Source: J Within-Family L1 Distance · p. 18, Table 17 · p. 18

K Bias-Corrected Estimates

  • 30 of 35 bias-corrected cells exceed 1.0, and 28 have entire confidence intervals above 1.0, so corrected values are treated as diagnostic rather than usable estimates.The raw conditional accuracies with bootstrap confidence intervals remain primary; the correction applies to content conditions and overall accuracy, not C5–C8.Source: K Bias-Corrected Estimates · p. 18, Table 18 · p. 19

L Exclusions and API Failures

  • GPT-4o falls below the five non-GPT judges’ mean on every condition except Scope-Omission, so the construction does not systematically advantage its generator.Gemini judges have about 2% shortfalls on C5–C7 from exhausted rate-limit retries; Llama’s transient billing interruption was resolved by successful retries.Source: L Exclusions and API Failures · p. 19, Table 19 · p. 19

M Estimators

  • Cross-benchmark consistency is Spearman correlation between seven-judge CAP vectors, while stress degradation compares JudgeBench with JudgeBench-Pro and positive values indicate worse stressed performance.Within-family divergence is the L1 distance between two judges’ measurable profiles on the same benchmark.Source: M Estimators · pp. 19–20

N Release Manifest and AI Use Statement

  • The release includes JUDGEREVA-STANDARD, prompts, raw judge outputs, augmentation outputs, the analysis pipeline, and a reference script for computing CAP profiles.The paper also discloses AI use in experiments, data generation and evaluation, and authoring assistance.Source: N Release Manifest and AI Use Statement · p. 20

Related papers

Loading 2610.09229v1…