PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We on the Right Way to Assessing LLM-as-a-Judge?

Yuanning Feng, Sinan Wang, Zhengxiang Cheng, Yao Wan, Dongping Chen

arXiv:2512.16041v1cs.CLcs.AI

TL;DR

Existing LLM-as-a-Judge benchmarks depend on costly and potentially biased human ground truth. Sage replaces that dependency with annotation-free local and global consistency tests, finding substantial robustness deficiencies in current judges while showing that self-generated rubrics improve consistency. Human judgments and human-derived fine-tuning remain important scope boundaries because they can contain inconsistency and bias.

  • Problem

    Existing LLM-as-a-Judge benchmarks rely on human-annotated ground truth, which is costly and vulnerable to bias and inconsistency.

  • Method

    Sage evaluates judges without human annotation using symmetrized pairwise comparisons, IPI for local consistency, and TOV for global logical coherence.

  • Results

    Sage’s metrics are stable and align with supervised benchmarks, while current state-of-the-art judges show significant robustness deficiencies; self-generated rubrics reduce IPI by 16.1% and TOV by 11.0%.

  • Takeaways & Limitations

    Situational preference helps explain inconsistent judging, and fixed self-generated rubrics, fine-tuning, and panels can improve judging consistency.

  • Takeaways & Limitations

    Fine-tuned judges trained on human evaluations can inherit biases and inconsistencies from human labels, while aggregate metric analysis assumes independently and identically distributed per-question scores.

Abstract

from arXiv · show

LLM-as-a-Judge has been widely adopted as an evaluation method and served as supervised rewards in model training. However, existing benchmarks for LLM-as-a-Judge are mainly relying on human-annotated ground truth, which introduces human bias that undermines the assessment of reliability and imposes scalability constraints. To overcome these limitations, we introduce Sage, a novel evaluation suite that assesses the quality of LLM judges without necessitating any human annotation. Inspired by axioms of rational choice theory, Sage introduces two new lenses for measuring LLM-as-a-Judge: local self-consistency (pair-wise preference stability) and global logical consistency (transitivity across a full set of preferences). We curate a dataset of 650 questions by combining structured benchmark problems with real-world user queries. Our experiments demonstrate both the stability of our metrics and their high correlation with supervised benchmarks like LLMBar and RewardBench2, confirming Sage's reliability as an evaluation suite for the robustness and accuracy of LLM-as-a-Judge. Based on Sage, we reveal that current state-of-the-art LLMs exhibit significant reliability problems when acting as judges in both scoring and pairwise settings; even the top-performing models, Gemini-2.5-Pro and GPT-5, fail to maintain consistent preferences in nearly a quarter of difficult cases. We attribute this to a new phenomenon called situational preference, which explains why explicit rubrics or criteria can help the model judge consistently across answer pairs. Our further analysis shows that finetuned LLM-as-a-Judge is a feasible method to boost performance, and the panel-based judge as well as deep reasoning can enhance the judging consistency. We also find substantial inconsistency in human judgments, which indicates that human annotation may not be a reliable gold standard.

1. Introduction

LLM-as-a-Judge offers scalable evaluation and automated feedback, but judge reliability is undermined by model biases and human-annotated benchmarks. Sage addresses this gap with annotation-free consistency metrics and a broad evaluation dataset.

  • LLM-as-a-Judge scales evaluation and can provide automated rewards during training and filtering at inference time.
  • Human-annotated ground truth is costly, labor-intensive, noisy, and vulnerable to disagreement and cognitive bias.These problems limit benchmark scale and diversity while weakening the reliability of human annotations.
  • Sage evaluates LLM-as-a-Judge robustness without human annotation by testing order-invariant pairwise preferences and transitivity across preference sets.The framework interprets coherence failures as evidence of unreliable situational preferences.
  • Sage introduces IPI for local pairwise consistency and TOV for global logical coherence.
  • The benchmark combines 650 questions from structured problems and real-world user queries, and evaluates thirteen prominent LLMs.The metrics show intrinsic stability and external alignment with LLMBar and RewardBench2.
  • Current judges show substantial robustness deficiencies, especially when distinguishing closely matched candidate answers.Specialized fine-tuning generally improves robustness, panels can improve performance by up to 15%, and self-generated rubrics reduce IPI by 16.1% and TOV by 11.0%.

2. Assessing LLM-as-a-Judge with Sage

Sage evaluates judges through symmetrized round-robin pairwise comparisons, separating local order sensitivity from global preference incoherence. Its metrics quantify reversals and the minimum changes needed to produce a transitive weak total order.

  • 2.1. Problem Formulation: The protocol encodes each pairwise outcome as superior, inferior, or tied, then aggregates per-question IPI and TOV across the question set.
  • 2.1. Problem Formulation: Sage performs a full round-robin evaluation of all unique candidate-answer pairs under a symmetrized protocol.Each question yields a complete set of pairwise judgments for coherence analysis, while reversed presentation exposes positional bias.
  • 2.2. Symmetrized Evaluation Protocol: A naive single-pass protocol exhibits positional bias, motivating two queries for each unordered answer pair.The inconsistency rate compares a judgment with the logical inverse produced after reversing answer order.
  • 2.3. Two Evaluation Metrics: IPI measures local inconsistency as disagreement between judgments of a pair and its reversed presentation.Higher IPI indicates greater local inconsistency caused by positional bias or judge stochasticity.
  • 2.3. Two Evaluation Metrics: TOV measures global incoherence as the minimum preference changes needed to transform observed judgments into a valid weak total order.The target order permits ties, and higher TOV signifies more severe logical contradictions.

3. The Construction of Sage

Sage combines benchmark categories with real-world queries into a 650-question dataset and constructs Easy and Hard tiers with differing answer-quality diversity.

  • Dataset composition: 650 questions combine five RewardBench2 categories with WildChat-1M queries, providing broad topical coverage for Sage.The questions are projected against 500k WildChat-1M questions, where they spread broadly across the embedding space.
  • Tier construction: Each question receives six candidate answers, which are organized into Sage-Easy and Sage-Hard tiers.The tiers are designed to create different levels of answer-quality diversity for judging.
  • Sage-Easy: Sage-Easy uses six models with a clear capability gradient, creating widely varying answer quality for relatively simple comparisons.Its setup reflects comparing outputs from models with differing capabilities.
  • Sage-Hard: Sage-Hard uses six answers from Gemini-2.5-Flash, requiring finer distinctions between more homogeneous responses.The tier targets cases where a judge must distinguish subtly varied outputs from one capable model.
  • Difficulty validation: The CV distribution shifts lower for Sage-Hard, confirming lower answer-quality dispersion and a more difficult judging task.CV is computed from six reward-model scores per question; human pairwise comparisons averaged 7.3 minutes for Sage-Easy versus 10.4 minutes for Sage-Hard.

4. Experiment and Analysis

Sage evaluates LLM-as-a-Judge consistency without human annotation using stable local and global preference metrics. Experiments find substantial robustness failures, especially on fine-grained comparisons, while rubrics, fine-tuning, and deeper reasoning can improve consistency.

  • 4.1. Validating Metric Stability and Robustness: Sage’s metrics are stable and externally aligned with established supervised benchmarks, supporting their use for evaluating judge robustness and accuracy.Theoretical analysis and empirical checks report minimal variance, while LLMBar results show strong positive correlation with Sage metrics.
  • 4.3. Evaluating LLM-as-a-Judge with Sage: Sage’s metrics show substantial robustness deficiencies in current state-of-the-art LLM judges, with Gemini-2.5-Pro among the most robust models.Lower IPI and TOV indicate stronger local self-consistency and global logical coherence.
  • 4.3. Evaluating LLM-as-a-Judge with Sage: Approximately 200% higher IPI and TOV scores on Sage-Hard show that models struggle when candidate answers differ only subtly.This weakness threatens inference-time enhancement methods such as rejection sampling and Monte Carlo Tree Search.
  • 4.4. In-depth Analysis: A single fixed rubric per question reduces IPI by 16.1% and TOV by 11.0%, indicating that situational preferences contribute to inconsistent judgments.The rubric prevents evaluation criteria from shifting across answer pairs.
  • 4.4. In-depth Analysis: Fine-tuning generally improves evaluation robustness, but JudgeLRM shows capacity-dependent effects: its 7B model improves whereas its 3B variant regresses.The authors associate the regression with vulnerability to training-data artifacts and inherited biases.
  • 4.4. In-depth Analysis: Human judgments are also inconsistent, while direct scoring often disagrees with pairwise preferences because its internal quality scale is poorly calibrated.These findings challenge the assumption that human annotation is a perfectly reliable reference.

5. Related Work

Prior work uses LLMs for scalable evaluation but documents substantial judge biases and develops specialized benchmarks whose human-annotated ground truth remains costly and unreliable.

  • LLM-as-a-Judge offers scalable evaluation of generative outputs, but its reliability is threatened by verbosity, position, self-enhancement, and authority biases.
  • Specialized benchmarks scrutinize adjudicative performance, yet their reliance on human gold standards inherits bias, disagreement, and limits in assessing complex outputs.
  • Fine-tuned judges learn preference patterns for automated evaluation, but human-derived training data can transfer annotator subjectivity and inconsistency into model predictions.

6. Conclusion

The paper introduces Sage, a human-annotation-free framework that evaluates judge consistency through local and global logical consistency. It identifies robustness deficiencies in current judges and shows that rubrics, fine-tuning, and reasoning can improve consistency while human judgments are also unstable.

  • Sage evaluates LLM judges without human annotation by measuring local and global logical consistency, providing a scalable framework for diagnosing evaluator reliability.
  • Current state-of-the-art judges exhibit significant robustness deficiencies, attributed to situational preference, while self-generated rubrics mitigate this inconsistency.
  • Sage shows that fine-tuning and explanatory reasoning improve evaluation performance or consistency, and that human judgments themselves can be fragile.

Reproducibility Statement

The paper supports reproducibility by releasing its code, dataset, and collected model responses, while documenting the framework and dataset construction in the main text and appendix.

  • The authors will release source code, the curated dataset, and collected model responses to support research reproducibility.
  • The formal framework, symmetrized protocol, IPI and TOV metrics, dataset curation, and implementation details are documented across Sections 2–3 and Appendix B.1.

Ethics Statement

The dataset combines RewardBench2 and WildChat-1M while addressing privacy and inappropriate-content risks through automated filtering and manual review.

  • The dataset draws from the public RewardBench2 benchmark and WildChat-1M corpus.
  • Automated filtering and manual review of every selected question address potential private information and inappropriate content while supporting data-sharing and privacy standards.

The Use of Large Language Models (LLMs)

The analysis models LLM-judge instability as stochastic variation in repeated pairwise judgments and uses conformal prediction to certify single-judgment stability. A calibration procedure approximates stable outcomes by modal judgments and bounds disagreement for new judgments.

  • Sources of instability: The framework treats instability as fluctuations in the judgment Jℳ(Q, Ai, Aj) when identical question-answer pairs are queried repeatedly.The analysis then certifies individual judgments, bounds per-question metric variance, and establishes aggregate-score stability.
  • Calibration procedure: A calibration set uses 800 question-answer pairs, each judged 20 times, producing 16,000 judgments for estimating modal stable outcomes.For each pair, the most frequently observed outcome is defined as its stable judgment.
  • Conformal certification: Conformal prediction converts disagreement with stable judgments into a prediction interval for an unseen judgment.The procedure calibrates nonconformity scores from repeated judgments and evaluates whether new judgments match their stable counterparts.
  • Conformal certification: With miscoverage rate α = 0.03, the analysis provides a probabilistic guarantee that a new pairwise judgment matches its stable judgment with high probability.The paper characterizes this guarantee as showing that individual pairwise comparisons are highly likely to remain stable.

A.2. Bounding the Variance of Per-Question Metrics

The variance analysis propagates instability in individual pairwise judgments into bounds on per-question IPI and TOV scores and then into aggregate Sage metrics. Under an i.i.d. assumption across 650 questions, the resulting aggregate variances are exceptionally small.

  • Per-question variance: The analysis bounds TOV changes by the number of unstable judgments, yielding a tight upper bound on per-question TOV variance.Because each unstable judgment can change TOV by at most one, |∆TOV(Q)|≤X supports the variance calculation.
  • Per-question variance: The same reasoning bounds IPI variance because each unstable judgment can affect at most one inconsistent answer pair among 15 unique pairs.The normalized deviation is conservatively bounded through X/N, with N = 15.
  • Aggregate variance: Assuming independent and identically distributed per-question scores, aggregate IPI and TOV variances equal their per-question variances divided by 650.The aggregate metrics are arithmetic means over the full Sage question set.
  • Aggregate variance: The resulting aggregate variances are exceptionally small, indicating that final IPI and TOV scores are highly concentrated around their expected values.The theoretical analysis is reported as aligning with empirical findings and supporting Sage’s consistency.

B.1. Dataset Curation

The dataset combines structured RewardBench2 questions with real-world user queries to balance formal evaluation criteria and natural interactions. Its construction targets semantic diversity, while reward-score dispersion is used to distinguish answer-set difficulty.

  • Dataset sources: The dataset draws manually selected, semantically unique questions from five RewardBench2 categories and supplements them with real-world user interactions.The structured categories are Factuality, Focus, Precise Instruction Following, Mathematics, and Safety.
  • Dataset sources: The dual-source design aims to cover a wide semantic space by balancing formal assessment criteria with genuine user-query unpredictability.This combination is presented as important for robust evaluation of LLM judges.

C.1. Evaluation Process

Arena-Hard-Auto evaluates models through pairwise comparisons against Gemini-2.5-Pro, with an LLM judge generating an ideal solution and using a two-game, position-swapped protocol. Bradley-Terry aggregation and bootstrap resampling convert judgments into scores with uncertainty estimates.

  • Pairwise evaluation: Each evaluated response is compared with a fixed Gemini-2.5-Pro baseline through an LLM-as-a-Judge pairwise evaluation.The judge compares the candidate response with the baseline response for every benchmark prompt.
  • Pairwise evaluation: The judge first generates an ideal solution, then assigns preference on a five-point Likert scale to capture the degree of superiority.This setup uses the judge’s generated solution as part of the evaluation process.
  • Bias mitigation: Each prompt is evaluated twice with model-output positions swapped to mitigate positional bias.The two rounds form a two-game setup for the same prompt.
  • Score aggregation: Bradley-Terry aggregates thousands of head-to-head judgments into a continuous latent-strength score for each model.The model converts discrete Likert-scale win, loss, and tie outcomes into comprehensive scores.
  • Uncertainty estimation: Bootstrapping repeatedly resamples pairwise judgments to estimate win-rate distributions, final average scores, and 95% confidence ranges.Each resampled dataset produces a recalculated win-rate against the baseline.

D.1. Metric Consistency across Temperatures

Sage’s IPI and TOV metrics remain stable across temperature settings, supporting their robustness against sampling variability. The supplied results also compare direct-judgment protocols and fine-tuned judges on Sage.

  • Metric stability: IPI and TOV scores remain stable across all tested temperature settings, indicating that Sage captures consistent judgment properties rather than random generation artifacts.The evaluation uses Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 under five temperatures.
  • Fine-tuned judges: Fine-tuning enhances judgment robustness on Sage-Easy compared with the corresponding base models.The supplied table reports performance for fine-tuned models and their base models.
  • Direct judgment protocol: Removing explanatory reasoning generally degrades robustness on Sage-Easy, while its effect on Sage-Hard is mixed across models.Weaker models become less stable on Sage-Hard, whereas some state-of-the-art models maintain or slightly improve consistency.
  • Direct judgment protocol: Without reasoning, alignment between direct scoring and pairwise comparison generally falls on Sage-Easy and varies substantially on Sage-Hard.The comparison is against the standard setting with explanatory reasoning.

E. Prompts and Case Study

The paper evaluates several judge-prompt formats and uses a case study to explain why debate-based judging can degrade. The analysis highlights persuasive hallucination, anchoring, rhetorical pressure, and redundant reasoning as failure mechanisms.

  • E.2. Prompt of RewardBench2: RewardBench2 uses a four-assistant comparison prompt requiring an explanation followed by exactly one assistant choice.The prompt emphasizes instruction following, helpfulness, relevance, accuracy, depth, creativity, and detail while asking judges to avoid presentation-order bias.
  • E.3. Prompt of Arena Hard Auto: Arena Hard Auto asks the judge to generate its own answer first, compare both assistants against it, and select among five graded pairwise verdicts.The prompt also directs the judge to correct mistakes and consider helpfulness, relevance, and ambiguity.
  • E.4. Additional Prompt: Additional prompts vary the evaluation criteria and verdict formatting while instructing judges to assess helpfulness, accuracy, relevance, depth, and presentation-order neutrality.Several variants require explanations before a discrete A, B, or C decision.
  • E.6. Case Study and Explanation of the Degradation of ChatEval: Rhetorical strength can drive convergence through plausible but fallacious arguments rather than logical consensus.The paper identifies authority appeals, hasty generalization, and emotive language as examples of this pressure.
  • E.6. Case Study and Explanation of the Degradation of ChatEval: Anchoring causes later debate turns to organize around an early judgment rather than reevaluating the underlying assessment criteria.The discussion shifts from seeking the optimal judgment to entrenching the first persuasive argument.
  • E.6. Case Study and Explanation of the Degradation of ChatEval: Repeated debate turns may add agreement noise instead of new evidence, making the final adjudication less able to distinguish genuine reinforcement from redundancy.The paper describes later turns that restate earlier claims and argues that this noisy stream reduces final-decision robustness.
  • E.6. Case Study and Explanation of the Degradation of ChatEval: The case study reports a debate that reaches the wrong result even though assistant A is correct, with persuasive hallucination and anchoring identified in the chat log.The example is presented as evidence supporting the proposed degradation mechanisms.
  • E.6. Case Study and Explanation of the Degradation of ChatEval: Confident but inaccurate claims can persuade other agents when authoritative language is accepted without independent verification.In the case study, subsequent agents validate the initial geological error instead of checking it.
Loading 2512.16041v1…