PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim

arXiv:2602.00521v2cs.AI

TL;DR

LLM-as-a-Judge validation often focuses on observed outputs, leaving the reliability of the judges as measurement instruments insufficiently diagnosed. The paper introduces a two-phase IRT-GRM framework for intrinsic consistency and human alignment, finding interpretable cross-setting signals including broader quality discrimination by LLM judges than humans and different alignment patterns across language and vision tasks.

  • Problem

    Existing validation methods provide limited diagnostic insight into whether LLM judges are stable measurement instruments and how their assessments correspond to humans.

  • Method

    The framework fits IRT-GRM models, uses prompt variations to assess intrinsic consistency, and then compares latent judge signals with human judgments.

  • Results

    θratio > 1 occurs across nearly all model-task combinations; NLP tasks show moderate human-score correlations (r = 0.2–0.5), while vision tasks approach zero.

  • Takeaways & Limitations

    IRT-GRM provides interpretable diagnostics that distinguish calibration mismatch from validity gaps and help identify sources of unreliability.

  • Takeaways & Limitations

    The framework currently applies only to point-scale judgments, and broader vision generalization requires additional multimodal benchmarks.

Abstract

from arXiv · show

While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge, grounded in Item Response Theory (IRT). The framework adopts Graded Response Model (GRM) of IRT and formalizes reliability along two complementary dimensions: (1) intrinsic consistency, defined as the stability of measurement behavior under prompt variations, and (2) human alignment, capturing correspondence with human quality assessments. We empirically examine diverse LLM judges with this framework, and show that leveraging IRT-GRM yields interpretable signals for diagnosing judgments systematically. These signals provide practical guidance for verifying reliablity of LLM-as-a-Judge and identifying potential causes of unreliability.

1. Introduction

LLM-as-a-Judge is widely adopted for scalable evaluation, but existing validation offers limited evidence that judges are stable measurement instruments. The paper introduces an IRT-GRM framework that jointly diagnoses intrinsic consistency and human alignment through interpretable signals.

  • LLM judges provide scalable, cost-effective evaluation across language, vision-language, and reinforcement-learning applications.
  • Reliability has two dimensions: intrinsic consistency under equivalent conditions and human alignment with human quality assessments.
  • Prior methods largely analyze observed scores, limiting separation of stable judge characteristics from variation caused by evaluated samples.
  • The framework uses IRT to analyze latent evaluation traits and applies GRM to ordered categorical ratings.
  • Two sequential phases test prompt-variation consistency first, then compare latent judge signals with human annotators only after consistency is established.
  • Across diverse evaluation settings, IRT-GRM yields interpretable signals for characterizing measurement behavior and identifying unreliability sources.

2. Related Work

Related work evaluates LLM-judge reliability through intrinsic consistency and human alignment, but commonly relies on outcome-level agreement or uncertainty measures. These approaches provide partial evidence while offering limited insight into underlying measurement behavior.

  • Intrinsic Consistency: Intrinsic-consistency studies examine inter-rater agreement, repeated judgments, internal consistency, and uncertainty across inputs or evaluation settings.
  • Intrinsic Consistency: Observed scores and discrete verdicts provide limited information about judgment stability and confidence.
  • Human Alignment: Human alignment is commonly assessed by comparing LLM and human evaluations with correlation and agreement metrics.

3. Methodology

The methodology adapts IRT-GRM to separate latent subject quality from prompt-dependent measurement properties. It uses controlled prompt perturbations and two diagnostic phases to assess intrinsic consistency and human alignment.

  • The framework fits GRM, generates controlled prompt variations, and extracts interpretable reliability metrics.
  • Graded Response Model: GRM models ordered ratings by estimating latent subject quality θ_j alongside prompt-specific discrimination and threshold parameters.
  • Graded Response Model: Sharing θ_j across original, typo, newline, and paraphrase prompts separates prompt effects from true quality differences.
  • Phase 1: Intrinsic Consistency: Prompt sensitivity is probed with typo, newline, and paraphrase perturbations that preserve semantic content.
  • Phase 1 evaluates measurement reliability without human reference, while Phase 2 compares quality perceptions with humans after Phase 1 criteria are satisfied.
  • Phase 1: Intrinsic Consistency: CV diagnoses prompt sensitivity, with CV < 0.1 treated as acceptable consistency.
  • Phase 1: Intrinsic Consistency: Marginal reliability ρ distinguishes true-quality variance from estimation uncertainty, with ρ > 0.7 indicating acceptable reliability.
  • Phase 1: Intrinsic Consistency: Phase 1 combines CV and ρ to distinguish prompt sensitivity from fundamental model limitations.

4. Demonstration

The demonstration evaluates rating-based judges across language and vision benchmarks using controlled prompts and probabilistic fitting. It covers multiple models, datasets, modalities, and rating scales.

  • The study evaluates rating-based judge methods across diverse criteria using human annotations for intrinsic-consistency diagnostics.
  • NLP: NLP experiments use G-Eval and HelpSteer-2 on summarization and dialogue evaluation tasks.
  • NLP: SummEval uses four 5-point quality scales, while TopicalChat uses binary and 3-point dialogue-quality scales.
  • Vision: Vision experiments use VIEScore on ImageHub subsets for control-guided generation and text- and mask-guided editing.
  • The study selects widely adopted frontier models across modalities and runs evaluations through OpenRouter with temperature 0.
  • Prompt variations include typos, inserted newlines, and paraphrases generated through the specified perturbation procedure.
  • GRM is fitted with PyMC and NUTS, while binary scales use a 2-parameter logistic model because GRM assumes K > 2.

5. Results 1: Intrinsic Consistency

Phase 1 evaluates intrinsic consistency through prompt sensitivity and marginal reliability across modality, model, and task axes. Results show that consistency varies substantially by evaluation setting, with no model maintaining acceptable consistency across all criteria.

  • Diagnostic setup: CV ≤0.10 and ρ ≥0.7 define the highlighted consistency criteria across modality, model, and task axes.CV measures prompt consistency, while ρ denotes marginal reliability.
  • Modality: VIEScore shows greater prompt sensitivity than NLP tasks, with CV ranging from 0.16 to 1.32 versus generally below 0.30 for NLP judges.Gemini-2.5 ranges from 0.03 to 0.29 on NLP tasks but exceeds CV > 1.0 across all VIEScore subtasks.
  • Model: Larger models improve prompt robustness in NLP but not VIEScore, where model-scale differences are inconsistent.For example, Qwen3-235B beats Qwen3-30B on SummEval, while VIEScore shows comparable CV values across the two models.
  • Task: SummEval is the most stable task, whereas VIEScore maintains high CV across models and subtasks, indicating task-level prompt sensitivity.Most SummEval models maintain CV < 0.20, while Gemini-2.5 reaches CV 1.08 on HelpSteer-2 Complexity.
  • Marginal reliability: VIEScore combines high prompt sensitivity with high marginal reliability, achieving ρ values of 0.80–0.96 once a prompt is fixed.SummEval also shows consistently high ρ, while TopicalChat and HelpSteer-2 vary by criterion.
  • Marginal reliability: TopicalChat Understandability is least reliable at ρ 0.34–0.53, and no model achieves acceptable consistency across every criterion.Phase 1 therefore recommends verifying prompt and model effects when introducing new judgment tasks.
  • Ablations: Detailed instructions and chain-of-thought prompting reduce CV, while rating-scale changes produce only modest and inconsistent ρ gains.A 5-point scale modestly improves ρ for graded criteria, but a 7-point scale does not consistently improve reliability.

6. Results 2: Human Alignment

Phase 2 evaluates human alignment through discrimination breadth and distributional distance, revealing task-dependent calibration patterns and score-level differences between NLP and vision evaluations.

  • Discrimination breadth: θratio shows broader discrimination in VIEScore, especially perceptual-quality subtasks, where values reach 2.03–4.40 versus 1.0–2.5 for NLP criteria.Values above 1 indicate broader quality perception than humans; several NLP criteria approach near-human calibration.
  • Discrimination breadth: Model scale has no consistent effect on θratio, with comparable calibration values across Qwen3 and GPT-4o model sizes.The results suggest discrimination calibration depends more on task characteristics than model capacity.
  • Discrimination breadth: HelpSteer-2 most frequently achieves near-human θratio values, while TopicalChat and VIEScore PQ consistently show amplified quality perception across models.HelpSteer-2 Coherence and Verbosity include values such as 1.03 and 0.99; VIEScore PQ remains consistently insensitive.
  • Distributional alignment: DW shows no systematic modality effect, with NLP and vision benchmarks spanning similar ranges of 0.15–0.61.Within-modality differences exceed between-modality differences.
  • Distributional alignment: HelpSteer-2 Coherence has the closest human distributional alignment, with DW = 0.15–0.29, whereas VIEScore TIE-PQ has the highest DW = 0.51–0.61.SummEval Coherence also performs well at DW = 0.24–0.32.
  • Score-level interpretation: θratio and DW provide complementary but incomplete alignment signals, while score-level patterns distinguish calibration mismatch from a possible validity gap.NLP trajectories remain monotonic but use different scales; vision trajectories can decrease or plateau and may reflect different quality constructs.
  • Practical recommendations: The framework recommends detailed rubric instructions and chain-of-thought prompting for high CV, rating-scale adjustment for unstable ρ, and post-hoc rescaling for some NLP calibration mismatches.VIEScore patterns instead suggest a construct-validity gap requiring caution about what judges measure.

7. Conclusion

The paper adapts IRT to diagnose LLM-as-a-Judge reliability using latent quality and prompt variations. Its sequential metrics distinguish prompt or scale instability from differences between LLM and human judgments.

  • Framework: The framework uses prompt variations as measurement items and latent quality θ as perceived sample quality.This adapts Item Response Theory to the LLM evaluation setting.
  • Framework: Four metrics are organized into two sequential phases: CV and ρ diagnose intrinsic consistency, while θratio and DW diagnose human alignment.The framework separates prompt sensitivity or rating-scale problems from calibration mismatch or validity gaps.
  • Implication: The framework provides a principled tool for diagnosing when LLM judges behave as reliable measurement instruments and how their unreliability may arise.Its diagnostic signals support practitioner decisions about LLM-based evaluation reliability.

8. Limitations

The framework has important scope and measurement limitations, including restricted evaluation formats, benchmark coverage, bias modeling, reasoning faithfulness, and language or modality coverage.

  • Scope: The framework currently applies only to point-scale judgments, while pairwise and open-ended evaluations remain future work.Structural prompt reformulations are treated as new measurement instruments and should undergo Phase 1 independently.
  • Scope: Vision-language findings rely on VIEScore alone, so their generalization to the broader vision modality requires additional multimodal benchmarks.The experiments also focus on English text and a single vision modality.
  • Modeling: GRM does not explicitly model self-preference, position, or verbosity biases, although Differential Item Functioning could incorporate them in future work.These concerns remain outside the current reliability diagnosis.
  • Measurement: The framework diagnoses measurement behavior from observed ratings without assessing whether judges’ internal reasoning is faithful to their decisions.This limits conclusions about the faithfulness of generated explanations.

Impact Statement

The work promotes more responsible LLM-as-a-Judge use by clarifying when automated evaluators behave as reliable measurement instruments and when they do not.

  • Responsible use: Separating intrinsic consistency from human alignment helps mitigate over-trust in LLM-based evaluation for benchmarking and model development.The impact is bounded by the framework’s limited evaluation formats, languages, and modalities.
  • Boundaries: The framework does not directly extend to pairwise or open-ended evaluations and does not assess the faithfulness of judges’ internal reasoning.These boundaries constrain its broader impact.
  • Generalization: Multilingual evaluation and a wider range of modalities remain unexplored and may exhibit different reliability characteristics.Additional validation is needed for diagnostics to generalize across settings.

The Use of Large Language Models

The paper derives and implements an IRT Graded Response Model for ordered judge ratings, treating prompt variants as repeated measurement items. It uses Bayesian posterior inference and controlled prompt perturbations to estimate interpretable reliability quantities.

  • IRT-GRM formulation: The Graded Response Model assigns probabilities to ratings at least k using latent subject quality, prompt-specific discrimination, and ordered thresholds.The model converts cumulative probabilities into exact rating probabilities by differencing adjacent categories.
  • IRT-GRM formulation: Prompt variants act as repeated, independently distorted measurements sharing one latent subject quality, making θ_j comparable across judges.The shared θ_j and variant-specific parameters jointly identify the latent quality when at least two prompt variants are available.
  • Inference: The model uses priors on θ_j, α_p, and β_pk, ordered thresholds, and NUTS-based posterior inference with multiple chains and samples.Posterior inference propagates estimation uncertainty into downstream reliability metrics such as marginal reliability ρ.
  • Prompt perturbations: Prompt sensitivity is tested with minimal typos, newline insertions, and synonym substitutions while preserving semantic content.The perturbations are generated and applied consistently across models within each dataset.
  • Experimental setup: The experiments use prompt templates from SummEval, TopicalChat, HelpSteer-2, and VIEScore, with selected output instructions modified for JSON formatting.Example variations are provided for SummEval and HelpSteer-2.

D. Intrinsic Consistency Results on VIEScore

VIEScore exhibits substantially greater intrinsic prompt sensitivity than the NLP benchmarks. The proposed diagnostics are compared with conventional internal-consistency measures to expose variation that aggregate agreement can miss.

  • VIEScore results: Table 6 reports intrinsic consistency results for VIEScore under detailed prompts and chain-of-thought prompting across the evaluated models.The table distinguishes Gemini-2.5 Flash and the Llama-4 variants, with Qwen3 models represented by their vision-language variants for VIEScore.
  • Score-level diagnostics: Figures 2–7 provide additional score-level latent quality trajectories for SummEval, TopicalChat, HelpSteer-2, and VIEScore subtasks.The figures summarize median θ by human score for the listed evaluation settings.
  • Comparison with existing metrics: McDonald’s ω remains high across most settings, but CV and ρ vary substantially across benchmarks and models, including VIEScore settings.The comparison indicates that ω does not fully capture variation across prompt conditions.

F.2. Human Alignment Metrics

The paper compares its human-alignment diagnostics with Pearson correlation, Cohen’s κ, and Krippendorff’s α. These conventional metrics show broadly similar patterns but provide limited insight into why LLM and human judgments disagree.

  • Conventional alignment metrics: Pearson correlation, Cohen’s κ, and Krippendorff’s α show broadly similar human-alignment patterns across the evaluated settings.The metrics are reported in Tables 8 and 9 for comparisons between LLM judges and human ratings.
  • Interpretability: The conventional alignment metrics provide limited insight into the source of disagreement between LLM and human judgments.The comparison motivates diagnostics that distinguish aspects of alignment rather than reporting agreement alone.
  • Stability analysis: Bootstrap standard errors remain consistently small across benchmarks and models, indicating that observed CV and ρ differences are robust to sampling variability.Table 10 reports the bootstrap standard errors for the Phase 1 and marginal-reliability measures.
  • Model differences: Kruskal–Wallis tests find statistically significant model differences for both CV and ρ in every benchmark–criterion pair, with p < .001.The result is reported across all settings in Table 11.
Loading 2602.00521v2…