Source-linked AI summary
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne
TL;DR
Aggregate human–LLM score alignment does not show whether the judges find the same evaluation cases difficult. The paper uses Many-Facet Rasch Models to compare residual-hardness patterns and finds that moderate alignment in latent summary quality does not imply shared residual hardness.
Problem
Human-alignment metrics largely measure scores, rankings, preferences, or labels, leaving less evidence about whether human and LLM judges share the same structure of judging difficulty.
Source: §1 · p. 1, §2.1 · p. 2Method
Results
Takeaways & Limitations
Psychometric residual diagnostics complement aggregate agreement metrics by revealing dimension-specific hardness gaps and cases that may be difficult for one judge group.
Source: §5 · p. 8The analysis uses one multi-rater summarization benchmark, limiting generalizability, and does not assume residual hardness has identical substantive meaning across panels.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs find the same evaluation cases difficult. In this paper, we study this problem in summarization evaluation from a psychometric perspective. We fit Many-Facet Rasch Models separately to human and LLM ratings to decompose scores into latent summary quality, rater severity, dimension severity, and rating-scale thresholds. Building on this decomposition, we define residual hardness as a model-adjusted measure of judging difficulty and compare whether human and LLM judges share the same hardness structure. Across 17 open-weight LLM judges on SummEval, we find that moderate alignment in latent summary quality does not imply alignment in residual hardness. Human and LLM judges differ in which summary--dimension units remain difficult, and this mismatch is strongly dimension-dependent. Consistency shows a pronounced LLM-hard shift, whereas coherence shows a human-hard shift. We further show that human-easy but LLM-hard cases are partially predictable from observable source--summary properties. These findings suggest that aggregate human alignment reflects only part of LLM-as-a-judge reliability, while psychometric residual diagnostics support more informative judge evaluation and more targeted human--LLM collaboration.
1 Introduction
Human–LLM agreement on aggregate scores may conceal differences in which individual evaluation cases are difficult. This paper uses a psychometric diagnostic to separate latent quality from residual judging difficulty and examines that mismatch in summarization evaluation.
1 Introduction
- Aggregate alignment with human scores gives limited information about whether humans and LLM judges find the same individual cases difficult.The paper argues that reliability on individual cases matters alongside aggregate agreement.Source: §1 · p. 1
- The paper defines residual hardness as average absolute rating residuals after a Many-Facet Rasch Model accounts for latent summary quality and systematic rating effects.The model separates quality from rater severity, dimension severity, and rating-scale thresholds; residual hardness is a post-fit diagnostic, not an intrinsic latent difficulty measure.Source: §1 · pp. 1–2
- The study tests whether human–LLM alignment in latent summary quality implies shared residual hardness, how mismatch varies by dimension, and whether human-easy/LLM-hard cases are predictable.SummEval summarization ratings span dimensions such as coherence, consistency, fluency, and relevance.Source: §1 · p. 2
- Human–LLM alignment in latent instance quality does not imply alignment in residual hardness, and the judge panels differ substantially in their residual-hardness patterns.The diagnostic is intended to distinguish alignment in output quality from alignment in judging difficulty.Source: §1 · pp. 1–2
- Human-easy but LLM-hard cases are partially predictable from observable summary-, source-, and system-level features, supporting more targeted human–LLM evaluation workflows.The paper identifies routing such cases to human review, source-grounding checks, or stronger judges as possible collaboration strategies.Source: §1 · p. 2
2 Background
Prior work commonly evaluates LLM judges by their alignment with human scores, while psychometric models offer a way to represent latent traits and systematic rating effects. This paper applies that perspective to residual-hardness patterns.
2.1 LLM-as-a-judge and Human Alignment
- LLM-as-a-judge research commonly assesses validity through agreement with human ratings, rankings, preferences, or labels, but these metrics provide less information about shared judging difficulty.Reliability varies across datasets, evaluated properties, judge models, and forms of human annotation.Source: §2.1 · pp. 2–3
2.2 Item Response Theory
- Item Response Theory models responses through latent traits and item properties, while Many-Facet Rasch Models can also represent systematic influences such as rater severity and category thresholds.Prior psychometric applications to LLM raters focus mainly on score- or rater-level reliability; this paper instead examines post-fit residual-hardness patterns.Source: §2.2 · p. 3
3 Method
The method compares human and LLM ratings using separately fitted Many-Facet Rasch Models, then examines whether model-adjusted residual hardness identifies similar difficult summary–dimension units. It also tests whether observable source–summary features predict human-easy, LLM-hard cases.
3 Method
- The study fits separate human and LLM Many-Facet Rasch Models, then compares latent measurement and residual-based judging hardness.The models account for summary quality, rater severity, dimension effects, and rating-scale thresholds.Source: §3 · p. 3
3.1 Dataset and LLMs
- SummEval contributes 1,600 summaries rated by eight human annotators across coherence, consistency, fluency, and relevance.The LLM panel contains 17 open-weight models rating the same summary–dimension units on a 1–5 scale.Source: §3.1 · p. 3
3.2 Many-Facet Rasch Measurement
- The rating-scale MFRM separates latent summary quality from rater severity, dimension severity, and thresholds between adjacent rating categories.Model adequacy is checked with residual diagnostics, rating-prediction errors, item separation reliability, and use of all five response categories.Source: §3.2 · pp. 3–4
3.3 Residual Hardness
- Residual hardness is the mean absolute residual across raters for a summary–dimension unit after accounting for the fitted MFRM facets.It is a post-fit diagnostic of local judging instability, not low summary quality or an intrinsic item-difficulty measure; larger values may reflect disagreement, model misfit, or unmodeled structure.Source: §3.3 · p. 4
3.4 Predicting Human-Easy and LLM-Hard Cases
- The prediction task classifies human-easy, LLM-hard summary–dimension units from source–summary features using class-balanced logistic regression and five-fold stratified cross-validation.Features include lexical and topical content, length and compression, sentence counts, and digit and negation counts; system IDs are excluded to avoid label leakage.Source: §3.4 · p. 4
4 Results
Human and LLM judges align moderately on latent summary quality, but their residual-hardness patterns align weakly, especially at the summary–dimension level. The mismatch varies by evaluation dimension, while human-easy/LLM-hard cases are partially predictable from observable features.
4 Results
- MFRM diagnostics support usable measurement models, and residual-hardness findings remain stable after rater exclusions and alternative normalizations and calibrations.The quality-versus-hardness contrast also persists when using only ≥7B LLM judges and expert human annotators.Source: §4.1 · p. 5, §4.2 · p. 6
- Quality alignment (Pearson r = 0.372; Spearman ρ = 0.371) weakens for summary-level hardness (r = 0.283; ρ = 0.176) and summary–dimension hardness (r = 0.145; ρ = 0.089).The correlations cover 1,600 summaries for quality and 6,400 summary–dimension units for hardness.Source: §4.1 · p. 5, Figure 1 · p. 6
- Consistency is 74.7% LLM-hard versus 50.8% human-hard, while coherence is 57.3% human-hard versus 34.8% LLM-hard.The opposing dimension-level shifts are also visible in Figure 2; the expert-only versus ≥7B LLM comparison retains a +26.2 percentage point consistency gap.Source: §4.2 · pp. 5–6, Table 2 · p. 6, Figure 2 · p. 7
- Extractive systems show a modest LLM-hard shift, whereas abstractive systems show a small human-hard shift.Table 3 reports percentages of summary–dimension units classified as hard, with the gap defined as LLM minus human hard-rate.Source: §4.2 · p. 6, Table 3 · p. 6
- The pooled classifier predicts human-easy/LLM-hard cases with AUROC 0.631 and AUPRC 0.341, against a positive rate of 23.3%.Relevance is the most predictable dimension, with AUROC 0.717; the predictive signal is reported as robust to feature ablations and stricter grouped splits.Source: §4.3 · p. 6, Table 4 · p. 6
- Feature associations vary by dimension: compression relates to relevance and coherence predictions, while consistency predictors emphasize factual-detail and verification cues.SHAP values are interpreted descriptively because surface features are correlated; fluency shows a weaker pattern, led by summary length.Source: §4.3 · pp. 6–7
5 Discussion
Moderate human–LLM alignment in latent summary quality does not entail alignment in residual hardness: the groups differ in which summary–dimension units remain difficult. The mismatch varies by dimension and is partly predictable from observable source–summary properties.
5 Discussion
- Moderate latent-quality alignment coexists with weak residual-hardness correlation, so human and LLM judges find only partly overlapping cases difficult.Residual hardness is measured after accounting for summary quality, rater severity, dimension severity, and rating thresholds.Source: §5 · p. 7
- Consistency shows a large LLM-hard shift, while coherence shows the opposite human-hard shift; fluency and relevance have smaller gaps.The authors suggest factual verification may burden LLM judges on consistency, while coherence relies more on holistic interpretation.Source: §5 · p. 7
- Human-easy/LLM-hard cases are partly associated with observable source–summary features, with different feature profiles across dimensions.Compression-related features dominate for relevance and coherence; digit count, sentence count, source length, negation count, and system type are associated with consistency predictions.Source: §5 · pp. 7–8, Figure 3 · p. 8
- Psychometric residual diagnostics complement score-level agreement by revealing dimension-specific difficulty gaps that aggregate metrics can miss.The authors suggest using predicted human-easy/LLM-hard cases to prioritize human review, source-grounding checks, or stronger judges.Source: §5 · p. 8
6 Conclusion
The paper uses a Many-Facet Rasch Model to study residual judging difficulty in summarization evaluation. It finds that moderate alignment in latent summary quality does not ensure shared difficulty patterns, which differ especially for consistency and coherence.
6 Conclusion
- Moderate human–LLM alignment in latent summary quality does not imply alignment in residual hardness after controlling for rating-process facets.The analysis uses a Many-Facet Rasch Model to account for summary quality, rater severity, dimension severity, and rating-scale thresholds.Source: §6 · p. 8
- Human and LLM judges differ substantially in summary–dimension residual-hardness patterns, especially for consistency and coherence.Source: §6 · p. 9
- Human-easy but LLM-hard cases are partly predictable from source–summary properties, with different predictive patterns across dimensions.The paper concludes that these diagnostics reveal failure modes hidden by aggregate agreement, while its empirical findings are specific to SummEval.Source: §6 · p. 9
Limitations
The study’s conclusions are bounded by its benchmark, judge panel, and interpretation of residual hardness. Its feature-prediction analysis identifies correlates but does not establish the mechanisms behind mismatch.
Limitations
- The analysis uses one multi-rater summarization benchmark, limiting generalizability to other LLM-as-judge settings.SummEval was selected because it provides repeated multidimensional ratings and identifiable raters needed for the MFRM.Source: Limitations · p. 9
- Results come from a fixed panel of open-weight judges under one prompting setup; other models, prompts, or protocols may yield different patterns.Source: Limitations · p. 9
- Residual hardness is a post-fit diagnostic, not a standalone latent construct, and may reflect disagreement, model misspecification, or uncaptured structure.The analyses do not assume residuals have identical substantive meaning across human and LLM panels.Source: Limitations · p. 9
- The RQ3 feature analysis identifies correlates of human–LLM mismatch but does not establish its causal mechanisms.Source: Limitations · p. 9
A.1 LLM Judge Prompt
The appendix describes a shared rating prompt for the open-weight LLM judges. It specifies the task, rating scale, and output format, and notes how prompts were formatted and scores parsed.
A.1 LLM Judge Prompt
- The study used 17 open-weight LLM judges, and parsed the first valid integer score, retrying once if none was found.Prompts included the full source document without explicit truncation.Source: §A.1 · p. 12, Table 5 · p. 13
- The prompt asks the judge to evaluate a news summary for a requested dimension using the corresponding SummEval rubric.The dimension is coherence, consistency, fluency, or relevance.Source: §A.1 · p. 12
- Judges return one integer from 1 to 5, where 1 means very poor and 5 means excellent.Source: §A.1 · p. 12
A.2.1 MFRM Estimation Details
Sensitivity analyses support the main residual-hardness findings: human–LLM hardness alignment remains weak, and the dimension pattern persists across alternative specifications. Prediction checks find partial, but not uniformly strong, predictability of human-easy/LLM-hard cases.
A.2.1 MFRM Estimation Details
- The MFRM is estimated by regularized maximum likelihood, with adjacent-category logits, mean-centered facet parameters, and ordered rating thresholds.Optimization uses PyTorch AdamW with learning rate 0.05 and weight decay 10^-4.Source: §A.2.1 · p. 13
A.2.2 Agreement-based Sanity Check for Residual Hardness
- Residual hardness is negatively associated with local agreement in both panels, more strongly for humans, but this is only a sanity check and not independent construct validation.Unlike the agreement diagnostic, residual hardness is conditional on fitted MFRM facets.Source: §A.2.2 · p. 13, Table 6 · p. 13
A.2.3 Judge-Capability and Human-Panel Sensitivity
- The consistency hard-rate gap remains +26.2 percentage points in the expert-only versus ≥7B comparison, and the main quality-versus-hardness contrast persists.The authors report that the patterns are not driven solely by the smallest LLM judges or expert–crowd disagreement in the human panel.Source: §A.2.3 · pp. 13–14, Table 7 · p. 13
A.2.4 Rater-by-Dimension Interaction Sensitivity
- With a rater–dimension interaction added, human–LLM residual-hardness alignment remains weak (r = 0.277, ρ = 0.122).The authors conclude the main alignment pattern is not solely attributable to omitted rater-specific dimension effects.Source: §A.2.4 · p. 14
A.2.5 Rater-Misfit Sensitivity
- After excluding misfitting raters, latent-quality alignment remains moderate and residual-hardness alignment weak; consistency remains LLM-hard and coherence human-hard.The dimension gaps for fluency and relevance vary more after exclusion because their original gaps were close to zero.Source: §A.2.5 · p. 14, Table 8 · p. 14, Table 9 · p. 14
A.2.6 MFRM Calibration Comparability Sensitivity
- Consistency remains LLM-hard and coherence human-hard across alternative calibration choices, while residual-hardness alignment remains weaker than quality alignment.Within-group z-scoring leaves correlations unchanged; rank normalization slightly reduces them, and pooled-model hardness alignment rises but remains weak at the summary–dimension level.Source: §A.2.6 · pp. 14–15, Table 10 · p. 14
A.3.2 RQ3 Prediction Robustness and Ablations
- Across feature ablations and grouped splits, prediction remains above chance, though performance decreases when entire source documents are held out.TF–IDF provides most of the signal; surface features remain predictive, and the full model improves over ablations. Removing compression ratio changes AUROC from 0.571 to 0.566 and AUPRC from 0.293 to 0.286.Source: §A.3.2 · p. 15, Table 12 · p. 15, Table 13 · p. 15
- Under stricter quartile thresholds, pooled human-easy/LLM-hard prediction reaches AUROC = 0.676, while coherence falls to AUROC = 0.578.The authors interpret this as pooled predictability rather than uniformly strong robustness across dimensions.Source: Table 14 · p. 15, §A.3.2 · p. 16