PaperlayerView exact revision on arXiv

Source-linked AI summary

Rating the Raters: Rasch Measurement Theory for LLM Evaluation

Pratik S. Sachdeva, Nathan Boudol

arXiv:2608.27463v1cs.AI

TL;DR

Standard LLM evaluation often collapses objects, items, and raters into aggregate scores, limiting what can be learned about the measurement process. This paper applies many-facet Rasch models to nine LLMs rating hate speech against a human-anchored corpus and finds systematic differences from human raters across multiple dimensions. It argues that RMT should be part of the toolkit for evaluating LLMs as examinees, judges, and raters.

  • Problem

    Standard evaluation reports collapse objects, items, and raters into aggregate scores, limiting understanding of what is being measured.

  • Method

    The paper fits many-facet Rasch models to annotations from nine LLMs using the RMT-constructed Measuring Hate Speech corpus.

  • Results

    LLM raters systematically differ from human raters in severity, item calibration, rating-scale use, question-order robustness, and target-identity sensitivity.

  • Takeaways & Limitations

    RMT supplies a principled vocabulary and toolkit for interrogating LLMs across examinee, judge, and rater evaluation paradigms.

  • Takeaways & Limitations

    The empirical analysis is restricted to the largely English MHS hate-speech corpus and does not demonstrate RMT’s application to other contexts or datasets.

Abstract

from arXiv · show

LLMs now sit on every side of evaluation: as examinees scored on benchmarks, judges of other models' outputs, and raters of human-generated content. Each paradigm can be viewed as a measurement problem, where a latent property of an object is probed with items from an instrument (e.g., benchmark) by raters. Standard evaluation practices often neglect the contributions of each core component to the end result, limiting our understanding of what is being measured. Rasch measurement theory (RMT) is well-suited to this kind of problem. RMT decomposes ordinal ratings into separable facets on a common scale. It further provides a battery of diagnostics that can identify miscalibrated measurements and rater biases. We present a case study of RMT applied to the LLM-as-rater paradigm using the Measuring Hate Speech corpus, whose construct was itself built under RMT. We fit a series of many-facet Rasch models to annotations from nine LLMs spanning families and capability levels. Our analyses show that LLMs systematically differ from human raters in severity, item-level calibration, question-order robustness, target-identity sensitivity, and rating scale use, which all would be obscured by standard evaluation practice. Overall, we argue that RMT belongs in the toolkit for evaluating LLM-as-examinee, -judge, and -rater paradigms.

1 Introduction

LLM evaluation spans examinee, judge, and rater roles, but standard aggregate scores collapse objects, items, and raters into a single quantity. The paper presents RMT as a principled framework for separating these contributions and diagnosing differences.

  • LLMs participate in evaluation as examinees, judges, and raters of human-generated content.
  • Standard accuracies and annotator-agreement reports collapse the measured object, item, and rater into one aggregate score.
  • RMT decomposes ordinal ratings into separate object, item, and rater contributions on a common measurement scale.
  • The study applies many-facet RMT models to annotations from nine LLMs using the RMT-constructed Measuring Hate Speech corpus.The corpus contains 50,070 comments labeled by 11,143 annotators on a ten-item instrument.
  • The paper argues that RMT provides a principled vocabulary for evaluating LLMs as examinees, judges, and raters.

2 Rasch Measurement Theory

RMT models measurement as the interaction of objects, instrument items, and raters, transforming ordinal responses into a shared latent scale. Its invariance requirement and fit diagnostics support interpretable comparisons across these facets.

  • Measurement theory seeks meaningful comparability by modeling interactions among objects, instrument items, and raters.
  • RMT requires invariance: the instrument should function similarly across raters, objects, and items.It uses additive models and requires observations to fit the model rather than altering the model to fit the data.
  • The many-facet model recovers a continuous scale from ordinal responses by modeling adjacent response-category choices.
  • The model uses four additive facets: comment hatefulness, item difficulty, rater severity, and response-category thresholds.
  • Because comments, items, and raters share a common logit scale, their parameters can be interpreted together within the construct.

3 Methods

The study uses the Measuring Hate Speech corpus and ratings from LLMs spanning model families, providers, and scales. It fits human-only, human-anchored, and LLM-only measurement models to compare rating behavior.

  • The Measuring Hate Speech corpus contains 50,070 comments labeled by 11,143 annotators across ten items spanning the hate-speech spectrum.Most items use five Likert-like responses, while the overall Hate Speech item is ternary; target identities are also annotated.
  • The analysis uses a 70-comment reference set and an expanded 5,990-comment set with human-rater agreement on target identity.
  • The primary model set contains nine LLMs spanning families, providers, and parameter scales.
  • Each LLM received the MHS survey items verbatim with task instructions and a specified JSON response format.
  • The researchers fit human-only, human-anchored, and LLM-only measurement models.Human-anchored models fixed comment scores and item difficulties to the human-only scale while estimating LLM-specific parameters.

4 Results

The results use RMT to anchor LLM annotations to a human measurement scale, diagnose item- and rater-level differences, and compare an LLM-only scale with its human counterpart.

  • Results overview: Nine LLMs were evaluated with RMT models anchored to the human measurement scale, alongside an LLM-only scale analysis.The analysis covers severity, item calibration, question-order bias, target-identity sensitivity, rating-scale functioning, and human–LLM scale comparison.
  • 4.1 Measurement Models Quantify Differences between LLM Raters: All nine LLMs had lower severity than the median human rater, indicating greater readiness to label comments as exhibiting hatefulness aspects.Across an expanded model set, severity and the AA Intelligence Index were positively correlated (Pearson r = 0.52, p < 10^-3).
  • 4.2 Item-Level Interaction Terms Diagnose Miscalibration: 30 of 90 item–LLM interaction terms were significant, with universally negative Hate Speech terms and generally positive Dehumanize, Violence, and Genocide terms.These interaction terms diagnose which items drive severity differences relative to the human baseline.
  • 4.3 Differential Rater Functioning to Interrogate Model Bias: Reversing question order increased severity for most models, with significant effects concentrated among the four lowest models by AA Intelligence Index.The induced odds ratios ranged from 1.31 for MiniMax M2.5 to 1.84 for GPT-OSS 120B.
  • 4.3 Differential Rater Functioning to Interrogate Model Bias: Target-identity effects were significant for 82 of 126 interaction terms, with lower hate thresholds for Muslim-targeting comments and generally higher thresholds for Gay-targeting comments.Jewish-targeting odds ratios were below 1 for most models, while Black-targeting effects split by AA Index.
  • 4.4 Constructing an LLM-Only Scale: The LLM-only scale preserved comment ordering relative to the human scale (Spearman ρ = 0.911) but diverged at the upper end, where severe items had greater relative separation.Respect, Attack, and Insult retained similar spacing, whereas Dehumanize, Violence, and Genocide separated more on the LLM scale.

5 Discussion

The paper argues that LLM evaluation should be treated as a measurement problem and that RMT offers a principled framework for interrogating LLMs across evaluation roles. Its findings show systematic differences between LLM and human raters, while applying RMT beyond hate-speech rating remains empirically unvalidated.

  • Broader applications: RMT can quantify item difficulty and model ability in LLM-as-examinee benchmarks, exposing whether benchmark items function as calibrated instruments.For subjective LLM-as-judge tasks, RMT can also quantify examinee performance and judge severity.
  • Methodological contribution: RMT unifies severity, rater bias, prompt sensitivity, and rating-scale diagnostics within a quantitative framework anchored to an explicit construct.The framework formalizes previously scattered evaluation phenomena, including differential rater functioning and Likert-scale disorder.
  • Empirical findings: LLM raters systematically differ from human raters in severity, item treatment, rating-scale use, and item ordering.These differences include lower severities, inconsistent category use on severe items, and different item positions in LLM-constructed scales.
  • Implication: Construct validity established for human raters may not transfer to LLM raters using the same instrument.The finding matters for both LLM-as-judge and LLM-as-rater settings.
  • Limitation and scope: The paper acknowledges substantial RMT overhead, including construct theorization, instrument design, and validation.These requirements are presented as costs of applying principled measurement practices.

6 Related Work

Prior work frames LLM evaluation as a measurement problem and applies psychometric approaches to model ability, item difficulty, and rater effects. This paper centers Rasch measurement theory to study how LLMs behave as scalable raters of hate speech and other perspectivist tasks.

  • LLM evaluation research increasingly addresses construct validity, scale construction, and score interpretation as measurement concerns.
  • Related IRT and psychometric studies examine item difficulty, model ability, and rater effects in NLP and LLM evaluation.
  • This work centers Rasch measurement theory and contributes evidence about modern LLM behavior in hate speech and other perspectivist rating tasks.

Limitations

The empirical analysis is restricted to hate speech in the largely English MHS corpus, while proposed applications to other LLM evaluation paradigms remain unvalidated. Prompt construction and sampling choices also bound interpretation.

  • The specific analysis is restricted to hate speech via the MHS corpus and does not empirically validate RMT in other contexts.Applying RMT elsewhere may require additional construct and instrument development or adaptation.
  • The study is largely English, uses one sample per comment, and does not establish how other prompt constructions would affect parameter estimates.Specific analyses nevertheless include up to 5,990 comments.

Ethical Considerations

The study uses the Measuring Hate Speech corpus under its intended-use conditions, with personal identifying information removed. It cautions that LLM annotation should not be treated as a straightforward replacement for human moderation on sensitive topics.

  • The Measuring Hate Speech corpus contains social media comments often including offensive language and slurs, and was used under the authors’ intended-use conditions.
  • Personal identifying information was removed from all comments before use.
  • The analyses are not intended to advocate replacing humans for sensitive annotation tasks such as hate speech detection.The paper highlights possible adverse and unintended consequences, particularly for specific identity groups, and warns that benchmark accuracy may not transfer well.

A System Prompts

The annotation system prompt instructs models to analyze a social-media comment, answer the target-group question and ten survey items, and return only a schema-conforming JSON object. The prompt encodes ordered response options for sentiment, respect, hate-related dimensions, and overall hate speech.

  • Output format: The required output uses uppercase letters, arrays for target groups, one letter for other fields, and a single valid JSON object without explanations.
  • Target-group question: The prompt asks models to identify whether a comment targets groups based on race, religion, national origin, gender, sexuality, age, disability, or political ideology.
  • Survey items: Several items use five ordered response categories ranging from strongly negative or disrespectful to strongly positive or respectful.
  • Survey items: The hate-speech item defines hate speech as bias-motivated, hostile, malicious language targeted at a person or group because of innate characteristics, with Yes, No, and Unclear responses.

B Exhaustive List of LLMs

The primary and expanded model sets are presented in separate tables, with refusal rates computed over different MHS comment sets.

  • Table 1 presents the primary model set.
  • Refusal rate uses 5,990 full-set MHS comments for Table 1 and 70 reference-set comments for Table 2.
  • Table 2 presents the expanded model set used for the Figure 1 scatter plot.
Loading 2608.27463v1…