Source-linked AI summary
The Rater Ising-Potts Model with LLM-Derived Weights: An Application to Multi-Category Scoring Reliability
Matthias von Davier
TL;DR
The paper addresses multi-category scoring reliability without requiring ordered or equidistant category thresholds. It introduces PARM, which combines category-specific agreement indicators with LLM-derived semantic similarities and tunable scaling; across short-answer and essay applications, it achieves strong agreement with human scores while preserving rubric order.
Problem
Multi-category scoring reliability needs a model that does not depend on ordered thresholds, equidistant scores, or arbitrary category-score assignments.
Method
PARM combines category-specific agreement indicators, LLM-derived embedding weights, and optional normalization and power transformation within a parsimonious Potts formulation.
Results
Accuracy = 0.971 on the plant-growth task and accuracy = 0.476, κ = 0.304 on AERA Essay 1; most misclassifications occurred between adjacent score levels.
Takeaways & Limitations
The findings support PARM as a flexible and interpretable framework for reliability auditing across educational scoring contexts.
Takeaways & Limitations
The paper discusses extensions to multiple raters and hierarchical rating processes, indicating that these designs are extensions rather than the presented core applications.
Abstract
from arXiv · showhide
The Ising model is extended to the Potts model for multinomial data. We introduce a Rater Ising-Potts model that uses agreement indicators between pairs of raters and category labels, with weights derived from LLM embeddings. The model does not presuppose ordered category thresholds or equidistant scoring; instead, it focuses directly on pairwise agreement among raters and assigns category-specific positive weights, making it particularly suited for multi-category scoring reliability when raters evaluate responses using a scoring guide. We demonstrate the model's effectiveness on diverse constructed-response tasks, including balanced short-answer items and more challenging, imbalanced essay prompts from the AERA dataset. Across these settings, the model achieves strong agreement with human scores, with the vast majority of misclassifications occurring between adjacent score levels, confirming its ability to preserve the ordinal structure of scoring rubrics without imposing rigid assumptions. A practical similarity normalization and optional power transformation is introduced as a tunable preprocessing step that sharpens semantic distinctions and can be adapted to different datasets. These findings suggest that LLM-derived semantic similarities, combined with this parsimonious Potts-type formulation and flexible similarity scaling, offer a robust and interpretable framework for reliability auditing in educational assessment contexts. Extensions to multiple raters and hierarchical rating processes are discussed.
1 Introduction
The paper introduces PARM, an Ising–Potts reliability model that uses category-specific agreement indicators and LLM-derived weights without assuming ordered thresholds. Applications to short answers and essays show strong agreement with human scores and preservation of rubric order.
- Contribution: PARM extends the Ising model to multinomial scoring by combining pairwise rater agreement indicators with category-specific weights derived from LLM embeddings.The formulation targets multi-category scoring reliability when raters use a scoring guide.
- Motivation: Unlike conventional polytomous item-response models, PARM does not presuppose ordered category thresholds or equidistant score categories.It focuses directly on pairwise agreement among raters instead.
- Empirical evaluation: The empirical evaluation spans 750 three-level plant-growth short answers and 1,338 four-point AERA Essay 1 responses with differing lengths, categories, and scoring difficulty.These datasets provide contrasting tests of model flexibility and performance.
- Results: Near-agreement rates were 1.000 for the plant-growth task and 0.890 for the AERA essay task.Near-agreement means predictions within ±1 category.
- Results: The model captures the ordinal structure of scoring rubrics even when exact agreement is challenging.The paper links this result to a parsimonious Potts formulation using semantic similarities.
- Implications: PARM is presented as a flexible and interpretable framework for reliability auditing in educational assessment contexts.The paper also discusses extensions to multiple raters and hierarchical rating processes.
2 A Potts Model for Reliability Auditing
PARM represents responses with embeddings, transforms pairwise cosine similarities onto a common scale, and uses them to build total and category-specific similarity predictors.
- Embedding similarities: PARM associates scored natural-language responses with high-dimensional embeddings and computes pairwise cosine similarities between them.Self-similarities are set to zero in the feature construction.
- Similarity preprocessing: Min–max normalization maps off-diagonal similarities into [0, 1] before optional power transformation.The minimum and maximum are taken over pairwise similarities excluding the diagonal.
- Similarity preprocessing: For p > 1, the power transformation emphasizes the strongest similarities and suppresses moderate ones, whereas 0 < p < 1 has the opposite effect.The default is p = 1.
- Model predictors: The transformed similarities construct a total similarity predictor and category-specific similarity sums used in the agreement model.These predictors enter the parsimonious model through category-specific parameters.
- Parameterization: The resulting model has 3C free parameters: β1, . . . , βC, γ1, . . . , γC, and µ1, . . . , µC.
3 Relationship to Multinomial Logistic Regression Models
The paper relates PARM’s conditional category probabilities to a multinomial-logistic form based on similarity predictors and category-specific parameters.
- Conditional model: PARM expresses the conditional probability of category c for response k using βcTk(c), γcSk, and µc in the exponent.The denominator sums over the possible categories.
- Category comparison: The model compares categories through conditional probability ratios while retaining the similarity-based category-specific terms.The ratio contrasts the exponents for candidate categories c and b.
- Identification: With β0 = γ0 = µ0 = 0, the denominator includes the baseline category.This parameterization identifies the baseline category in the multinomial specification.
- Interpretation: The paper characterizes each PARM variable as having a multinomial logistic regression on similarities with the other variables.
4 Estimation
PARM is estimated by pseudo-likelihood optimization using embedding-based similarity features, optional power tuning, and class weighting for imbalanced categories.
- Estimation: Parameter estimation maximizes the pseudo-likelihood under conditional independence given the other variables.The optimization uses Newton-type methods and an analytically computed gradient.
- Implementation: The implementation generates sentence embeddings, computes pairwise cosine similarities scaled by 1/K, and constructs Ti(c) and Si features.L-BFGS-B minimizes the negative log pseudo-likelihood.
- Tuning: The similarity power p is tunable through grid search or bounded continuous optimization against a selected training-data metric.Available metrics include overall accuracy, Cohen’s kappa, and near-agreement rate.
- Tuning: The continuous search typically converges in 20-40 evaluations and can find a more precise optimum than a coarse grid.
- Adaptability: The preprocessing and tuning mechanism is intended to adapt PARM to different response distributions and scoring rubrics.The paper presents this adaptability as supporting reliability auditing across diverse settings.
- Class imbalance: Inverse-frequency class weights make each category’s pseudo-likelihood contribution approximately equal, improving minority-category prediction at a possible cost in overall accuracy.
5 Example Data Analysis
The PARM examples demonstrate strong reliability-auditing performance across balanced short-answer and more challenging, imbalanced essay-scoring tasks. Its predictions generally preserve rubric ordinality, while similarity transformation substantially improves performance in one essay-set comparison.
- Example datasets: The analyses evaluate PARM on a balanced three-category short-answer task and longer AERA essay responses scored on four categories.The short-answer dataset contains 750 responses; the AERA examples use four-point scoring rubrics and longer responses.
- Short-answer results: Most short-answer misclassifications were between adjacent score levels, preserving the ordinal structure of the scoring rubric.The near-agreement rate, defined as predictions within ±1 category, was perfect at 1.000.
- Implications: The examples support PARM as a reliability-auditing tool across balanced short-answer and imbalanced, longer constructed-response settings.The authors describe the framework as applicable to realistic, skewed scoring settings and educational assessment contexts.
- Model interpretation: Across the examples, increasing βc estimates indicate stronger reliance on semantic similarity among responses sharing the same score level.The reported pattern is interpreted as consistent with the ordinal nature of the scoring rubrics.
- AERA essay results: On AERA Essay 6, accuracy was 0.809 and near-agreement was 0.956 despite severe class imbalance.Macro-averaged precision, recall, and F1 were 0.457, 0.567, and 0.493, respectively, with Cohen’s kappa of 0.467.
6 Conclusions
The paper introduces PARM as a flexible, interpretable framework for multi-category reliability auditing that combines category-specific agreement indicators with LLM-derived semantic similarities. Across three- and four-category tasks, it achieved strong performance and preserved rubric ordinality, while extensions and preprocessing choices remain areas for further study.
- Model and estimation: PARM combines category-specific agreement indicators with embedding-based pairwise weights for multi-category scoring reliability.The model is estimated via pseudo-likelihood with balanced class weights and analytical gradients, enabling L-BFGS-B optimization.
- Model scope: The formulation accommodates any number of ordered or unordered rating categories and extends naturally to multiple raters and hierarchical rating designs.Multiple raters are represented as separate variables, while hierarchical designs can incorporate random effects for raters.
- Empirical results: Accuracy reached 0.971 for the plant-growth task and 0.476 for the AERA Essay 1 dataset, with misclassifications occurring predominantly between adjacent score levels.The AERA result remained informative despite the more challenging setting, while the adjacent-level pattern preserved the ordinal structure of the scoring rubric.
- Interpretability: Category-specific parameters were interpretable: βc represents agreement influence, while γc and µc capture semantic-similarity effects and category intercepts.In both examples, βc increased with higher categories, suggesting stronger semantic coherence among high-scoring responses sharing the same score.
- Preprocessing and future work: Min–max normalization followed by an optional power transformation allows similarity scaling to emphasize or downweight moderate similarities.The exponent p can be treated as a hyperparameter, but its effects on predictive performance and parameter interpretability warrant further investigation.
- Future directions: Future work will examine multiple-rater and hierarchical extensions, larger datasets, and tasks requiring fine-grained semantic distinctions.These directions define the current framework’s expansion beyond the demonstrated applications.