PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts

Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, Chris Kedzie

arXiv:2501.00274v1cs.CL

TL;DR

Automated text evaluation is attractive because manual assessment is expensive and difficult to scale, but raw LLM ratings may not align with human judgments. LLM-RUBRIC combines responses to multiple rubric questions with a feed-forward calibration network containing shared and judge-specific parameters. In dialogue evaluation, multidimensional calibration improves prediction of human overall-satisfaction judgments, while robustness beyond closely related domains remains unevaluated.

  • Problem

    Manual evaluation is expensive and difficult to scale, while LLM ratings leave open the problem of alignment with human judgments.

  • Method

    LLM-RUBRIC prompts an LLM with multiple rubric questions and calibrates the resulting distributions with a feed-forward network using shared and judge-specific parameters.

  • Results

    Combining response distributions across all questions improves RMSE and linear and rank-based correlations on held-out synthetic and real dialogues.

  • Takeaways & Limitations

    LLM-RUBRIC can predict individual human judges’ responses, including overall satisfaction, from multidimensional LLM evaluations.

  • Takeaways & Limitations

    Robustness to new test domains was not evaluated beyond a shift from synthetic to real dialogues on the same information-seeking Azure topics.

Abstract

from arXiv · show

This paper introduces a framework for the automated evaluation of natural language texts. A manually constructed rubric describes how to assess multiple dimensions of interest. To evaluate a text, a large language model (LLM) is prompted with each rubric question and produces a distribution over potential responses. The LLM predictions often fail to agree well with human judges -- indeed, the humans do not fully agree with one another. However, the multiple LLM distributions can be $\textit{combined}$ to $\textit{predict}$ each human judge's annotations on all questions, including a summary question that assesses overall quality or relevance. LLM-Rubric accomplishes this by training a small feed-forward neural network that includes both judge-specific and judge-independent parameters. When evaluating dialogue systems in a human-AI information-seeking task, we find that LLM-Rubric with 9 questions (assessing dimensions such as naturalness, conciseness, and citation quality) predicts human judges' assessment of overall user satisfaction, on a scale of 1--4, with RMS error $< 0.5$, a $2\times$ improvement over the uncalibrated baseline.

1 Introduction

LLM-assisted evaluation can reduce the cost and scaling burden of manual text assessment, but its ratings still need alignment with human judgments. LLM-RUBRIC addresses this by combining multidimensional LLM evaluations with judge-specific calibration for dialogue assessment.

  • Manual text evaluation is expensive, time-consuming, and affected by quality and reliability issues, motivating scalable automated assessment.
  • LLM ratings address time, scaling, and possibly cost constraints but leave open whether they align with human judgments.
  • Dialogue quality depends on competing factors including response correctness, citation accuracy and helpfulness, and response length and complexity.
  • LLM-RUBRIC prompts an LLM with multiple rubric questions, then aggregates and calibrates the resulting response distributions with a small feed-forward network trained to match individual human judges.
  • Using 8 additional criteria, LLM-RUBRIC improves evaluation metrics by over 2ˆ over the uncalibrated LLM, while personalized calibration alone still works poorly.
  • Automated assessment could support writing feedback, dialogue decisions, text matching, fact-checking, and training text generation.

2 The LLM-RUBRIC Method

LLM-RUBRIC uses multiple rubric questions to expose evaluation dimensions and calibrates their LLM-derived distributions to each human judge. Its architecture combines shared representations with judge-specific parameters and models full response distributions rather than only point predictions.

  • Human judges differ in preferred textual properties, how they combine preferences, and how they express assessments numerically, so judge-independent prediction cannot match all score distributions.
  • LLM-RUBRIC asks finer-grained questions about criteria such as naturalness, relevance, attribution, citation quality, and conciseness, alongside an overall-quality question Q0.
  • For each rubric question, the LLM produces probabilities over allowed responses, which a feed-forward calibration network maps into judge-specific predicted response distributions using information from all questions.
  • For L2 evaluation, the system decodes a predicted score as the mean of the calibrated response distribution, guaranteeing outputs between 1 and 4.
  • Modeling the full calibrated distribution slightly outperforms direct regression in pilot experiments and also provides predictive uncertainty through quantities such as entropy or variance.
  • The method assumes rubric-question responses are conditionally independent given the text, treats integer scores as interval-scaled, and notes that these assumptions may be relaxed or inappropriate.
  • The calibration network shares learned representations and parameters across judges while retaining judge-specific weights, and uses auxiliary rubric questions to regularize prediction of the main task Q0.

3 Data

The study evaluates dialogue assessment in English Azure information-seeking conversations using synthetic training data and real user interactions. Its datasets combine mined web resources, varied LLM-generated systems, and professional human annotations.

  • 3 Data: The data focus on English information-seeking dialogues in the enterprise Azure-help domain, where evaluation access is difficult because dialogue data may be proprietary or private.
  • 3.2 Synthetic Dialogue Generation: Three randomly assigned professional judges evaluated each synthetic dialogue, yielding 741 personalized evaluation points after quality checks.
  • 3.1 Mining Topics for RAG: Topics come from 2275 Azure-related search queries selected using satisfactory clicks from Bing logs, supporting diverse information-seeking conversations.
  • 3.1 Mining Topics for RAG: The retrieval corpus contains 23,243 filtered webpages mined from 37,982 clicked URLs, with pages converted into plain text documents averaging 1246 ˘ 1651 words.
  • 3.2 Synthetic Dialogue Generation: Five LLM-based systems, DS1–DS5, generate 250 synthetic dialogues across 50 topics with varying quality and different document access conditions.
  • 3.3 Real Dialogue Collection and Evaluation: Real evaluation collected 223 conversations from 13 judges interacting with DS1–DS3, whose assistants used no RAG, oracle RAG, or BM25 RAG.
  • 3.3 Real Dialogue Collection and Evaluation: In the real setting, the human playing the user also judged the dialogue, making that person particularly qualified to assess overall user satisfaction Q0.

4 Experiments

The experiments predict human judges’ overall dialogue-quality scores from synthetic training data and test both held-out synthetic and real conversations. They compare LLM-RUBRIC with uncalibrated, single-question, retrieval-based, and oracle baselines using error and correlation metrics.

  • Evaluation measures prediction of Q0 using RMSE, Pearson’s ρ, Spearman’s ρ, and Kendall’s τ on held-out synthetic and real conversations.
  • Synthetic evaluation uses 5-fold dialogue-level cross-validation on 741 dialogues, while real evaluation trains on all synthetic dialogues before testing on 223 real dialogues.
  • The baselines include random scoring, argmax and expected LLM Q0 predictions, calibrated LLM Q0 using only Q0, and FActScore.
  • LLM-RUBRIC’s headline comparison reports RMSE and correlations against human Q0 responses, with Table 1 marking statistically significant improvements using paired permutation tests.
  • The Oracle supplies the target judge’s answers to auxiliary questions as an approximate upper bound, while the Depersonalized Oracle bounds judge-independent performance.

5 Results

LLM-RUBRIC substantially improves prediction of human judges’ overall satisfaction compared with simple baselines, while direct LLM scoring and factuality-based scoring perform poorly.

  • LLM-RUBRIC roughly halves the constant baseline’s 0.82 RMSE and explains about 3/4 of the variance in human Q0 judgments.Its Q0 predictions have tolerably low error and correlate reasonably with human judges.
  • Direct LLM responses to Q0 perform worse than the constant baseline, while judge-specific calibration alone explains only 5–10% of variance and achieves about 0.2 correlation.These results motivate using finer-grained rubric dimensions to assess overall satisfaction.
  • FActScore does not correlate better with overall satisfaction, indicating that the percentage of factually correct sentences is not a strong Q0 indicator.

6 Analysis

The analysis examines calibration, rubric dimensions, architectural choices, and oracle inputs, showing that multidimensional and personalized signals are central to prediction quality.

  • Calibration: LLM-RUBRIC achieves smECE values below 0.05 for each Q0 response value on synthetic data, indicating well-calibrated predicted probabilities.
  • Ablation Studies: Each calibration-network design step improves real-conversation predictions, with personalized weighting having the greatest impact.Multi-task pre-training followed by fine-tuning is also useful.
  • Ablation Studies: All rubric dimensions except Redundancy contribute significantly to Q0 prediction, while adding dimensions increases annotation and LLM-computation costs.
  • Oracle study: Oracle access to true Q1–Q8 responses reaches 0.72 Pearson’s ρ on real-dialogue satisfaction, with almost all performance obtainable from judge responses alone.
  • Oracle study: Better LLM prediction of judge-specific or judge-independent rubric responses is identified as a future route for improving LLM-RUBRIC inputs.
  • Analysis of rubric dimensions: LLM-RUBRIC improves RMSE by more than 0.5 on Redundancy, Conciseness, Efficiency, and overall Satisfaction, where zero-shot LLM correlations are close to zero.

7 Related Work

Prior work establishes LLM evaluation, calibration, and subjectivity-aware evaluation, while LLM-RUBRIC combines multidimensional responses with judge-specific modeling to predict individual judgments.

  • LLM Evaluation: LLM evaluators often align better with humans than lexical-overlap or earlier embedding metrics and increasingly assess dimensions such as factuality, interpretability, and relevance.
  • Calibration of LLM evaluators: Unlike calibration methods that assume specific ground-truth answers, LLM-RUBRIC models distributions over reasonable responses.Its focus is combining attributes to predict all judgments, especially overall quality, for a given judge.
  • Subjectivity in Evaluation: Because expert disagreement is common, subjectivity-aware evaluation treats disagreement as signal rather than noise and models judge distributions.
  • Subjectivity in Evaluation: LLM-RUBRIC extends judge-specific modeling by using auxiliary questions Q1–Q8 alongside the main Q0 task.
  • Subjectivity in Evaluation: Prior measurement work suggests judges may have idiosyncratic or conflated mappings across dimensions, an analysis LLM-RUBRIC does not perform.The authors regard such improvements as orthogonal to benefits from modeling reasonable judge disagreement.

8 Conclusions

The paper presents LLM-RUBRIC as a multidimensional, judge-calibrated framework for automatic text evaluation. Combining LLM response distributions predicts individual human judgments, including overall satisfaction, across held-out synthetic and real dialogues.

  • LLM-RUBRIC performs multidimensional evaluation with a black-box LLM and calibrates the aggregated responses for each human judge.
  • Combining response distributions across rubric questions predicts each judge’s responses, including overall satisfaction, despite weak raw LLM–human correlation.
  • The method improves RMSE and both linear and rank-based correlations on held-out synthetic and real conversations.

Limitations

The paper identifies limits in robustness, efficiency, downstream validation, fine-grained evaluation, and proposed generation uses. These boundaries constrain where LLM-RUBRIC has been demonstrated and how broadly it should be deployed.

  • The method was not evaluated across new domains, beyond a modest shift from synthetic to real dialogues on the same information-seeking topics.The authors note that privacy constraints, limited judge competence, and changing test distributions make robustness especially important.
  • Efficiency: LLM-RUBRIC uses a large pretrained LLM for every rubric question, making efficiency a concern that adaptive rubrics or smaller models might address.The proposed adaptive strategy would select questions by expected information gain and stop when returns diminish, but was not tested.
  • A multilingual system may expose rubric gaps when language-specific quality problems were absent, underweighted, or unassessed during training.
  • Downstream Evaluation: The paper did not test whether predicted scores support difficult downstream choices, and more rubric questions may be needed for sufficient accuracy.
  • The authors have not addressed using LLM-RUBRIC to improve natural-language generation, although predicted quality could serve as a reward signal.
  • Fine-Grained Evaluation: The study evaluates entire texts rather than automating or calibrating finer-grained judgments about problematic or relevant spans.

Ethics Statement

The ethics discussion emphasizes that calibrated agreement with human judges does not establish fairness, safety, or objective ground truth. It also describes privacy, aggregation, and monitoring considerations for deploying judge-specific predictions.

  • Ethics Statement: Evaluating dialogue quality should include harms to users, system owners, and third parties rather than treating overall user satisfaction as the sole goal.
  • Ethics Statement: Matching human judges does not establish fairness: accuracy, calibration, and rubric validity should be audited across important data subsets.The paper notes that errors may remain unevenly distributed even when total error is low.
  • Ethics Statement: Human-judge data reveal personal preferences and should be safeguarded; differential privacy may sometimes be appropriate for training the calibration network.
  • Ethics Statement: LLM-RUBRIC may optimize outputs for an individual's preferences or engagement in harmful ways, including misinformation, outrage, fraud, or self-harm.
  • Ethics Statement: Repeated LLM calls make the method compute-intensive at scale, increasing environmental costs for high traffic, tuning loops, or reinforcement-learning rewards.Distillation or an adaptive rubric could reduce these costs.
  • A Aggregating Predicted Scores: Judge-specific predictions model individual idiosyncrasies rather than ground truth, so final scores require an explicit aggregation choice such as a trusted-judge mean or a conservative maximum.A mean assumes an interval-scale overall question and equal concern for the selected judges.
  • A Aggregating Predicted Scores: Aggregated scores can hide disagreement, whereas disaggregated views expose dialogues predicted to be problematic for particular judges.

B Handling Other Types of Datasets

This section details how LLM-RUBRIC handles additional dataset types and evaluation settings, including irregular data, heterogeneous and comparative responses, and synthetic dialogues. It specifies the rubric questions, prompts, judge screening, generation approaches, and response-scale considerations.

  • B Handling Other Types of Datasets: LLM-RUBRIC can incorporate embeddings, additional LLM questions, internal LLM states, manually extracted features, judge attributes, and metadata, while masking metadata for fair cross-domain comparisons.Embeddings may be pretrained or jointly fine-tuned; high-dimensional embeddings can be introduced after initial training to reduce overfitting.
  • B Handling Other Types of Datasets: The framework can handle missing or evolving features through dropout and by predicting missing LLM responses to select subsequent questions using expected information gain.This supports adaptive rubrics that ask only as many questions as needed and accommodates feature sets that change when the underlying LLM is replaced.
  • B Handling Other Types of Datasets: Training can use irregular collections of text-question-judge tuples, with retraining on growing heterogeneous data, question and judge routing, active learning, and end users as judges.Questions may be filtered by applicability and routed according to topic, dialect, user type, or specialized policy expertise; annotation can prioritize high-variance predictions.
  • B Handling Other Types of Datasets: The output model can support real-valued, textual, large finite, and ordinal responses through density models, autoregressive language models, response embeddings, or latent continuous values.Comparative questions can be modeled directly as yes-probabilities or by integrating over latent response distributions.
  • C LLM-RUBRIC Questions: The evaluation rubric includes questions about naturalness, reference-based answerability, citation coverage, citation accuracy, and whether cited sources are the best available.Questions with no applicable references receive “NA” and are excluded from training and evaluation.
  • C LLM-RUBRIC Questions: The rubric includes questions on redundancy, conciseness, exchange-turn appropriateness, and overall user satisfaction.Q8 asks whether the number of back-and-forth exchanges matches the information need’s complexity, while Q0 rates overall satisfaction.
  • D Evaluation Prompt for LLM: The LLM evaluates each conversational text with a rubric question using a prompt containing the dialogue and question, while omitted citation questions receive an NA probability vector when references are absent.The prompt instructs the model to output only one response label, 1 through 4.
  • D Evaluation Prompt for LLM: Q8’s numeric responses are not ordinal, so using L2 for decoding and RMSE evaluation affects only Q8 metrics; swapping labels 2 and 3 would presumably improve that line without affecting other results.The issue arises because response 3 is reasonably closer to 1 than to 2.
  • E Evaluation Prompt and Preliminary Data Quality Questions for Humans: Human judges rate naturalness, citation quality, and overall user satisfaction, while preliminary DQQ0–DQQ2 questions screen generated synthetic dialogues and are omitted when judging real dialogues.If the answer to DQQ is “No,” the remaining questions are skipped, leaving 741 rather than 750 synthetic training examples.
  • F Synthetic Dialogue Generation: Synthetic dialogues are generated with five approaches ranging from an internal-knowledge assistant without citations to retrieval-augmented systems using oracle documents, BM25 retrieval, dialogue-history queries, or LLM-generated queries.DS2 uses the five most-clicked documents with an unrealistically knowledgeable simulated user, whereas DS3–DS5 progressively use BM25 retrieval with different query and prompting strategies.

G Quality of the Generated Synthetic Dialogues

The study evaluates synthetic and real human-agent dialogues across multiple rubric dimensions, using synthetic conversations for training and live interactions for final evaluation.

  • G Quality of the Generated Synthetic Dialogues: DS2 and DS3 averaged at least 3.0 across all questions except Q8, indicating generally strong simulated dialogues.The synthetic dialogues support calibration training and development rather than serving as the study’s primary evaluation target.
  • G Quality of the Generated Synthetic Dialogues: DS2 achieved the highest citation ratings, while DS1 produced the most natural conversations and DS1 performed worst on overall satisfaction.DS2’s citation advantage was attributed to its oracle RAG, whereas DS1’s overall-satisfaction disadvantage was associated with retrieved-document quality.
  • H The User Interface for Human-Agent Dialogue Collection and Evaluation: The interface randomly assigned Azure-related topics and dialogue systems, then let users end conversations and evaluate their interactions.The collection procedure concealed system identity from users, although DS1 could be distinguishable because it provided no citations.
  • I Evaluating the Collected Human-Agent Dialogues: Thirteen trained judges contributed 223 real conversations, with each judge evaluating between 14 and 27 conversations.Scores for the three evaluated dialogue systems were summarized in Table 5.

J How much human judge data is needed to train calibration?

Calibration was trained on random fractions of synthetic data and tested on real conversations, with performance improving as per-judge data increased. Performance was already reasonable with 20% of training data and essentially converged by 80–100%, while calibration plots showed low error.

  • J How much human judge data is needed to train calibration?: The learning-curve experiment trained on synthetic data, tested on real conversations, and averaged results over 50 random samples with one-standard-deviation variability.Increasing the training fraction improved average performance and reduced variance.
  • J How much human judge data is needed to train calibration?: Performance was reasonably good with 20% of the training set and essentially converged by 80–100%.At 100%, the training set contained 741 dialogues, with each judge evaluating only about 30 on average.
  • J How much human judge data is needed to train calibration?: Calibration plots for Q0 on held-out synthetic dialogues showed low calibration error.The plots examine predicted responses for y0 values 1, 2, 3, and 4.

K Calibration Plots (Reliability Diagrams)

The calibration plots show that LLM-RUBRIC’s predicted probabilities closely track observed judgment frequencies, supporting meaningful uncertainty estimates for downstream decisions. The evaluation also notes that whole-dataset calibration does not guarantee calibration within underrepresented subsets.

  • Calibration result: The red calibration curves stay close to the diagonal, indicating that the system is well-calibrated on held-out synthetic dialogues.Calibration is measured with smoothed expected calibration error (smECE).
  • Plot interpretation: The reliability diagrams compare predicted probabilities with observed outcomes for each possible score, using tick marks to show held-out examples and red curves to show empirical frequencies.The gray band represents a 95% bootstrap confidence interval.
  • Calibration metric: smECE summarizes the average absolute gap between predicted and actual probabilities, weighted by the density of examples.Each graph reports smECE with a 95% confidence interval.
  • Downstream uses: Meaningful calibrated probabilities let the system assess its uncertainty and use it for selection, dynamic question choice, distillation, and rubric improvement.The proposed uses include choosing high-reward responses, selecting questions by information gain, training faster scorers, and identifying difficult or controversial texts.
  • Caveat: Calibration plots summarize the dataset as a whole, so calibration within user categories, judges, or dialogue topics still requires sufficient representation and separate checking.The authors report that predicted scores nevertheless provide a reasonable ranking of examples.
Loading 2501.00274v1…