Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge

1 INTRODUCTION

• Automatic evaluation includes N-gram and task-specific model methods, as well as LLM-as-a-Judge systems prompted to evaluate.Common comparisons use agreement or rank-correlation metrics, with higher machine-human correlation treated as evidence of greater effectiveness.Source: §1 · p. 1
• A single aggregate correlation score can obscure discrepancies between machine labels and human labels, motivating measurements stratified by label uncertainty.The paper frames this as a way to compare automatic evaluation more robustly.Source: §1 · p. 2
• Human-label variation is difficult to avoid, and whether it is acceptable depends on the task; perception-based judgments can vary more than NLI judgments.The paper notes that uncertainty may be a feature rather than a bug in perception-based evaluation.Source: §1 · p. 2
• The paper asks how uncertainty affects correlation metrics, how to measure agreement under perceptual variation, and how to visualize machine-human evaluation comparisons.These questions correspond to stratification, binned Jensen-Shannon divergence, and perception charts.Source: §1 · p. 2

2.1 CORRELATION MEASUREMENTS AND ASSUMPTIONS

• Randolph’s-κ assumes equally likely labels, while Fleiss’s-κ and Krippendorff’s-α estimate chance agreement from observed labels; these human-rater metrics assume interchangeable raters.Krippendorff’s-α also presumes duplicated rating processes and no known correct answers, conditions the paper says do not fit machine-human comparisons.Source: §2.1 · p. 2

2.2 DIFFERENCES BETWEEN CORRELATION MEASURES AND RAW ACCURACY

• Accuracy and F1 do not account for variation among noisy human labels, while common chance-adjusted agreement metrics target random human errors rather than systematic machine errors.The paper notes that IRA metrics can treat agreement as good and variance as bad, even when items are ambiguous or perception-based.Source: §2.2 · pp. 2–3

2.3 TYPES OF RAW DATA COLLECTED DURING EVALUATION

• Nominal evaluation includes binary fact-checking and model preferences, whereas ordinal Likert ratings encode ordered judgments such as quality.Ordinal agreement measures can account for distance between ratings; rank correlations may be difficult to interpret when many items are tied.Source: §2.3 · p. 3
• Continuous scores are typically produced by automatic evaluation methods such as Rouge and BERTScore.Source: §2.3 · p. 3

3 ANALYSIS SETTINGS AND RESULTS

• The analysis covers Likert ratings, NLI labels, and model preferences, using datasets with multiple human annotations and automatic evaluator labels.Uncertainty is primarily stratified by percentage agreement and also by the number of unique human labels; prompts were not optimized to rank the underlying judges.Source: §3 · pp. 3–4, §3.1 · p. 4

3.1 RQ1: HOW DOES UNCERTAINTY IN HUMAN LABELS IMPACT CORRELATION METRICS WHEN WE MEASURE THE EFFICACY OF AUTOMATIC EVALUATION METHODS?

• Overall HM–HH correlation can look similar, but the gap widens under perfect human agreement; at low certainty, HM can exceed HH.The same uncertainty-linked pattern is reported for ordinal and preference evaluations.Source: §3.1 · p. 4, Table 1 · p. 5, Figure 1 · p. 5

3.2 RQ2: HOW CAN WE MEASURE HUMAN-MACHINE AGREEMENT THAT ACCOUNTS FOR HUMAN UNCERTAINTY AS A RESULT OF VARIATION IN HUMAN PERCEPTION?

• Across ordinal and preference evaluations, high uncertainty can make HM seem stronger than HH, and evaluator rankings can change with the human reference.For MT-Bench preference data, the table reports differing IRA and win-rate comparisons between evaluator models.Source: Table 2 · p. 6, Table 3 · p. 7
• Binned JSD compares human and machine rating distributions within median- or majority-defined bins without requiring a single gold label.The authors note that infrequent item-level mismatches may remain hidden in the aggregate, so they also recommend uncertainty-stratified correlations.Source: §3.2 · pp. 6–7

3.3.1 PERCEPTION-BASED ORDINAL RATING VISUALIZATION

• For Mistral on Coherence, machine ratings omit 1 and rarely assign 5, although humans give over 24% of items a median rating of 5.The ordinal chart bins by human median and plots all human judgments alongside corresponding machine ratings.Source: §3.3.1 · p. 8, Figure 3 · p. 8

3.3.2 PERCEPTION-BASED PREFERENCE VISUALIZATION

• The pairwise preference chart bins items by human majority and compares human and machine preference distributions for each model pair.For Claude-v1 versus GPT-3.5-turbo, the reported overall human preference for Claude is 38.9%, while items with a human majority for GPT-3.5 show almost no contention.Source: §3.3.2 · pp. 8–9, Figure 4 · p. 9

4.1 HUMANS ARE UNCERTAIN AND THEIR LABELS ARE NOISY, SO MACHINES ARE BETTER?!

• A single gold label is adequate only when ambiguity is little or absent; over 40% of MNLI samples with five human labels show some uncertainty.Some recent datasets collect only one human label per item, while the authors argue that quantifying uncertainty is essential for evaluating automated methods.Source: §4.1 · p. 9
• Under high human-label uncertainty, even a random labeler can appear to match or outperform human-to-human correlation, while machine gaps emerge when human labels are highly certain.Uncertainty can also change evaluator rankings and correlation scores depending on the human reference; in safety-critical settings, machine disagreement with robust human labels may indicate serious deficiencies.Source: §4.1 · p. 9

4.2 CHALLENGES IN METRICS AND INTERPRETABILITY

• Inter-rater agreement metrics were designed around typical human behavior, whereas LLM errors can be reproducible and systematic yet unpredictable, complicating machine–human comparison.The paper cautions that metric assumptions should fit the data and study aim.Source: §4.2 · p. 10
• Metric suitability depends on the task: Randolph’s-κ may fit near-unanimous pairwise preferences, while Krippendorff’s-α and Fleiss-κ may better capture consequential minority ratings in ordinal perception tasks.When one majority label cannot represent preferences, the authors identify binned JSD for perception as a possible alternative.Source: §4.2 · p. 10
• Significance testing should account for sample variation, human-label uncertainty, and the difficult-to-estimate error rate of systematically unpredictable automated labels.The authors leave investigation of these challenges, including resampling approaches, to future work.Source: §4.2 · p. 10
• Perception charts trade plotting every data point against aggregation, aiming to show how uncertainty varies across labels and affects interpretation of machine performance.The charts are presented as a step toward making aggregate correlation more interpretable.Source: §4.2 · p. 10

4.3 RECOMMENDATIONS FOR REPORTING EFFECTIVENESS OF AUTOMATIC METHODS

• The authors recommend stratifying by uncertainty, reporting multiple metrics from different families, and visualizing results because no single aggregate metric captures all relevant data patterns.The proposed perception charts aim to make aggregate correlation more interpretable and reveal automatic labeler strengths and gaps.Source: §4.3 · p. 10

• The prompt presents a premise and a hypothesis and asks whether the hypothesis is entailed by the premise.Source: §A.1 · p. 15
• It offers three labels: entailment, insufficient information to determine truth, and logical contradiction.Source: §A.1 · p. 15
• The evaluator is instructed to select the correct answer label and provide no reasons.Source: §A.1 · p. 15

• The prompts for SummEval are reused from Liu et al. (2023b) and are used in G-Eval.Source: §A.2 · p. 15
• Coherence is rated from 1 to 5 for the summary’s collective quality, including organization and logical ordering of key points.Source: COHERENCE · p. 15
• Consistency is rated from 1 to 5 for factual alignment with the source, with annotators instructed to penalize unsupported facts.Source: CONSISTENCY · p. 16
• Fluency is rated from 1 to 3 based on grammar, spelling, punctuation, word choice, and sentence structure.Source: FLUENCY · p. 16
• Relevance is rated from 1 to 5 for selecting important source content, while penalizing redundancy and excess information.Source: RELEVANCE · pp. 16–17

• The evaluator sees two user–assistant conversations, labeled A and B, and judges which better follows instructions and answers the user.Source: §A.3 · p. 17
• The prompt defines A and B as choices for the better response and C for responses that are equally good or poor.Source: §A.3 · p. 17
• Evaluators consider helpfulness, relevance, accuracy, depth, creativity, and detail, avoid positional and length bias, and return only a choice label.Source: §A.3 · p. 17

A.4 PERCENTAGE AGREEMENT

• For n items rated by R raters across C categories, percentage agreement summarizes the most frequent label per item.The measure uses the maximum percentage of votes received by the most frequent label, provided it receives more than one vote.Source: §A.4 · p. 17
• The measure counts only items whose most frequent label receives more than one vote.Items without a label receiving multiple votes do not contribute to the defined maximum-percentage calculation.Source: §A.4 · p. 17
• The annotation x_i^r denotes the label assigned to item x_i by the rth rater.The indicator function 1_A returns 1 when condition A is true and 0 otherwise.Source: §A.4 · p. 17

A.5 IMPACT OF LABEL UNCERTAINTY ON ORDINAL SUMMEVAL DATASET

• The simulation creates subsets of 100 samples with predefined proportions of perfect human-label agreement.Samples are randomly selected from the dataset according to each subset’s specified proportion.Source: §A.5 · p. 18
• Figure 5 presents the impact of noise on the ordinal SummEval dataset.The supplied caption identifies the topic but provides no specific plotted outcomes.Source: Figure 5 · p. 18

A.6 VISUALIZATION OF ROUGE-SCORE CORRELATION

• Figure 6 contrasts Rouge-1 and LJ (Mistral) against human median relevance ratings, with continuous Rouge scores and integer human and LJ scores from 1 to 5.The figure presents Rouge-1 on the left and LJ (Mistral) on the right.Source: Figure 6 · p. 18

A.7 CODE FOR BINNED-JSD-PERCEPTION

• Binned-JSD uses distributions of human and machine labels within each human-label bin and weights each bin’s JSD by its sample share.The worked example computes 2/3 * 0.31 + 1/3 * 0.56 = 0.39.Source: §A.7 · p. 19

A.8 TOY EXAMPLE: APPLICATION OF BINNED-JSD AND WHERE IT CAN A BETTER METRIC

• In the toy example, binned-JSD scores the model choosing human-selected values better than one choosing a different value, while single-label correlations can favor the poor model.The example warns that metric advantages and shortcomings are difficult to identify without knowing which model is better; the authors recommend stratification, visualization, and multi-metric reporting.Source: §A.8 · pp. 20–21, Table 6 · p. 21

A.9 VISUALIZATION OF PREFERENCE DATA

• The preference perception chart compares human preference distributions with GPT-4 and Claude Sonnet 3 judgments for two model pairs, with ties marked T.Each pair is shown for a given majority label.Source: Figure 7 · p. 22

A.10 EXTENDED TABLE RESULTS

• The MTBench table reports that higher proportions of noisy or uncertain samples generally coincide with human-to-machine correlation outperforming human-to-human correlation, with exceptions in partitions of five or fewer samples.MTBench items have between three and five human preference annotations, so possible percentage-agreement strata depend on the number of annotations.Source: Table 8 · p. 23

A.11.1 RANDOM SIMULATION CODE - BINARY CLASSIFICATION (NOMINAL DATA)

• The random simulations create synthetic binary and three-way classification datasets with two human annotators and a random labeller; for the three-way task, labels are [1,2 or 3].Source: §A.11.1 · p. 24, §A.11.2 · p. 24

A.12.1 ANALYSIS OF RESULTS

• With more uncertain samples, human-to-machine correlation can exceed human-to-human correlation, while it falls below human-to-human correlation as human labels become nearly consistent.The Topical Chat and QAGS aggregate Randolph-κ gap is 19 points (0.93 → 0.74), shrinking to 6 points (0.93 → 0.87) among perfectly agreed samples; DICES performance remains poor across metrics and strata.Source: §A.12.1 · pp. 25–26, Table 10 · p. 25

A.12.2 ANALYSIS OF ORDINAL DATA

• In the ordinal Topical Chat criteria, label imbalance and low agreement within the dominant human-median bin help explain low correlation scores, while Overall labels show a machine-human difference on most samples.For Engaging, over 96% of human labels have median 3 and only 3.3% have median 2; the Overall criterion’s H=4 bin covers 8.3% of samples, while human uncertainty is low for the remaining 91%.Source: §A.12.2 · pp. 26–27, Table 11 · p. 26, Figure 11 · p. 28

A.13 EXAMPLES COMPARING HUMANS AND MACHINE LABELS

• The SNLI examples include cases where the model disagrees with a unanimous human label and cases where human disagreement accompanies a different machine majority.Tables 12 and 13 present these two kinds of examples, respectively.Source: §A.13 · p. 29, Table 12 · p. 29, Table 13 · p. 30