PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-hoc Alignment of LLM-judges to Human Judgment Distribution

Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani

arXiv:2609.01073v1cs.CL

TL;DR

Current LLMaJ evaluation often reduces human annotations to aggregated labels, leaving Human Label Variation and human judgment distributions underused. This paper evaluates hard- and soft-label prediction across five datasets and introduces NAPHA, an entropy-aware post-hoc alignment method. LLMaJ is near human level on hard labels but poor on soft labels, while NAPHA consistently improves soft-label prediction, with especially strong gains on high-entropy instances.

  • Problem

    Aggregating human annotations overlooks Human Label Variation, so the paper studies whether LLMaJ can predict both hard labels and soft labels representing Human Judgment Distributions.

  • Method

    NAPHA assigns each base LLMaJ soft-label prediction to a discrete entropy class and routes it to a specialized trained alignment model.

  • Results

    LLMaJ performs near human level on hard-label prediction but poorly on soft-label prediction, while NAPHA consistently improves soft-label prediction across models and datasets.

  • Takeaways & Limitations

    NAPHA is applicable when different human perspectives can make multiple labels simultaneously valid, especially in subjective tasks.

  • Takeaways & Limitations

    Three of five datasets have sparse soft labels because they use fewer than six annotators, and NAPHA’s routing may conflate model uncertainty with predicted human disagreement.

Abstract

from arXiv · show

The LLM-as-a-judge (LLMaJ) framework offers a cost-effective and reproducible solution for automatic evaluation. However, current evaluation practices typically compare LLMaJ judgments against aggregated ground-truth labels, overlooking the valuable information contained in Human Label Variation (HLV). Inspired by an increasing line of work that proposes to leverage HLV, we systematically study LLMaJ performance on predicting both a single, aggregated ground truth hard-label and unaggregated soft-labels that represent Human Judgment Distributions (HJD). Our results across five diverse datasets reveal that while LLMs achieve near human-level performance at hard-label prediction on most tasks, they exhibit poor performance when predicting soft-labels. To address this limitation, we propose NAPHA (eNtropy-Aware Post-Hoc Alignment), a simple yet effective lightweight post-hoc alignment method that matches the LLM distribution to the HJD by first assigning an instance to a discrete entropy class and then routing it to specialized, trained alignment models. We find that NAPHA consistently improves soft-labels prediction across base LLM models and datasets, with particularly strong gains on high-entropy instances where capturing diverse human perspectives is most critical. We also show via oracle experiments that improving entropy class prediction can substantially enhance NAPHA's practical effectiveness.

1 Introduction

The paper argues that aggregating annotations into hard labels discards meaningful Human Label Variation (HLV), motivating evaluation of LLMaJ predictions against both hard labels and human judgment distributions. Across five datasets and multiple state-of-the-art LLMs, LLMaJ performs near human level on hard-label prediction but poorly on soft-label prediction, while NAPHA improves soft-label prediction across models and datasets.

  • Motivation: Aggregating multiple annotations into a single label eliminates annotator disagreement, although recent HLV work treats differing labels as potentially simultaneously correct.HLV can reflect annotator backgrounds, task ambiguity, uncertainty, and irreconcilable variation.
  • Research focus: The study evaluates whether state-of-the-art LLMaJ models can predict both aggregated hard labels and soft labels representing Human Judgment Distributions.The evaluation spans five datasets covering typical NLG and classification tasks.
  • Findings: LLMaJ achieves near human-level hard-label performance on some tasks but performs poorly on soft-label prediction across the evaluated datasets and models.The paper explicitly validates this hard-label versus soft-label contrast using multiple datasets and state-of-the-art LLMs.
  • Proposed approach: NAPHA is a lightweight post-hoc alignment method that assigns instances to entropy classes and routes predictions to specialized alignment models.The method is designed to improve alignment between LLMaJ soft-label predictions and human judgment distributions.
  • Contribution: NAPHA consistently improves soft-label prediction across base models and datasets, especially where multiple human perspectives may make several labels simultaneously valid.The paper presents NAPHA as particularly relevant to subjective tasks and settings that retain disagreement as useful information.

2 Background and Related Work

The background motivates automatic evaluation through inexpensive, reproducible LLMaJ judgments and frames HLV as a pluralistic alternative to assuming one correct label. It also distinguishes the paper’s post-hoc alignment from conventional calibration and notes that valid variation can be difficult to separate from annotation noise.

  • Automatic evaluation: Automatic evaluation seeks to reduce annotation cost and improve reproducibility, while LLMaJ has shown better alignment with human annotations than earlier metrics.The background contrasts prompt-engineered LLM judges with metrics such as ROUGE and BERTScore.
  • Human Label Variation: HLV and perspectivism accept irreconcilable annotation disagreement and seek to leverage it through soft-label distributions.The paper places its study within distributionally pluralistic model alignment.
  • Caveat: Annotation errors can still produce label noise, and the paper does not disentangle such errors from valid human variation.For its experiments, the paper treats all HLV in the datasets as valid.
  • Related methods: The paper distinguishes post-hoc alignment of LLMaJ predictions to HJD from calibration methods that assume access to the model architecture.Related calibration work includes temperature-based and other parameterized approaches.

3 Problem Statement

The paper formalizes two evaluation settings over human annotations: prediction of an aggregated hard label and prediction of a categorical soft-label distribution representing human judgment variation. Hard labels are scored with F1 or correlation, whereas soft labels are compared with a distributional distance.

  • Setup: Each task has n possible discrete labels, and each dataset instance contains m human annotations drawn from that label set.The formalization represents the annotation tuple as r = (r_1, ..., r_m).
  • Hard-label evaluation: In the hard-label setting, an aggregation function such as majority voting or the mean combines human annotations into a ground-truth label.A predicted hard label is evaluated against this ground truth across instances.
  • Soft-label evaluation: In the HLV setting, the ground truth is a categorical distribution over possible labels estimated by maximum likelihood from the observed human annotations.This distribution is the soft-label representation of human judgment variation.
  • Soft-label evaluation: Predicted soft labels are evaluated against the ground-truth soft labels using a distance metric dist(ŷ, y).This setting is the focus of the work.

4 Predicting soft-labels

The paper compares several base approaches for predicting soft labels and proposes NAPHA, which uses entropy-based routing to apply specialized post-hoc alignment models. NAPHA requires only model output tokens, adds lightweight transformations, and improves performance particularly for high-entropy predictions, although entropy-class prediction remains difficult for LLMs.

  • Base models: The base soft-label models include SimAnn sampling and direct prompting with either hard-label or soft-label in-context examples.SimAnn uses ten simulated annotators, temperature t = 1, and different in-context examples across runs.
  • Motivation: LLMs perform poorly at soft-label prediction independently of the base model, motivating NAPHA as a post-hoc alignment method.NAPHA operates on top of a base LLMaJ model rather than replacing it.
  • NAPHA pipeline: NAPHA computes prediction entropy, bins instances into low, medium, and high entropy classes using dataset entropy terciles, and routes each prediction to a class-specific alignment model.The method does not require access to LLM internals and uses only output tokens.
  • Alignment models: Each alignment model applies a nonlinear transformation to predicted soft labels and is trained to better match ground-truth soft labels.The alignment models use a lightweight single-hidden-layer MLP with ReLU and softmax outputs.

5 Experimental Setup

The study evaluates LLMaJ predictions against human judgment distributions across five datasets, using entropy-stratified sampling, hard- and soft-label metrics, and multiple LLM configurations.

  • Datasets and sampling: Five datasets cover subjective NLG and classification tasks with score-based or selection-based outputs and multiple human annotations per sample.The datasets include varied sources of Human Label Variation.
  • Datasets and sampling: 1,500 examples each are used for DynaSent and Anecdotes, with 500 examples sampled from each low-, medium-, and high-disagreement quantile.Disagreement is measured using binary entropy across soft-labels.
  • Evaluation metrics: Hard-label evaluation uses macro-average F1 for majority labels and Kendall’s τ and Advantage Probability for average ratings.Human performance is estimated through sampling or leave-one-out bootstrapping, depending on the task.
  • Evaluation metrics: Soft-label evaluation uses Distribution Calibration Error and Jensen-Shannon Distance to measure discrepancies between human and predicted distributions.DistCE is the maximum probability discrepancy over possible events.

6 Results and Discussion

LLMaJ models perform near human level on many hard-label tasks but poorly predict soft-label distributions. NAPHA improves soft-label alignment across models and datasets, especially at high entropy, while routing accuracy limits gains in some strata.

  • Hard-label prediction: 0.99, 0.81, and 0.61 are Claude-4’s F1 scores on ChaosNLI at low, medium, and high entropy, versus human scores of 0.93, 0.74, and 0.53.On Anecdotes, the corresponding LLM scores are 0.48, 0.48, and 0.34 versus human scores of 0.50, 0.42, and 0.35.
  • Hard-label prediction: 0.480 versus 0.542 is the average LLM versus human Kendall’s τ on SummEval.The passage attributes the remaining gap partly to minimized annotator disagreement in SummEval.
  • Overall soft-label alignment: 0.299 to 0.272, 0.200 to 0.174, and 0.272 to 0.265 are SLP-SE DistCE reductions on Anecdotes, ChaosNLI, and DynaSent after NAPHA.Lower DistCE is better, and JSD changes in the same direction.
  • Oracle routing: 0.172, 0.153, and 0.172 are oracle-routing DistCE values on Anecdotes, ChaosNLI, and DynaSent, substantially improving over predicted entropy routing.Other tested alignment models do not outperform NAPHA.
  • Entropy-stratified performance: NAPHA consistently improves high-entropy instances, where diverse valid human perspectives make soft-label alignment particularly important.Base-model soft-label prediction worsens as entropy increases.
  • Entropy-stratified performance: Low-entropy alignment becomes worse with NAPHA, which the authors attribute to incorrect entropy routing; omitting NAPHA for that stratum is suggested as a mitigation.The degradation disappears when oracle entropy labels are used.
  • Data requirements: 10% of the training data is sufficient for NAPHA performance to stabilize.This indicates that the alignment method can improve alignment without requiring many human labels.

7 Conclusion and Future Work

The study finds that NAPHA consistently improves LLMaJ soft-label prediction across base models and datasets, with substantial gains for high-entropy instances. Future work could extend NAPHA to model internals and advance HLV-aware model development and data collection.

  • Conclusion: NAPHA consistently improves soft-label prediction across all evaluated base models and datasets.The method is designed to align LLMaJ predictions with human judgment distributions.
  • Conclusion: NAPHA provides substantial gains on high-entropy instances, where capturing diverse human perspectives is most critical.
  • Conclusion: Improving entropy-class assignment increases NAPHA’s effectiveness and practical applicability in oracle experiments.
  • Future Work: NAPHA could be extended to white-box settings by leveraging logits across possible labels.
  • Future Work: Future HLV research should improve model training for pluralism and collect explanations, confidence indications, and sociodemographic metadata.These data may help distinguish annotation error from valid human label variation and improve soft-label prediction.

Limitations

The study’s limitations concern sparse soft-label distributions, routing uncertainty, non-natural entropy sampling for two subsets, limited human-performance estimates, and unoptimized alignment models. Its theoretical framing formalizes alignment as entropy-stratified optimization.

  • Limitations: Soft-labels are sparse for three of the five datasets because they contain fewer than six annotators.The authors note that large-annotator datasets are rare and expensive, and the required annotator count for fair estimation remains unclear.
  • Limitations: NAPHA routes instances using predicted label distributions, potentially conflating model uncertainty with predicted human disagreement.The current design cannot disentangle these two sources of uncertainty.
  • Limitations: Anecdotes and DynaSent subsets use balanced entropy classes rather than their natural entropy distributions.The other three datasets are used as-is, preserving their natural entropy distributions.
  • Limitations: The study does not tune hyperparameters or optimize the alignment-network architecture, instead demonstrating NAPHA with a lightweight approach.
  • Theoretical Formalization: NAPHA’s global expected alignment error is formalized as a probability-weighted sum of conditional errors across entropy strata.The framework partitions the input space into low-, medium-, and high-entropy subsets and routes each subset to a specialized model.
  • Evaluation Scope: Human-performance estimates have limited meaning on DynaSent, SummEval, and TopicalChat because those datasets contain relatively few annotators.

C Analysis of HJD: sources of disagreement in the data

The datasets represent different plausible sources of human label variation, including ambiguity, sarcasm, preferences, culturally influenced ethical understandings, and under-specified examples.

  • DynaSent: DynaSent may exhibit HLV through ambiguity or sarcasm that is difficult to detect in text.
  • SummEval and TopicalChat: SummEval and TopicalChat may reflect individual preferences, such as preferred summarization style.
  • Anecdotes: Anecdotes represents diverse, culturally influenced ethical understandings through ethical dilemmas.
  • ChaosNLI: ChaosNLI was selected for previously elicited HLV, which likely arises from ambiguous or under-specified examples despite its mainly logical task.

D Results per entropy class

The appendix reports per-entropy-class and per-dimension metrics, alongside an analysis of prediction entropy and entropy-class F1-scores. These materials support examining how performance varies across entropy strata.

  • Per-Entropy-Class Results: Tables 4 to 9 contain the full results broken down by entropy class or dimension.
  • Entropy Analysis: Figure 4 analyzes entropy across predictions with and without NAPHA.
  • Entropy-Class Prediction: Table 10 reports F1-scores for predicting the entropy class.

F Ablations

The ablations examine post-hoc alignment variants, human-performance estimation, and soft-label evaluation across datasets. They also compare entropy-aware training and alternative alignment architectures.

  • Soft-label evaluation: Table 5 evaluates DistCE and JSD after sampling 20% of human annotations against the full human distribution.Results are reported as mean ± standard deviation from 1,000 bootstrap runs.
  • Alignment architectures: Linear transformation and temperature scaling perform slightly worse, while Dirichlet calibration matches the neural-network alignment model closely.The authors therefore treat neural networks and Dirichlet calibration as equally valid architecture choices.
  • Human-performance estimation: Human-performance estimates use leave-one-out bootstrapping for datasets with three annotators to avoid optimistic self-comparison bias.The procedure averages 1,000 iterations and compares one annotator with the other two aggregated.

H Results with other Backbone LLMs

Additional experiments report hard- and soft-label predictions with GPT-OSS-120B and Qwen3-32B backbones. Case studies illustrate how model judgments can either align with or diverge from distributed human perspectives.

  • Experimental materials: The study presents prompts for hard- and soft-label prediction and reports entropy analyses without NAPHA, with predicted classes, and with oracle classes.The prompts shown for Anecdotes use the same structure as those for the other datasets, with minimal task-specific changes.
  • Case studies: In one Anecdotes case, the LLM’s individualist judgment agrees with the human ground truth while omitting collectivist perspectives.The example concerns whether a family member should receive commissioned work for free.
  • Case studies: In the second case, the LLM avoids assigning blame, whereas human judgments are polarized, with roughly 2/3 blaming the author and 1/3 the other party.The example concerns differing expectations about sharing relationship news.
  • Backbone results: Hard-label results for GPT-OSS-120B and Qwen3-32B are reported by entropy class using macro-Average F1 Scores.The corresponding results appear in Table 15.
  • Backbone results: Soft-label results for the two alternative backbones are reported separately for Anecdotes, ChaosNLI, DynaSent, SummEval, and TopicalChat.Tables 16–20 contain the dataset-specific results, with notes for cases lacking entropy-stratified train-test splits.
Loading 2609.01073v1…