STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

Akash Bonagiri; Gerard Janno Anderias; Saee Patil; Angelina Lai; Devang Borkar; Gezheng Kang; Ishant Gandhi; Setareh Rafatirad; Houman Homayoun

Source-linked AI summary

STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

Akash Bonagiri, Gerard Janno Anderias, Saee Patil, Angelina Lai, Devang Borkar, Gezheng Kang, Ishant Gandhi, Setareh Rafatirad, Houman Homayoun

arXiv:2605.02122v2May 4, 2026cs.LGcs.AI

TL;DR

Human evaluation is vulnerable to unstable system comparisons because annotator disagreement and variability are discarded by majority vote. STABLEVAL models latent correctness, annotator confusion, and posterior uncertainty while explicitly optimizing ranking stability, yielding more stable rankings in heterogeneous, high-disagreement settings.

Problem
Annotator disagreement, bias, and variability make majority-vote system rankings fragile because majority vote discards annotator reliability and item-level ambiguity.
Method
STABLEVAL models latent item correctness and annotator-specific confusion patterns to produce posterior expected item credit and calibrated agent-level scores.
Results
STABLEVAL consistently improves error and ranking stability over Majority Vote under annotator heterogeneity, with the largest gains in high-disagreement regimes.
Takeaways & Limitations
Modeling disagreement and preserving posterior uncertainty improves robustness and stability in realistic AI evaluation settings with substantial annotator variability.
Takeaways & Limitations
STABLEVAL assumes conditionally independent annotators and requires sufficient annotation density; correlated biases or sparse labels may make estimates unstable.

Abstract

from arXiv · show

Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation. Majority vote discards annotator reliability and item-level ambiguity, often yielding unstable comparisons across annotator subsets. We introduce STABLEVAL, a disagreement-aware evaluation framework that models latent item correctness and annotator-specific confusion patterns to produce posterior expected item credit and calibrated agent-level scores. Unlike label-denoising approaches such as Dawid-Skene, STABLEVAL is explicitly designed for stable and uncertainty-aware system evaluation rather than hard label recovery. We formalize ranking stability as a first-class evaluation objective and analyze how aggregation methods preserve or distort underlying annotator behavior. Across controlled synthetic experiments and multiple real-world human-annotated benchmarks, majority vote exhibits increasing score error and ranking instability under annotator heterogeneity and adversarial noise, while STABLEVAL yields more stable and statistically grounded system rankings. These results demonstrate that modeling disagreement is essential for robust and reproducible AI evaluation.

1. Introduction

Human evaluation is central to AI assessment, but heterogeneous and structured annotator disagreement makes majority-vote scores and rankings fragile. STABLEVAL models disagreement and ranking stability directly, producing more robust comparisons across synthetic and real-world benchmarks.

Motivation: Human judgments are noisy, biased, and heterogeneous because annotators differ in expertise, strictness, interpretation, and adversarial behavior.
Motivation: Majority vote collapses judgments into hard labels, discarding annotator reliability and item-level ambiguity and making rankings sensitive to annotator composition.
Method: STABLEVAL models latent item correctness and annotator-specific confusion patterns to compute posterior expected item credit and calibrated agent-level scores.Unlike hard-label denoising, it preserves graded correctness and structured ambiguity for uncertainty-aware aggregation.
Method: The framework formalizes ranking stability as consistency of agent orderings under annotator subsampling and shows majority vote unstable while STABLEVAL is stable in expectation.
Empirical validation: STABLEVAL produces more stable system comparisons than majority vote under annotator heterogeneity and adversarial noise across synthetic tests and real-world benchmarks.Majority vote shows increasing score error and ranking volatility, whereas STABLEVAL yields statistically grounded comparisons.
Method: STABLEVAL distinguishes robust evaluation from latent-label recovery, clarifying that denoising a label does not guarantee stable system rankings.

2. Related Work

Related work increasingly treats annotator disagreement as structured information for uncertainty, calibration, and richer evaluation. However, existing approaches rarely define or optimize stable system rankings under annotator variability, which motivates STABLEVAL.

Distributional and Soft-Label Approaches: Distributional and soft-label methods retain annotation distributions rather than consensus labels to improve calibration, ranking accuracy, and uncertainty estimation.
Evaluation and Annotation Variation in Learning: Research on annotator variation shows that models often fail to capture disagreement patterns, especially in tasks requiring emotional intelligence or contextual understanding.
Distinction From Traditional Aggregation: Few existing approaches formally define or optimize stable system rankings under annotator variability, leaving ranking stability as the gap addressed by STABLEVAL.

3. STABLEVAL Framework

STABLEVAL evaluates AI agents from noisy, heterogeneous judgments by modeling annotator-specific behavior and preserving posterior uncertainty instead of collapsing annotations into hard labels. It converts this uncertainty into posterior item credit and agent scores designed to reduce sensitivity to annotator variability and disagreement.

Problem formulation: Each item has an unobserved graded correctness level, and annotators may differ systematically in strictness, leniency, expertise, interpretation, or noise characteristics.
Annotator model: Annotator-specific confusion matrices model the probability of each observed label given true correctness, separating item difficulty from annotator reliability.
Posterior inference: Expectation–Maximization estimates latent correctness and annotator reliability, while STABLEVAL retains each item's full posterior distribution instead of selecting a single hard label.
Posterior expected item credit: Posterior expected item credit assigns task-specific real-valued correctness credit, preserving partial credit and item-level ambiguity when annotations disagree.
Agent scoring: Agent scores average posterior item credit, incorporate annotator reliability and item uncertainty, and receive bootstrap confidence intervals for calibrated comparisons.
Problem formulation: STABLEVAL targets stable, uncertainty-aware agent rankings under noisy and heterogeneous judgments rather than recovery of a single consensus label.

4. Ranking Stability Analysis

The section defines ranking stability as agreement between full-set and subsampled-annotator rankings, then explains why majority vote is unstable while STABLEVAL becomes asymptotically stable under stated assumptions.

4.1. Ranking Stability Definition: Ranking stability is the expected Kendall’s τb correlation between rankings from all annotators and uniformly subsampled annotator sets.The subsample has fixed size and is drawn without replacement.
4.1. Ranking Stability Definition: A method is perfectly stable at 1, meaning agent rankings remain invariant under annotator subsampling.This measures robustness to annotator composition for reproducible evaluation.
4.2. Majority Vote Instability: A one-vote majority margin can flip after subsampling, perturbing scores and changing rankings when agent score gaps are sufficiently small.The resulting ranking-change probability is strictly positive, so expected stability is below one.
4.3. STABLEVAL Stability: STABLEVAL replaces discrete thresholding with smooth posterior expected credit and reliability modeling through annotator confusion matrices.Its procedure combines EM estimation, posterior item-credit computation, and bootstrap confidence intervals for agent scores.
4.3. Asymptotic Stability of STABLEVAL: Under correct specification, identifiability, EM consistency, and distinct limiting scores, STABLEVAL rankings converge to identical rankings across annotator subsets as item count grows.Smooth score functionals and averaging make subsampling-induced score differences vanish asymptotically.

5. Experiments

The experiments compare Majority Vote, Dawid–Skene, and STABLEVAL/PEC across diverse human-annotated benchmarks and synthetic or real disagreement regimes using stability, score, ambiguity, and annotator diagnostics.

5.1. Experimental Goals: Experiments test stability under annotator subsampling, robustness to heterogeneity and adversarial noise, and agent-score differences against Majority Vote and Dawid–Skene.STABLEVAL is instantiated as Posterior Expected Credit (PEC).
5.2. Real-World Benchmarks: Four datasets span preference evaluation, safety assessment, hate-speech detection, and medical summarization with varied pool sizes, agreement rates, and subjectivity.This diversity supports testing across disagreement regimes.
5.2. Real-World Benchmarks: MT-Bench contains 6,712 examples rated by 65 annotators, while ConvAbuse evaluates two safety agents with eight annotators on a five-point abuse scale.MT-Bench is converted to three correctness levels and exhibits substantial disagreement; ConvAbuse is converted similarly and has relatively high agreement.
5.2. Real-World Benchmarks: QAGS uses binary factual-support labels from 169 annotators, while MSLR uses facet-level 0–2 medical summarization scores aggregated into three quality levels.MSLR retains item-agent pairs with at least two independent annotators, enabling analysis under limited annotator regimes.
5.3. Methods and Metrics: STABLEVAL uses a Bayesian latent-variable model with Dirichlet priors, EM estimation, bootstrap uncertainty, and identical preprocessing across baselines.Bootstrap resampling uses 1,000 iterations to produce 95% confidence intervals.
5.3. Methods and Metrics: Evaluation reports agent scores, score adjustments, annotator diagnostics, item ambiguity, and Kendall’s Tau ranking stability under repeated annotator subsampling.The stability analysis directly tests robustness to annotator composition.

6. Results

Across synthetic and real benchmarks, PEC generally improves stability over Majority Vote under disagreement, while Dawid–Skene best recovers latent labels and high-consensus datasets make method differences small.

6.1. Synthetic Results: Across synthetic settings, Dawid–Skene has the lowest latent-ground-truth MSE, while PEC consistently improves MSE over Majority Vote.Low MSE does not necessarily imply stable agent rankings, motivating separate stability analysis.
6.1. Synthetic Results: Under adversarial noise up to 40%, Majority Vote degrades sharply, although all methods retain ranking accuracy above 98.8%.Dawid–Skene maintains the lowest MSE in this regime.
6.1. Synthetic Results: Under heterogeneous strict and lenient annotators, PEC improves over Majority Vote in both error and ranking consistency, while Dawid–Skene and PEC remain comparatively robust.Tight agent-quality gaps make rankings more noise-sensitive, where PEC and Dawid–Skene outperform Majority Vote in Kendall’s Tau.
6.1. Synthetic Results: PEC achieves Kendall’s Tau 1.000 when no hard items are present and near-perfect ranking stability at moderate annotation density.Increasing labels per item reduces MSE for every method, while Dawid–Skene retains the lowest MSE.
6.2.1. MT-Bench: On MT-Bench, PEC yields the most stable rankings under annotator subsampling, while Majority Vote is least stable and Dawid–Skene is intermediate.MT-Bench has 16.6% unanimous agreement and 53% average pairwise agreement; 8.23% of responses show high ambiguity.
6.2.2. ConvAbuse: On high-agreement ConvAbuse, all methods produce identical rankings and perfect stability, with only small score differences.Reliability differences do not alter rankings because label consensus is strong.
6.2.3. QAGS and MSLR: QAGS also shows perfect stability for all methods, whereas MSLR’s small-annotator setting increases ranking variability and makes PEC slightly more variable than the baselines.PEC nevertheless provides richer item-level ambiguity estimates in MSLR.
6.3. Summary of Empirical Findings: Overall, PEC improves over Majority Vote in error and ranking stability under heterogeneity, with the largest gains in high-disagreement regimes; high-consensus settings make methods converge.Dawid–Skene remains strongest for latent-label MSE, reflecting its denoising objective.

7. Discussion

STABLEVAL is most beneficial when annotator disagreement is substantial, because it preserves uncertainty and models annotator behavior to stabilize rankings. Its gains are limited in high-consensus settings, and its assumptions and data requirements constrain use.

7.1. When Does Disagreement-Aware Evaluation Matter?: In high-disagreement regimes such as MTBench, STABLEVAL produces substantially more stable agent rankings than majority vote and classical aggregation methods.In high-consensus settings such as ConvAbuse, aggregation methods converge to similar rankings, so disagreement-aware modeling provides limited additional benefit.
7.2. Evaluation Stability vs. Label Recovery.: STABLEVAL distinguishes stable system evaluation from latent-label recovery, since Dawid–Skene can achieve lower latent-ground-truth error without producing more stable rankings.The framework prioritizes ranking stability under annotator subsampling rather than exact latent label recovery.
7.3. Interpretability and Evaluation Practice: The framework models annotator reliability and preserves posterior uncertainty, producing diagnostics about annotator strictness, leniency, reliability, and item ambiguity.These diagnostics can inform annotation quality control, benchmark refinement, and targeted data collection.
7.5. Limitations: STABLEVAL assumes conditional independence given latent correctness, requires sufficient annotation density, and may be unstable with sparse labels or very few annotators.Its computational overhead and three-level correctness discretization may also affect practical use and quantitative outcomes.
7.6. Future Work: Future work could model annotator dependencies, task-specific difficulty, hierarchical correctness, and adaptive annotation strategies.These extensions target settings where the current assumptions or evaluation protocol are insufficient.

8. Conclusion

STABLEVAL evaluates AI systems by modeling annotator reliability and posterior uncertainty rather than collapsing judgments into hard consensus labels. It produces more robust rankings under annotator subsampling, with the clearest gains in high-disagreement settings.

8. Conclusion: STABLEVAL models annotator-specific reliability and posterior expected item credit to produce rankings more robust to annotator subsampling.Across synthetic stress tests and diverse real-world benchmarks, it targets stable and uncertainty-aware evaluation rather than exact latent-label recovery.
8. Conclusion: In high-disagreement regimes, STABLEVAL yields more stable rankings, whereas aggregation methods converge in high-consensus settings.This pattern indicates that disagreement-aware modeling is most valuable when disagreement is substantial.

Impact Statement

STABLEVAL is intended to make evaluation more transparent by exposing annotator heterogeneity and item ambiguity. Its benefits must be balanced against modeling assumptions, possible bias propagation, and the fact that stable rankings are not necessarily fair rankings.

Impact Statement: By modeling annotator reliability and preserving uncertainty, STABLEVAL aims to improve the robustness and reproducibility of system comparisons.This is especially relevant when fragile or annotator-dependent rankings could influence high-stakes model selection.
Impact Statement: Highlighting annotator heterogeneity and item-level ambiguity may encourage more careful benchmark interpretation and reduce overconfidence in marginal performance differences.The framework promotes transparency rather than treating disagreement as a single consensus label.
Impact Statement: Disagreement-aware aggregation can obscure minority perspectives or overcorrect variability when its assumptions are inappropriate or data are insufficient.Stable rankings also do not necessarily imply fair or unbiased evaluation because annotation biases may propagate through reliability-aware modeling.

B. Dataset Licenses

The study uses publicly available datasets with academic-use licenses and converts MT-Bench pairwise preferences into a three-level correctness scheme. Alternative conversion mappings were tested because absolute scores vary while relative rankings remain largely stable.

B. Dataset Licenses: The datasets used are publicly available and licensed for academic use, including CC BY-NC-SA 4.0 and Apache 2.0 sources.The listed sources include MT-Bench, ConvAbuse, MSLR-Shared Task, and QAGS.
C. MT-Bench Conversion Sensitivity: MT-Bench win/tie/loss preferences are mapped to correct/partial/incorrect outcomes through a monotonic transformation that preserves relative ordering.The conversion retains the ordinal preference structure while enabling per-response correctness modeling.
C. MT-Bench Conversion Sensitivity: Alternative binary and three-class mappings change absolute PEC scores but leave agent rankings stable across configurations, except for one top-rank exchange under Tie→Win.The exchange occurs between gpt-4 and gpt-3.5-turbo, whose top-system ranks are statistically indistinguishable; ranks 3–6 remain invariant.

D. Additional Evaluation Metrics

Additional metrics compare aggregation error, ranking stability, and ranking accuracy across synthetic stress tests and real-world benchmarks. The results distinguish low MSE from stable rankings and show PEC's robustness under heterogeneous and adversarial annotation conditions.

Synthetic MSE and ranking metrics: Across synthetic settings, Dawid–Skene achieves the lowest MSE, while PEC consistently outperforms Majority Vote; low MSE does not necessarily imply stable agent rankings.The comparison spans annotator heterogeneity, adversarial behavior, item difficulty, agent quality gaps, and annotation density.
Synthetic MSE and ranking metrics: PEC maintains near-perfect ranking accuracy as adversarial, strict, or lenient annotator fractions vary, whereas Majority Vote generally declines and Dawid–Skene can become volatile.The supplied figures report Majority Vote declining from 0.998 to 0.988 with adversarial annotators, from 0.9982 to 0.9952 with lenient annotators, and volatility under strict annotator shifts.
Synthetic MSE and ranking metrics: As labels per item increase from 3 to 9, Majority Vote ranking accuracy rises from 0.9948 to 1.0000, while all methods converge near 1.000 at nine labels.PEC's MSE decreases from 0.00560 to 0.00098 over the same range, while Majority Vote's decreases from 0.00478 to 0.00110.
Real-world benchmark metrics: The evaluation includes agent scores and ranking stability comparisons across MT-Bench, ConvAbuse, QAGS, and MSLR using Majority Vote, Dawid–Skene, and PEC.The benchmark materials also report dataset statistics, annotator quality, and ambiguous responses for these datasets.