PaperlayerView exact revision on arXiv

Source-linked AI summary

Bias-Corrected Ceilings of Emotion Predictability from Human Label Variation Based on Instance-Level Fano Bounds

Keito Inoshita

arXiv:2608.15619v1cs.AI

TL;DR

Whether emotion recognition has reached an accuracy ceiling remains unclear because human label variation and estimator choice affect ceiling estimates. BACE brackets bias-corrected estimates and separates error components, finding that reachability ranges from 0.38 to 1.03 while at least about 33% of GoEmotions evaluation error is irreducible, with the pattern recurring on offensiveness and irony.

  • Problem

    Whether emotion recognition has an achievable accuracy ceiling, and how close classifiers are to it, remains insufficiently established amid human label variation.

  • Method

    BACE brackets an anchored Dirichlet-mixture empirical Bayes estimator between plug-in and NSB while splitting annotators, deconvolving noise, and applying a fixed claim gate.

  • Results

    Reachability swings from 0.38 to 1.03, while the claim gate supports at least about 33% of GoEmotions evaluation error as irreducible, recurring on offensiveness and irony.

  • Takeaways & Limitations

    A plateau verdict cannot rely on a single point estimate; the supported conclusion is a conservative lower bound on irreducible error rather than saturation.

  • Takeaways & Limitations

    The reported 33% lower bound is specific to GoEmotions and the observed annotator counts, where deconvolution denoising cannot be separated from finite-sample bias.

Abstract

from arXiv · show

Emotion recognition from text keeps improving on benchmarks, yet whether an accuracy ceiling has been reached is seldom asked with discipline. Our aim is not to pin this ceiling to a single number, but to quantify how far it depends on finite annotation, estimator choice, annotation noise, and the evaluation protocol, and thereby to discipline how confidently saturation can be claimed. We propose Bias-corrected Affective Ceiling Estimation (BACE), an analysis framework that estimates a bias-corrected ceiling, separates irreducible from reducible error, and disciplines the resulting claims. An anchored Dirichlet-mixture empirical Bayes estimator, bracketed between plug-in and NSB, recovers the human-consensus distribution; an annotator split, a noise deconvolution, and a fixed claim gate then attribute error without circularity. Methodologically, unconstrained point estimates place reachability anywhere from 0.38 to 1.03, so saturation cannot be decided by any single estimator. Substantively, the only assertion passing the claim gate is that at least about 33% of a representative classifier's error on GoEmotions is irreducible, with the same pattern recurring on offensiveness and irony.

1 Introduction

The introduction argues that emotion-recognition progress cannot be judged against a single accuracy ceiling because human label variation, finite annotations, estimator choice, noise, and evaluation design materially affect reachability. BACE addresses these issues by estimating human consensus, separating irreducible from reducible error without circularity, and admitting only claims that pass a fixed discipline.

  • Motivation: Emotion-recognition benchmarks show continued improvement, but whether a ceiling exists and how close classifiers are to it remain rarely examined.Without identifying whether residual errors reflect limited model capacity or intrinsic ambiguity in human judgment, reported improvements may be genuine progress or fine-tuning toward saturation.
  • Motivation: Human label variation is a candidate source of irreducible error because annotators routinely assign different labels to subjective emotions.The introduction contrasts treating disagreement as signal with using annotation entropy to characterize ambiguity.
  • Challenges: Four challenges obstruct safe saturation claims: downward-biased entropy estimates, estimator-dependent ceilings, circular evaluation, and conflation of intrinsic ambiguity with annotation noise.These issues can respectively obscure uncertainty, reverse conclusions about remaining room, suggest superhuman performance, and overestimate the ceiling.
  • BACE: BACE quantifies how ceiling estimates depend on finite annotation, estimator choice, noise, and evaluation protocol rather than reducing the ceiling to a single point.It estimates the human consensus distribution and an information-theoretic ceiling.
  • BACE: BACE separates classifier error into aleatoric and epistemic components without circularity and disciplines conclusions through a fixed claim gate.Its workflow combines bias-corrected entropy estimation, systematic bracketing, annotator splitting, deconvolution, and predictability maps across emotion, offensiveness, and irony.
  • Implication: Reachability depends heavily on the estimator, so saturation cannot be asserted on its own.The framework is designed to admit only robust claims under a fixed discipline.

2 Related Work

Prior work treats annotator disagreement as meaningful signal and studies accuracy ceilings, entropy-estimation bias, uncertainty decomposition, and multiply annotated affective datasets. However, existing approaches do not quantify the ceiling imposed by disagreement while addressing finite-sample estimator bias and systematic versus sample uncertainty together.

  • Annotator disagreement: Annotator disagreement is established as pervasive, intrinsic, and informative, motivating alternatives to single-gold evaluation and including soft labels and annotator modeling.Related work surveys hard-label aggregation, soft labels, and annotator modeling while treating disagreement as signal rather than noise.
  • Annotator disagreement: Perspectivist methods model individual annotators, standardize soft-label evaluation, and separate annotator noise from genuine ambiguity, but do not quantify the attainable-accuracy ceiling.The cited methods preserve disagreement without addressing the ceiling it imposes or finite-sample estimator bias.
  • Accuracy ceilings: Fano’s inequality links conditional entropy to predictor-error lower bounds, while direct Bayes-error estimation uses class-uncertainty labels without relying on entropy.For a known conditional distribution, the tight 0–1 lower bound is the complement of the maximum posterior probability.
  • Estimator bias: Finite-sample entropy estimation is difficult because plug-in estimates are downward-biased, whereas NSB is low-bias in undersampled regimes alongside coverage, shrinkage, and Bayesian alternatives.The literature provides multiple estimators and closed-form results, but their finite-sample behavior remains central to ceiling estimation.
  • Uncertainty and affective datasets: Uncertainty decomposition and multiply annotated datasets support treating human disagreement as a lower bound, but affective-computing work does not jointly organize systematic and sample uncertainty.Examples span ChaosNLI, CIFAR-10H, GoEmotions, SemEval emotion tasks, BRIGHTER, MD-Agreement, EPIC, and MultiPICo.

3 Methodology

BACE estimates emotion predictability ceilings by defining human consensus relative to a specified annotator population, guideline, context, and label space, then separating irreducible from reducible error. It controls finite-sample entropy bias with an anchored Dirichlet-mixture empirical Bayes estimator, fixes evaluation protocols, and uses annotator splitting to avoid circularity.

  • Consensus definition: Human consensus pi is the probability that a random annotator from population A assigns label y under guideline G and context C.Thus, the consensus distribution is relative to (A, G, C, Y), not an absolute property of the task.
  • Evaluation protocols: Three protocols separate random-annotator 0–1 error, majority-vote evaluation, and soft-label cross-entropy to pi.Every results table carries a protocol column to prevent category inconsistencies.
  • Error attribution: Reachability R(P) is the principal quantity, with classifier error decomposed into irreducible C(P) and reducible G(P) components.C(P) is aleatoric uncertainty, G(P) is epistemic uncertainty, and R(P) measures the irreducible fraction of error.
  • Bias correction: The plug-in entropy bias is approximately −(K∗−1)/(2n ln 2), so BACE brackets principal quantities between conservative plug-in and NSB estimates.Underestimating entropy inflates the epistemic gap, while overestimating it can falsely suggest that a classifier has reached the ceiling.
  • Bias correction: The anchored Dirichlet-mixture empirical Bayes estimator transfers base-rate and confusion-pair structure across instances to stabilize entropy estimates for low-n cases.A finite mixture represents both near-unanimous and split instances, while responsibility-based assignment avoids structural leakage from reusing observed counts.
  • Evaluation design: Disjoint annotator splits construct the ceiling and oracle from estimation split A, then score the classifier and oracle on evaluation split B.This prevents the ceiling from overfitting to the same annotation noise used for evaluation; an odd extra annotator is assigned to A.

4 Experiments

Experiments across GoEmotions, offensiveness, and irony show that predictability ceilings and saturation judgments vary with estimator, deconvolution, and classifier denominator. Conservative decompositions nevertheless identify positive irreducible error across tasks.

  • Experimental setup: Experiments use GoEmotions’ 27 emotions plus neutral, binary offensiveness and irony datasets, BRIGHTER, and the public SamLowe/roberta-base_go_emotions classifier.Additional classifiers are evaluated against the same ceiling and instances.
  • Ceiling estimates: [0.258, 0.391] is the L2 ceiling bracket, corresponding to roughly 13 points of error-rate variation across estimators.Mixture empirical Bayes follows plug-in ≤ EB ≤ NSB at L2, but reverses this ordering at L1.
  • Binary decomposition: 0.0369 is the exact binary Bayes-error lower bound once n ≥2, with coincident Hamming, macro, and micro bounds.Per-label irreducible-error lower bounds range from 0.0033 for grief to 0.218 for neutral.
  • Saturation analysis: 0.386 at L2 and 0.383 at L1 give conservative reachability at β = 0.7, while the minimum across tested β values and split seeds is 0.3375.The minimum occurs at L1 with β = 0.5.
  • Saturation analysis: 0.38 to 1.03 is the reachability range across conservative deconvolved plug-in, raw plug-in, mixture empirical Bayes, and NSB estimates.Deconvolution lowers plug-in reachability but raises mixture empirical Bayes and NSB reachability.
  • Cross-classifier and cross-task results: [0.732, 1.009] is the raw empirical Bayes reachability span across classifiers, while irony has the largest base-rate-controlled ambiguity at [0.206, 0.261].Conservative irreducible error remains positive for offensiveness, irony, and emotion, making estimator dependence task-general.

5 Discussion

The discussion shows that ceiling and plateau judgments depend strongly on estimator, denominator, annotation noise, and finite annotation, so no single point estimate can establish saturation. The only claim passing the gate is that at least about 33% of GoEmotions evaluation error is irreducible, with comparable structure across offensiveness and irony.

  • Estimator dependence: 0.38–1.03 reachability across estimators shows that plateau judgments cannot rely on a single point estimate.Reachability can also cross 1 when the strong public classifier used as denominator changes, even with ceiling and estimator fixed.
  • Claim discipline: At least about 33% of GoEmotions evaluation error is irreducible, making the remaining improvement room smaller than residual benchmark error alone suggests.This is the only assertion reported as passing the fixed claim gate.
  • Label structure: 0.218 is the per-label lower bound for neutral, while rare, high-agreement emotions fall below 0.01.Average accuracy conceals this substantial internal variation across labels.
  • Noise attribution: After base-rate control, irony has the largest irreducible ambiguity, but irony and offensiveness nearly converge after deconvolution.The convergence indicates that much raw disagreement reflects annotation noise rather than irreducible ambiguity.

6 Conclusion

BACE estimates a bias-corrected predictability ceiling from human label variation while preventing circular reasoning and over-claiming. Reachability varies from 0.38 to 1.03, but the only claim passing the gate is that at least about 33% of GoEmotions evaluation error is irreducible, with the pattern recurring on offensiveness and irony.

  • Method: BACE brackets an anchored Dirichlet-mixture empirical Bayes estimator between plug-in and NSB, then decomposes error using an annotator split, deconvolution, and fixed claim gate.This estimates a bias-corrected ceiling from human label variation while blocking circular reasoning and over-claiming.
  • Findings: 0.38 to 1.03: reachability swings with the estimator and evaluated classifier, making a plateau verdict impossible without estimator discipline.
  • Findings: At least about 33% of GoEmotions evaluation error is irreducible, the only statement passing the claim gate; the same pattern recurs on offensiveness and irony.

Limitations

The reported ceilings apply only to a particular annotator population, observation channel, taxonomy, and evaluation setting. Finite-sample bias, denoising, leakage, and annotator overlap constrain how confidently the estimates can be interpreted.

  • Scope: Ceilings concern the observed annotator pool’s label distribution, not the unknowability of emotion, and do not preclude models using extra-textual information.The reference population, observation channel, and taxonomy define the estimand.
  • Estimator sensitivity: In L1 sentiment, estimator-ordering reversal persists after correction, so the mixture empirical Bayes estimate is only a reference within the systematic bracket.Claims are made at the bracket’s two ends rather than from the mixture point alone.
  • Finite-sample and denoising limits: 33% is specific to GoEmotions because deconvolution denoising cannot be separated from finite-sample bias at the observed annotator counts.The deconvolved plug-in endpoint is therefore used only as a conservative lower bound.
  • Leakage and overlap: ˆG < 0 occurs only at upper-end L2 ceilings and indicates overestimation or leakage, not superhuman performance.Annotator-level overlap may remain when classifiers encounter the same annotators’ labels during training.

Ethical Considerations · A Dataset Statistics

The paper frames emotion predictability ceilings as population- and protocol-relative conservative bounds, not limits on reading individuals’ emotions. It uses licensed public datasets and directs readers to dataset statistics summarized in Table 4.

  • Ethical Considerations: Annotator disagreement is treated as signal rather than noise, preserving perspectival differences instead of collapsing them into majority-vote labels.The framework makes this choice explicit because emotion labels are subjective and contestable.
  • Ethical Considerations: The ceiling is relative to a specific annotator population, guideline, observation channel, and taxonomy, rather than any individual’s “true” emotion.This qualification limits how the estimated ceiling should be interpreted.
  • Ethical Considerations: Only publicly released datasets were used under their respective licenses, including Apache-2.0, non-commercial, and research-use restrictions.GoEmotions is Apache-2.0; MD-Agreement and EPIC are non-commercial; BRIGHTER is restricted to research use.
  • Ethical Considerations: The code is publicly available, while datasets are excluded from the repository and retrieval scripts are distributed instead.No new human-subject data collection was conducted, and no personally identifying information was added.
  • Ethical Considerations: A foreseeable misuse is interpreting predictability ceilings as fundamental limits on the ability to read an individual’s emotions.The paper identifies this interpretation as an ethical risk requiring explicit safeguards.
  • Ethical Considerations: The ceiling is presented at dataset and population levels as a conservative lower bound to curb overclaiming in affective computing.The framing also encourages reporting achieved accuracy relative to an irreducible floor.
  • A Dataset Statistics: Table 4 summarizes the statistics of the datasets used in Section 4.1.The supplied passage identifies the table’s purpose but does not report its individual dataset values.

B Comparison with Existing Bayes-Error Estimators … E Fano Bound: Three Roles and the Per-Instance Limit

The comparison shows that existing estimators span materially different Bayes-error ceilings, while model reachability varies broadly across classifier capability. The anchored mixture estimator and per-instance Fano bound provide complementary, bias-aware ceilings and diagnostics.

  • B Comparison with Existing Bayes-Error Estimators: 0.0274 is shared by Ishida et al.’s direct estimator, Ushio et al.’s simplified soft-label isotonic estimator, and the plug-in lower end.The bias-corrected Beta–Binomial point is 0.0369, while the NSB upper end is 0.112, indicating finite-sample downward bias in the direct estimator.
  • C Multi-Model Reachability: All five evaluated classifiers remain below the ceiling at the conservative end, so a stronger model does not break it within this range.The comparison uses four fine-tuned public classifiers and GPT-4o-mini against the same GoEmotions ceiling and 5,330 evaluation instances.
  • C Multi-Model Reachability: 0.285 is the conservative-band reachability of GPT-4o-mini, the weakest classifier on the 28-class task.Its raw empirical Bayes band reaches 0.732, and reachability spans a broad capability range across the evaluated classifiers.
  • D Mixture Empirical Bayes: Closed Form and Fitting: The anchored Dirichlet-mixture estimator computes instance-level posterior quantities as responsibility-weighted sums of per-component closed forms.Posterior entropy uses the Wolpert–Wolf closed form, while E[max_y p_iy] for exact Bayes error uses responsibility-proportional Monte Carlo allocation.
  • D Mixture Empirical Bayes: Closed Form and Fitting: S = 8 at L2 and S = 5 at L1 are the selected GoEmotions mixture sizes, both equal to K + 1.The holdout criterion improves log marginal likelihood by +0.111 over S = 1; fitting caps concentration at τs ≤ 10^4 to avoid divergence.
  • D Mixture Empirical Bayes: Closed Form and Fitting: The estimator, not the data, sets the target for GoEmotions’ irreducible-error ceiling under the three estimators.Figure 5 visualizes the resulting ceilings, emphasizing estimator dependence in the target itself.
  • E Fano Bound: Three Roles and the Per-Instance Limit: Per-instance Fano is tighter than applying the bound once to aggregated entropy because Jensen’s inequality acts through the convex inverse f_K.At the dataset level, the bound averages instance-level terms f_K(H_i) before comparing with f_K(H(Y | X)).
  • E Fano Bound: Three Roles and the Per-Instance Limit: Three roles distinguish Fano from the exact Bayes ceiling: a model-agnostic guarantee, an independent entropy-based estimator diagnostic, and a bridge to log-loss evaluation.The exact Bayes error is primary for P1; Fano also addresses upward plug-in bias and connects through E[CE] ≥ H(Y | X) to P3.

F Annotator-Split and Bootstrap Details … I Validation of the Estimators on Synthetic Data

The paper fixes annotation splitting and uncertainty procedures, then validates estimator coverage, noise deconvolution, and entropy behavior on synthetic data. These checks support systematic estimator bracketing rather than relying on a single point estimate or confidence interval.

  • F Annotator-Split and Bootstrap Details: The frozen A/B split uses seed 20260723, yielding a structural A ratio of 0.613 and a per-annotator A-ratio mean of 0.593.Records are sorted lexicographically, annotators are permuted with one fixed-seed generator, and the resulting split table is reused across experiments.
  • F Annotator-Split and Bootstrap Details: At β = 0.7, conservative-reachability uncertainty intervals are [0.367, 0.401] at L2 and [0.366, 0.401] at L1 under outer noise-rate and evaluation bootstrapping.These intervals are essentially as wide as the instance-only intervals, indicating robustness to re-estimating the noise rate.
  • F Annotator-Split and Bootstrap Details: Across nine β-and-seed combinations, conservative reachability spans [0.345, 0.411] at L2 and [0.338, 0.409] at L1, with a minimum of 0.3375 at L1.The minimum uses β = 0.5 and underpins the headline lower bound of at least about 33%.
  • G Confidence-Interval Coverage Simulation: Synthetic coverage remains below nominal because of finite-sample residual bias and slight prior misspecification, supporting systematic brackets over a single estimator’s confidence interval.The simulation uses ground truth Ctrue = 0.261, Msim = 100 synthetic datasets, and B = 1000.
  • G Confidence-Interval Coverage Simulation: The bias-corrected mixture empirical Bayes estimator reaches coverage of 0.75 at n = 3 and 0.89 at n = 5, approaching nominal 0.95 as annotation count increases.Its bias is only +0.001 and +0.003, whereas plug-in coverage is 0 at both sample sizes because it is negatively biased.
  • H Annotation-Noise Deconvolution: EM and Recovery: With observed annotator counts, noise-rate estimation has mean absolute error about 0.16 and mean 0.375 versus true 0.212, while rank correlation remains between 0.88 and 0.96.The error falls to 0.018 at n = 15, and the removal effect cannot be separated from finite-sample bias before then.
  • I Validation of the Estimators on Synthetic Data: For K = 7 and n = 3, entropy bias is −1.498 bits for plug-in, −1.118 for Miller–Madow, −0.789 for NSB, and −0.781 for symmetric Dirichlet concentration 0.5.For K = 28 and n = 3, plug-in bias reaches −3.298 bits, confirming severe undersampling bias.

J Irreducible-Error Floors for All Labels

Across all labels and tasks, irreducible-error floors vary with rarity, agreement, ambiguity, and base rates, while ceiling diagnostics distinguish genuine limits from estimator alarms. For binary tasks with similar base rates, irony has the greatest irreducible ambiguity, followed by offensiveness and emotion presence.

  • Label-level floors: The instance-level irreducible-error floor is smallest for rare, high-agreement emotions and largest for frequent, ambiguous emotions.Table 7 lists floors for all 28 GoEmotions labels, although the main text reports only representative values.
  • Label-level floors: 0.357 at macro and 0.407 at micro are the oracle marginal-threshold floors for binary F1.The evaluated classifier uses threshold 0.5, and Figure 6 shows that reaching oracle F1 is unrelated to floor height.
  • Reachability diagnostics: 0.28 to 0.85 at L2 and 0.30 to 0.86 at L1 are the soft-label P3 reachability spans against the cross-entropy floor.Negative ˆG values from −0.001 to −0.013 at the upper L2 ceiling are reported as an alarm rather than superhuman performance; conservative-end gaps are positive.
  • Cross-task comparison: [0.206, 0.261] for irony, [0.173, 0.212] for offensiveness, and [0.116, 0.149] for emotion presence are the ambiguity intervals at base rates of about 0.3.Without controlling for base rates, ceilings are not comparable across tasks; among these binary tasks, irony has the largest irreducible ambiguity.

K Cross-Task SOTA Decomposition

Across offensiveness, irony, and emotion, irreducible error remains positive, while estimated reachability varies substantially by estimator. These cross-task results are conservative lower bounds because public models are not fine-tuned in-domain, and emotion labels involve a mismatched synthetic mapping.

  • Irreducible error: 0.137 for offensiveness, 0.139 for irony, and 0.116 for emotion establish positive irreducible-error floors across all three corpora.The result supports C > 0 beyond GoEmotions.
  • Estimator dependence: Rcons rises from 0.413 to REB = 0.640 for offensiveness and from 0.359 to 0.672 for irony, showing strong estimator dependence.The conservative post-deconvolution plug-in end is smallest on every task, while raw empirical Bayes and NSB increase reachability by roughly 1.3 to 1.9 times.
  • Scope and limitations: 0.36 to 0.42 for cross-task Rcons is a lower-bound range because public classifiers are not fine-tuned on the target corpora.Their inflated error means reported reachability underestimates what an in-domain fine-tuned classifier could attain.
  • Scope and limitations: 0.297 to 0.540 is the reported per-emotion reachability span, from composite fear–anxiety to anger, under a synthetic BRIGHTER-to-GoEmotions mapping.Composite categories such as fear–anxiety and social-warmth combine several GoEmotions labels, lowering reachability further.
Loading 2608.15619v1…