PaperlayerView exact revision on arXiv

Source-linked AI summary

Fear Is Not Always Fear: Variation in Emotion Label Equivalence Across Datasets

Emily Öhman, Anna Koufakou

arXiv:2610.12841v1cs.CL

TL;DR

Same-named emotion labels may not refer to the same target across datasets, and the forms of cross-resource differences remain unclear. The paper compares resource-specific models on identical texts and under cross-corpus transfer. Transferred models lose 55.3% of native discrimination in aggregate, while surprise can show ordering disagreement that monotonic correction cannot remove.

  • Problem

    Although cross-corpus emotion classification is known to be difficult, the forms of difference between same-named labels across resources remain unclear.

    Source: §1 · p. 1

  • Method

    The paper compares resource-specific heads on identical texts and tests independent cross-corpus transfer using a four-relationship framework for label equivalence.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    Transferred models lose 55.3% of native discrimination in aggregate, and surprise shows ordering disagreement that no monotonic correction removes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A shared label name should be treated as a hypothesis of operational equivalence, not evidence that resources measure the same target.

    Source: §9 · p. 8

    The study covers three English resources, five labels, and one encoder family; its 30 transfer units make correlations descriptive, and the comparisons do not isolate causes such as annotation guidelines, label semantics, or domain.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Emotion classifiers are routinely applied beyond their training data, on the assumption that same-named labels such as fear or surprise mean the same thing across resources. We test this assumption for five labels shared by GoEmotions, XED, and BRIGHTER, comparing models on identical inputs and under cross-corpus transfer. On the same texts, one fear classifier flags 71% as fearful and another 8%, yet the two largely agree on how the texts rank. Across 30 resource-label comparisons, training-label prevalence explains about half of this kind of scale mismatch, and a fitted intercept removes most of it. What recalibration cannot repair is larger: transferred models lose 55% of native discrimination in aggregate, agreement in ranking between classifiers is only weakly related to that loss, and for surprise the resources disagree about which texts express the emotion. The choice of training resource can therefore change conclusions: on two external tweet collections, BRIGHTER- and XED-trained classifiers give opposite answers about which shows more surprise, even after rank normalization, with intervals excluding zero in both directions. A shared emotion label name does not establish an interchangeable measurement target.

1 Introduction

Same-named emotion labels may not measure the same target across resources: models can agree on text ranking while differing in probability scale, or disagree about which texts express an emotion. The paper characterizes these mismatches and asks whether they change measurement conclusions.

1 Introduction

  • On 2,765 BRIGHTER test texts, the BRIGHTER fear head classified 70.8% as fearful versus 7.6% for the GoEmotions head, despite rank correlation ρ = .897.The authors attribute much of the scale mismatch to base-rate differences; for surprise, XED and BRIGHTER heads substantially disagree about ranking.Source: §1 · p. 1
  • Emotion resources differ in taxonomies, annotation guidelines, aggregation rules, and text domains, potentially changing category boundaries, prediction confidence, or which texts are most representative.For example, fear in a 28-category scheme competes with nervousness, while in an eight-category scheme it does not.Source: §1 · p. 1

2 Related Work

  • Prior work shows cross-corpus emotion classification is difficult and unified inventories help some datasets but not others, while leaving the forms of cross-resource difference unclear.The paper retains each resource’s label function rather than collapsing resource differences into a common inventory.Source: §1 · p. 1, §2 · p. 2

1 Introduction

  • The paper asks which mismatches occur across anger, fear, joy, sadness, and surprise, and tests whether they change corpus-level conclusions.Its contributions include a four-relationship equivalence framework, same-input comparisons across 30 directed resource-label pairs, and empirical analyses of portability and repairability.Source: §1 · p. 2
  • Transferred models lose 55.3% of native discrimination in aggregate, and surprise shows ordering disagreement that no monotonic correction removes.Resource choice can also reverse corpus-level comparisons, including surprise comparisons between two tweet collections external to the training resources.Source: §1 · p. 2

2 Related Work

  • The analysis distinguishes shared ranking from shared thresholds, but is operational rather than a latent-variable or formal measurement-invariance analysis.It also uses posthoc calibration and prior-adjustment methods to study repairability and decomposes Brier-score differences into miscalibration and discrimination components.Source: §2 · pp. 2–3

3 Levels of Label Equivalence

The paper defines four progressively weaker operational relationships between same-named prediction functions, distinguishing changes in score scale from disagreement about text ordering.

3 Levels of Label Equivalence

  • L1 intercept equivalence relates logits by a constant shift, consistent with pure prior shift under usual label-shift assumptions but not proof that those assumptions hold.The fitted intercept is compared with the log-odds shift implied by training-label prevalences.Source: §3 · p. 3
  • L2 positive affine equivalence preserves text ordering while allowing probability scale or boundary sharpness to differ.Its logit mapping has a positive slope and an intercept.Source: §3 · p. 3
  • L3 nonlinear monotonic equivalence preserves ordering while relating probability scales through a monotone-increasing function.The relationship can be expressed on probabilities or equivalently on logits.Source: §3 · p. 3
  • L4 ordering divergence occurs when no monotone-increasing mapping adequately relates the predictions, so resources disagree about which texts score relatively higher.Unlike L1–L3, L4 permits the ordering itself to differ.Source: §3 · p. 3
  • These are reference relationships, not discrete labels assigned to each empirical comparison; rank agreement complements mapping error, and residual error after isotonic recalibration is interpreted alongside rank agreement.Residual error alone may also reflect estimation noise.Source: §3 · p. 3

4 Experimental Setup

The experiments compare three English resources on five shared labels, using both same-input head comparisons and independent cross-corpus transfer. Mapping and recalibration procedures are evaluated on held-out data, with target-side corrections compared against equally treated native models.

4.1 Data

  • The main analyses use anger, fear, joy, sadness, and surprise from GoEmotions, XED, and BRIGHTER, retaining multi-label items and masking labels a resource does not annotate.XED receives a deterministic 80/10/10 text-hash split; external SemEval-2018 tweet collections are sampled for comparison, not treated as representative of year-to-year population change.Source: §4.1 · p. 3, Table 1 · p. 4

4.2 Two model sources

  • A shared encoder with resource-specific heads scores identical texts for compatibility, while separately trained native specialists make six directed transfers across five labels for portability.The compatibility measures are Spearman correlation and mean absolute probability difference; transfer metrics include AUROC, average precision, F1, Brier score, and log loss.Source: §4.2 · p. 4

4.3 Direct same-input mapping

  • For each of 30 directed resource-label comparisons, intercept, positive-affine, and isotonic mappings are fitted on target-domain development predictions and evaluated on held-out target texts without gold emotion labels.A source-to-target training-prevalence log-odds shift is the parameter-free L1 baseline; held-out MAE and Spearman’s ρ are reported.Source: §4.3 · p. 4, Table 2 · p. 5

4.4 Recalibration ladder

  • Target-side repair compares intercept recalibration, prior-shift EM, Platt scaling, and isotonic regression against native models receiving the same calibration treatment.Raw Brier scores are decomposed into miscalibration, discrimination, and uncertainty; the diagnostic CORP isotonic fit uses the evaluation sample, not a held-out recalibration procedure.Source: §4.4 · pp. 4–5

5 Same Labels, Different Operational Behavior

Same-named emotion heads often rank texts similarly while using different probability scales, but that compatibility does not guarantee transferred models preserve target discrimination. The mismatch varies by label and transfer direction, with surprise showing particularly strong ordering divergence.

5 Same Labels, Different Operational Behavior

  • Mean same-input rank agreement is ρ = .756, while residuals after monotonic mapping average 1.21 times the conservative seed-stability reference.A training-prevalence shift cuts mean held-out MAE from .193 to .096; a fitted intercept reduces it further to .076.Source: §5 · p. 5, Figure 1 · p. 6
  • On BRIGHTER texts, fear heads retain high rank agreement (ρ = .897) despite a large probability-scale difference.The converse comparison has ρ = .609, while raw MAE .377 falls to .012 after monotonic mapping, within conservative same-head seed variation.Source: §5 · p. 5
  • For XED→BRIGHTER surprise, rank agreement is ρ = .329 and the monotonic residual is 2.97 times the conservative seed-stability reference.The reverse comparison has ρ = .368, but its postmonotonic residual falls within same-head seed variation.Source: §5 · p. 5

6.1 Agreement vs. discrimination

  • Across transfer directions, mean AUROC ranges from .710 to .855, but threshold selection does not repair every decision-level deficit.For XED→GoEmotions, mean F1 changes only from .171 to .169 after target-threshold selection.Source: §6.1 · p. 5
  • For BRIGHTER surprise, native AUROC is .884, but it falls to .641 from GoEmotions and .659 from XED.XED→BRIGHTER is also broadly weaker for several labels, especially anger.Source: §6.1 · p. 6
  • Rank agreement is only weakly related to discrimination loss (ρ = −.275), and 15 of 23 units with agreement of at least .70 still lose half their native discrimination.AUROC loss is also only weakly related to rank agreement (ρ = −.215).Source: §6.1 · p. 6

6.2 Probabilistic transfer error

  • Across 30 transfer units, transferred models lose 55.3% of native discrimination in aggregate, and 22 of 30 lose at least half.Surprise loses 81.7% of native discrimination across transfers.Source: §6.2 · p. 6
  • Excess Brier score averages .112 across transfer units, comprising .088 excess miscalibration and .024 discrimination loss.Probability-scale disagreement between compatibility heads tracks excess Brier error (Spearman ρ = .973), though the correspondence is partly structural.Source: §6.2 · p. 6

7 Recalibration Repair

Prevalence shifts and target-side recalibration reduce transfer error, but their effectiveness varies across resource pairs. In XED→BRIGHTER, some errors remain after recalibration, including a surprise ordering divergence supported by target labels.

7 Recalibration Repair

  • A training-prevalence shift halves mean excess Brier from .112 to .055 and improves 28 of 30 units; a target-supervised intercept removes about 72% of raw excess.Platt scaling reduces it to .029 and isotonic regression to .023, while label-free EM correction worsens aggregate portability to .122 and .125.Source: §7 · p. 6
  • For BRIGHTER→GoEmotions, an intercept reduces raw excess Brier .172 to .008; for XED→BRIGHTER, it reduces .191 to .081 and isotonic regression to .047.The effectiveness of an intercept therefore varies considerably across resource pairs.Source: §7 · pp. 6–7
  • About 50 random target annotations approach full-development performance for XED, while BRIGHTER residual error changes little with sample size.Sparse GoEmotions labels require more annotations because small samples often contain no positives.Source: §7 · p. 7

7.1 What remains after recalibration?

  • For XED→BRIGHTER fear, rank agreement is ρ = .799 and isotonic regression reduces excess Brier to .039.The raw excess Brier is .404; an intercept reduces it to .094, while Platt scaling yields .098.Source: §7.1 · p. 7, Table 5 · p. 7
  • For XED→BRIGHTER surprise, same-input agreement is ρ = .329, isotonic excess error remains .073, and the threshold-tuned F1 gap is .206.The XED head’s ordering is stable across seeds (mean ρ = .848), while BRIGHTER surprise is well discriminated natively (AUROC .884).Source: §7.1 · p. 7
  • The surprise ordering disagreement includes target-supported cases: 64.1% of items ranked top by BRIGHTER and bottom by XED are positive under BRIGHTER gold labels.Only 3.6% of the reverse group is positive, compared with 28.9% overall.Source: §7.1 · p. 8
  • The initial association between information lost in aggregated BRIGHTER annotations and transfer error disappears after matching complete five-label configurations.Thus, annotation aggregation loss does not independently explain the divergence.Source: §7.1 · p. 8

8 Resource Choice Can Reverse Corpus Comparisons

Training-resource choice can reverse corpus-level emotion comparisons, including for surprise between external collections after rank normalization. Sampling intervals quantify item uncertainty conditional on seed-averaged predictions, not training variability or multiple-comparison adjustment.

8 Resource Choice Can Reverse Corpus Comparisons

  • 8 of 15 corpus-pair–label comparisons reverse direction across heads; rank normalization removes five reversals, while three remain.The analysis uses corpora from which the models were learned, so it demonstrates dependence on training resource, not reversal frequency on arbitrary external corpora.Source: §8 · p. 8
  • Only surprise has opposite-direction external intervals that both exclude zero; gold surprise prevalence does not establish which head gives the correct direction.Intervals reflect item-sampling uncertainty conditional on fitted, seed-averaged predictions and omit training variability and multiple-comparison adjustment.Source: §8 · p. 8
  • For surprise, the 2016-minus-2017 pooled-rank difference is +.024 under BRIGHTER and −.015 under XED, with both intervals excluding zero.The reversal holds across all three seeds; the GoEmotions estimate is close to zero.Source: Table 6 · p. 8, §8 · p. 8

9 Discussion and Conclusion

Prevalence and calibration explain much of the level mismatch between same-named labels, but not ranking differences that persist after correction. The authors recommend checking both ranking and target discrimination, and testing whether conclusions depend on training resource.

9 Discussion and Conclusion

  • Check prevalence before attributing scale mismatch to deeper construct differences, use intercept-only recalibration first when target labels are available, and test conclusions across training resources.Platt scaling adds only modest average improvement, and needed target sample sizes vary, particularly for sparse labels; high rank agreement alone does not establish target-label validity.Source: §9 · p. 8
  • A latent-variable measurement-invariance analysis and cross-language same-input comparisons with BRIGHTER are proposed as future work.BRIGHTER annotates 28 languages under a single scheme, which could hold guidelines and inventory fixed across languages.Source: §9 · p. 8

Limitations

  • The study covers three English resources, five labels, and RoBERTa-base; patterns may differ for other languages, resources, or architectures.The 30 directed units come from three dataset pairs, so correlations over them are descriptive; the design does not identify whether head differences arise from guidelines, label semantics, or domain.Source: Limitations · p. 9

Ethical Considerations

  • The study analyzes existing annotations without recruiting participants or collecting new personal data, and predictions are not individual psychological assessments.The authors do not claim source texts are anonymized or screened, and advise against consequential individual decisions without separate validation.Source: Ethical Considerations · p. 9

A Shared-Label Union Training

Shared-label union training reduced native Macro-F1 for BRIGHTER and GoEmotions, while the structural analysis links probability-scale disagreement to excess Brier error. That correspondence is qualified by a conditional-probability assumption.

A Shared-Label Union Training

  • Macro-F1 falls by −.041 for BRIGHTER and −.050 for GoEmotions under shared-label union training, while XED changes negligibly (−.007).Resource-specific heads on a shared encoder recover the BRIGHTER result and part of the GoEmotions loss; changed data mixture and loss balance prevent isolating label incompatibility.Source: A Shared-Label Union Training · p. 11

C Structural Relationship Between Scale Disagreement and Excess Brier Error

  • Probability-scale disagreement closely tracks excess Brier error across 30 units (Spearman ρ = .973), with robustness across omitted labels or dataset pairs and after controlling for prevalence difference.The relationship remains strong when leaving out any label (ρ = .964–.981), for log loss (ρ = .902), and after prevalence control (partial ρ = .941).Source: C Structural Relationship Between… · pp. 11–12
  • The observed association is treated as convergent evidence, not an independent predictive result; the cross-term cancels only if target probabilities equal the target conditional probability.Ordinary marginal calibration alone is insufficient for that cancellation.Source: C Structural Relationship Between… · p. 11

D Few-Shot Recalibration

Few-shot calibration requirements vary by target resource, while BRIGHTER analyses find that the apparent raw-only annotation association does not persist after controlling for released label structure.

D Few-Shot Recalibration

  • About 50 random target annotations approach full-development performance for XED, while class-stratified 25-example samples already approach it for GoEmotions.Sparse GoEmotions labels require larger random samples because small samples often contain no positives.Source: D Few-Shot Recalibration · p. 12
  • BRIGHTER residual excess error changes little as calibration samples grow, indicating that its remaining transfer gap is not primarily a small-sample calibration problem.Intercept-only recalibration follows the same qualitative pattern as Platt scaling.Source: D Few-Shot Recalibration · p. 12

E BRIGHTER Annotation-Compression Robustness Analysis

  • Raw-only-diametric items made up 10.8% of recovered BRIGHTER annotations, with diametric structure in raw annotations absent from the released representation.Individual annotations were recovered for 5,623 of 5,643 English items; pairwise label-set Jaccard agreement rose from 0.572 to 0.594 after projection.Source: E BRIGHTER Annotation-Compression… · p. 12
  • Raw-only items initially showed greater excess error for both GoEmotions→BRIGHTER and XED→BRIGHTER, but this difference was no longer evident after Platt or isotonic calibration.Table 9 reports the raw-only-minus-other item-level excess squared-error difference, averaged over the five shared labels.Source: E BRIGHTER Annotation-Compression… · p. 12, Table 9 · p. 13
  • Adjusted raw-only-diametric excess transfer error was +0.0009 [-0.0098, +0.0115] after label-density adjustment and +0.0012 [-0.0087, +0.0111] under exact-pattern matching.Both estimates are close to zero; the controlled analyses do not support an independent compression effect.Source: E BRIGHTER Annotation-Compression… · pp. 12–13, Table 10 · p. 13
  • Recalibration left no additional raw-only-item surprise error, so annotation compression remains an unsupported explanation for the XED–BRIGHTER surprise ordering divergence.The reported Platt differences were +0.0063 [-0.0193, +0.0312] for GoEmotions→BRIGHTER and −0.0104 [-0.0367, +0.0156] for XED→BRIGHTER.Source: E BRIGHTER Annotation-Compression… · p. 13

F In-Domain Corpus Comparisons

Some in-domain corpus-comparison directions remain dependent on the classifier head after rank normalization; the external comparison reports item-sampling uncertainty separately from variation across seeds.

F In-Domain Corpus Comparisons

  • Three corpus-pair–label comparison directions still depend on the head after rank normalization, and the compared corpora are also the heads’ training domains.These reversals show training-resource dependence, not how often reversals occur on external corpora.Source: F In-Domain Corpus Comparisons · p. 13
  • Table 11 reports rank-normalized corpus differences averaged across three seeds, where positive values mean the first corpus has a higher mean pooled percentile rank.Item-sampling intervals are not estimated for these in-domain comparisons.Source: Table 11 · p. 13

G Complete External Collection Comparison

  • For Table 12, differences are 2016 minus 2017 and 1,000 shared item-bootstrap resamples produce pointwise 95% intervals conditional on seed-averaged predictions.Seed ranges separately give the minimum and maximum differences across seeds 13, 42, and 87.Source: G Complete External Collection… · p. 13

H Exploratory LLM-Assisted Annotation Revision

The exploratory revision pilot changed flagged training annotations and produced small, mixed native-performance changes rather than a consistent benefit across resources.

H Exploratory LLM-Assisted Annotation Revision

  • An exploratory audit used gpt-5.6-sol to assess 240 items: 180 flagged candidates and 60 unflagged controls, with 80 items per resource; candidates were selected using annotation.The LLM received each item’s text, assigned labels, and available label inventory, but not its dataset identity, audit scores, or model predictions.Source: H Exploratory LLM-Assisted Annotation… · pp. 13–14
  • The deliberately enriched audit sample is not representative of annotation-error prevalence, and the pilot neither establishes original annotations were correct nor rules out benefits from more extensive revision.The experiment also did not test an LLM used directly as an emotion predictor.Source: H Exploratory LLM-Assisted Annotation… · pp. 14–15
  • Revision changed 10 BRIGHTER, 31 GoEmotions, and 21 XED training items, involving 13, 64, and 47 binary label changes, respectively.Development and test annotations remained unchanged.Source: H Exploratory LLM-Assisted Annotation… · p. 14
  • Changes were small and mixed: GoEmotions improved slightly on all three metrics, BRIGHTER worsened, and XED had mixed F1 changes with slightly worse Brier.Table 13 reports paired native test-set changes after revision relative to original specialists.Source: H Exploratory LLM-Assisted Annotation… · pp. 14–15, Table 13 · p. 14

Related papers

Loading 2610.12841v1…