PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

Fardeen Sadab, Adib Sakhawat

arXiv:2609.29445v1cs.CL

TL;DR

Small-panel preference benchmarks are attractive for low-resource affective generation, but this audit finds their conclusions sensitive to measurement choices. The authors reanalyze a multilingual emoji-generation benchmark and introduce a reference-based decodability probe. Under the appropriate random-annotator analysis, the systems do not differ significantly, while output length explains much of the preference ordering.

  • Problem

    Low-resource affective generation is often evaluated through small native-speaker panels, despite the lack of an external anchor for their preferences.

    Source: §1 · p. 1

  • Method

    The authors audit a benchmark of eight instruction-tuned LLMs generating emojis for Bangla, English, and Hindi, using a crossed human-rating study and a reference-based decodability metric.

    Source: §1 · p. 1, §6 · p. 5

  • Results

    With annotators treated as random, no system differs significantly from another (F(7, 14) = 0.59, p = 0.76), unlike the conventional analysis’s 19 significant pairwise differences out of 28.

    Source: §1 · p. 1, §4.2 · p. 3

  • Takeaways & Limitations

    Output length accounts for 78.7% of between-system preference variance, and length-matched comparisons reverse the raw ordering.

    Source: §1 · pp. 1–2, §5 · p. 4

    EAD inherits biases in source-corpus emotion labels and measures consistency with those labels, not whether emoji use is good, useful, or culturally appropriate.

    Source: Limitations · p. 8

Abstract

from arXiv · show

We audit a multilingual affective generation benchmark eight instruction-tuned LLMs producing emoji summaries for 17,100 Bangla, English and Hindi sentences, with 6,960 human judgements and find its headline conclusions to be artefacts of the measurement instrument rather than properties of the systems. Treating annotators as a random rather than a fixed factor, no system differs significantly from any other ($F(7,14)=0.59$, $p=0.76$), although the conventional analysis declares 19 of 28 pairwise differences significant. Annotator identity explains far more rating variance than system identity, and the winning system changes whenever any single annotator is removed. The ordering that does emerge tracks output length: mean emoji count explains 78.7\% of between-system variance, and a within-item length-matched comparison over 2,599 pairs reverses the leaderboard. We further show that cross-provider anisotropy differences vanish under mean-centring, that per-language token costs change sign with the normalising unit, and that multi-view row-wise splits inflate macro-F1 by $3.1$ points and change the top-ranked system. In place of preference scoring we propose **emoji-affect decodability**, a reference-based probe whose rankings are stable to $\pm0.003$ macro-F1 across seeds.

1 Introduction

A full audit finds that the benchmark’s apparent system conclusions are substantially weakened or reversed by measurement choices. The paper identifies small-panel preference scoring, output length, and evaluation design as central issues, and proposes a reference-based alternative.

1 Introduction

  • Small-panel preference evaluation can appear attractive because it avoids target-language gold annotation and produces a clean leaderboard, but the audit finds its conclusions fragile in this setting.The authors connect these failure modes to low-resource affective NLP designs that use small native-speaker panels.Source: §1 · p. 1
  • Treating annotators as random yields no significant system differences, while conventional analysis declares 19 of 28 pairwise differences significant.The authors also report that no number of items makes a ranking reliable with three annotators.Source: §1 · p. 1
  • Output length accounts for 78.7% of between-system preference variance, and length-matched comparisons reverse the ordering.The paper characterizes the previously reported “smaller-model-wins” effect as a verbosity effect.Source: §1 · pp. 1–2
  • Emoji-affect decodability is a reference-based, annotator-free metric, and cross-language decoding costs 0.252 macro-F1 despite > 99.7% vocabulary overlap.The transfer result indicates that the emoji code models emit is language-specific.Source: §1 · p. 2
  • The audit finds that provider anisotropy gaps are a mean-offset artefact, language separability without parallel corpora is not alignment, and token-premium direction depends on the normalising unit.These are corrections to three geometric and economic claims.Source: §1 · p. 2
  • Multi-view row-wise splits add 3.1 macro-F1 points of optimism and change the top-ranked system.The paper quantifies this leakage rather than merely asserting it.Source: §1 · p. 2

2 Related Work

Related work situates the audit among research on human-judgement reliability, length bias, emoji and affect, embedding geometry, and data leakage. The cited literature provides established methods and cautions that frame the paper’s measurement corrections.

Human evaluation and its reliability

  • Prior NLP work has raised concerns about statistical treatment of human judgements; generalisability theory is standard but only sporadically applied in NLP.The section also notes that product-moment correlation misses systematic rater bias and points to alternative agreement coefficients.Source: §2 · p. 2

Length and verbosity bias

  • Human and model-based preference judgements favour longer outputs, and length-controlled preference metrics have been proposed for open-ended text.The authors report the same bias in emoji generation, where it accounts for almost all apparent system quality.Source: §2 · p. 2

Emoji, affect and low-resource languages

  • Emoji affect is partly conventional and partly culturally specific; Bangla and code-mixed South Asian resources also face transliteration and script-noise challenges.The paper frames its contribution as a measurement protocol, not a new emoji resource.Source: §2 · p. 2

Embedding geometry

  • Embedding anisotropy statistics can be confounded with the embedding mean, while cross-lingual alignment is conventionally assessed using parallel-sentence retrieval.The authors argue that alignment claims without parallel data measure something else.Source: §2 · p. 2

Contamination and leakage

  • Multi-view leakage is a special case of contamination, and grouped splitting is standard in clinical and speech corpora but often omitted in multi-model NLP benchmarks.Source: §2 · p. 2

3 Benchmark and Study Design

The study evaluates emoji-based affective expression across three languages and audits how its human-rating design affects system comparisons. With annotators treated as a sample from a population, system differences are not significant, and the leaderboard is sensitive to who rates the outputs.

3 Benchmark and Study Design

  • The benchmark covers 17,100 Bangla, English, and Hindi sentences, but the corpora use different emotion-label schemes, limiting cross-language label comparisons.Cross-lingual analyses are restricted to five shared emotions and single-label items; three fluent Bangla speakers rated a fully crossed sample of 290 sentences across eight systems.Source: §3 · pp. 2–3

4.1 Agreement

  • Three annotators show almost no shared signal: bias-aware agreement is αordinal = 0.014 and ICC(2, 1) = 0.116, despite Pearson correlations of 0.17–0.18.The annotators’ mean ratings range from 2.86 to 4.35, illustrating their different scale use.Source: §4.1 · p. 3

4.2 The system effect disappears under the correct error term

  • F(7, 14) = 0.59, p = 0.76: no system differs significantly when annotators are treated as a random factor, although the conventional analysis finds 19 of 28 pairwise differences significant.The system effect is significant against the residual but not against the system×rater interaction, the appropriate error term when annotators represent a population.Source: §4.2 · p. 3, Table 1 · p. 3

4.3 Variance components and the decision study

  • Annotator identity accounts for 22.1% of total variance, compared with 0.7% for the system; item difficulty accounts for 18.8%.Among random factors, annotator identity accounts for 29.1%, item×annotator interaction for 22.9%, and residual noise for 32.6%.Source: §4.3 · pp. 3–4
  • With 290 items, G = 0.336; reaching G ≥ 0.8 requires 27 annotators, and three annotators cannot reach that threshold with any number of items.The decision study’s reliability curves saturate in the number of items when annotator count stays fixed.Source: §4.3 · p. 4

4.4 The winner depends on who is on the panel

  • Removing one annotator changes the top-ranked system in two of three cases, and Claude-3-Haiku ranges from first to last.Between-annotator Kendall correlations over the eight system means are −0.36, 0.07, and 0.00.Source: §4.4 · p. 4, Table 2 · p. 4

5 What the Ratings Actually Track

The apparent preference ordering tracks how many emojis systems emit, while comparing systems on the same-item, same-length outputs yields a different ordering. Length bias also varies across annotators.

5 What the Ratings Actually Track

  • Mean emoji count explains 78.7% of between-system MOS variance, and the judgement-level estimate is β = 0.159 MOS points per additional emoji.The eight systems' best-to-worst MOS spread is 0.341 points—slightly more than two emojis' worth of output length.Source: §5 · p. 4, Table 3 · p. 4
  • In 2,599 within-item comparisons matched on emoji count, the ordering correlates with raw MOS at ρ = 0.12 (p = 0.78); Gemini-2.0-Flash moves from sixth to first.Its win rate is 0.603 (95% CI [0.540, 0.664]); the reported smaller-model preference does not survive.Source: §5 · p. 4, Table 3 · p. 4
  • Per-annotator slopes on emoji count are 0.085 (p < 10−5), −0.024 (p = 0.14), and 0.329 (p < 10−48), with system-level length correlations of ρ = 0.95 and ρ = −0.26 for two annotators.The passage characterizes length bias as an annotator property aggregated by the panel average.Source: §5 · p. 4

6 Emoji–Affect Decodability

Emoji–affect decodability (EAD) evaluates how much corpus affect information is retained in generated emoji, using existing emotion labels rather than an emoji–affect lexicon. It yields stable, language-specific scores, but does not track human preference.

6 Emoji–Affect Decodability

  • Figure 2 reports that EAD does not correlate with MOS, unlike mean output length, which explains 78.7% of between-system MOS variance.The within-item length-matched win rates reorder the leaderboard; the figure gives 95% binomial confidence intervals.Source: Figure 2 · p. 5
  • EAD is the 5-fold cross-validated macro-F1 of a regularised multinomial logistic regression on sublinear TF-IDF emoji features predicting corpus emotion labels.It uses the labels already carried by source corpora and is defined without an emoji–affect lexicon.Source: §6 · pp. 4–5
  • Across ten random seeds, the largest per-system standard deviation is 0.0033 macro-F1; item bootstraps yield CIs of width ≈0.032 separating three tiers.EAD has no annotator requirement and is deterministic for fixed model outputs; its MOS correlation is ρ = −0.19 (p = 0.65).Source: §6 · p. 5
  • Bangla EAD ranges from 0.560 (DeepSeek) to 0.657 (Gemini), English from 0.515 to 0.593, and Hindi from 0.376 to 0.458.The union of all eight systems' emojis reaches 0.686 in Bangla, above any single system.Source: §6 · pp. 5–6
  • Cross-language decoder transfer costs 0.252 macro-F1 on average despite target-language vocabulary coverage exceeding 99.7% in every direction.The passage reports that the same emojis are used to mean different things across languages.Source: §6 · p. 6
  • On matched items, the best emoji channel reaches 0.673 macro-F1 versus 0.595 for the best sentence embedding in Bangla, 0.613 versus 0.736 in English, and 0.430 versus 0.387 in Hindi.The embedding learning curve is still rising steeply at n = 400, so the paper claims competitiveness at matched sample size, not superiority.Source: §6 · p. 6

7 Geometry: Two Corrections

The audit revises two geometric interpretations: provider anisotropy gaps disappear after mean-centring, and language separability does not establish cross-lingual alignment. The latter cannot be tested with these nonparallel corpora.

7 Geometry: Two Corrections

  • After mean-centring, every provider's anisotropy lies in [−0.0008, 0.0013], removing the apparent raw differences attributable to a common displacement vector.Raw anisotropy does not predict pooled emotion-probe macro-F1 across the five encoders (r = −0.48, p = 0.41, n = 5).Source: §7 · p. 6
  • Because the three corpora are not parallel, language-distribution comparisons are confounded with topic, register, and domain and cannot establish alignment.A language classifier reaches ≥0.993 accuracy on four of five encoders, but low separability is not a virtue: Gemini has the weakest emotion probe (0.290 pooled macro-F1 vs. 0.593 for Qwen3).Source: §7 · p. 6

8 Cost Accounting Changes Sign with the Unit

The apparent per-language token-cost conclusion changes with the normalising unit, and raw completion-token counts can include internal reasoning traces. The reported cost measures do not predict EAD.

8 Cost Accounting Changes Sign with the Unit

  • Measured in prompt tokens per source byte, Bangla is cheapest (1.108 vs. 1.450 for English and 1.557 for Hindi); per source character, Bangla costs 2.93 against English's 1.45.The passage attributes the 2.0× per-character premium to Bangla text averaging 2.64 UTF-8 bytes per character.Source: §8 · p. 6
  • Grok-4-Fast averages 294.7 completion tokens per Bangla item while emitting 2.60 emojis (124 tokens per emoji), whereas the other seven systems average 1.4–4.4.The passage says billed completion-token comparisons reflect reasoning configurations, not task efficiency.Source: §8 · p. 6
  • Spearman correlations with EAD are −0.14 for total tokens, −0.12 for completion tokens, and −0.14 for tokens per emoji, none significant.The only cost-like quantity predicting MOS is emoji count (ρ = 0.90), which the passage identifies as a bias rather than a quality signal.Source: §8 · pp. 6–7

9 Multi-View Leakage, Measured

Row-wise splits of multi-view benchmarks place the same source sentence in training and test data, making the expected test-row overlap nearly complete. In the measured Bangla benchmark, this leakage inflated macro-F1 and changed the top-ranked system.

9 Multi-View Leakage, Measured

  • At V = 8 and ρ = 0.8, 99.998% of test rows are expected to share their source item with training; at V = 5, the figure is 99.84%.The paper identifies expected test-row overlap, rather than the probability that no item is split, as the decision-relevant quantity.Source: §9 · p. 7
  • A row-wise 80/20 split scored 0.630 ± 0.003 macro-F1 versus 0.598 ± 0.014 for source-item grouping, a +3.1-point increase that changed the top-ranked system.Rankings under leaky and clean protocols correlated at ρ = 0.76; Mistral-Large became Gemma-3-27B.Source: §9 · p. 7

10 Recommendations

The recommendations call for better agreement reporting, random-factor treatment, decision studies, length matching, reference-based metrics, and appropriate controls for geometry, corpora, token costs, and splitting. The authors qualify these recommendations by the narrow human-study panel and language scope, EAD’s inherited biases, endpoint-specific geometry measurements, and the single-benchmark audit.

10 Recommendations

  • Report Krippendorff’s α or an ICC with a bootstrap CI, alongside annotator means and SDs, rather than treating Pearson correlation as agreement.A paired test over items on panel-averaged scores is insufficient for claims about systems generally; include the system×annotator interaction in the error term.Source: §10 · p. 7
  • Estimate the needed panel size with a small fully crossed pilot and decision study before collecting a large set of judgements.The current study’s three annotators are a narrow, demographically similar panel; its variance-component estimates are correspondingly imprecise.Source: §10 · p. 7, Limitations · p. 8
  • Length-match outputs in a within-item comparison rather than length-adjusting scores, and report it alongside the raw score.Source: §10 · p. 7
  • Report a reference-based metric such as EAD alongside preference scores; its number does not move when the panel changes.EAD is not a substitute for properly powered human judgement, inherits source-label biases, and supports only within-language system comparisons here.Source: §10 · p. 7, Limitations · p. 8
  • Centre embeddings, avoid alignment claims from non-parallel corpora, disclose token-cost normalisation by characters and bytes-per-character ratio, and split multi-view tables by source item.Source: §10 · p. 7

Limitations

  • Human-study reliability is demonstrated only for Bangla, while the three studied languages are South Asian or English, not multilingual affective AI generally.The authors expect reliability to be worse, not better, where fewer annotators are available.Source: Limitations · p. 8
  • EAD’s scores reflect source-corpus emotion-label biases and consistency with corpus emoji use, not necessarily usefulness or cultural appropriateness.The three corpora differ in label scheme, granularity, and provenance, so the authors compare systems only within a language.Source: Limitations · p. 8
  • Geometry findings concern five commercial API encoders measured on 1,202 sentences, and may change as endpoints change; the audit covers one benchmark built by the authors.Training data were unavailable, and generalisation of the documented failure modes has not been demonstrated.Source: Limitations · p. 8

Ethics Statement

The ethics statement describes the annotators and data sources, flags risks from attributing emotion to identifiable people, and cautions against consequential deployment of emoji-based affect systems.

Ethics Statement

  • The three fluent Bangla annotators worked independently of the research team, volunteered, and were informed of the study purpose.Source: Ethics Statement · p. 8
  • The sentences came from public Bangla, English, and Hindi corpora and were not additionally filtered for personally identifying content.The work used sentence identifiers and model outputs under the source corpora’s licences.Source: Ethics Statement · p. 8
  • The authors caution against consequential deployment because emotion attribution to identifiable people risks misrepresentation; the best system recovers only 66% of Bangla five-way emotion distinctions.Offensive emoji use was not exhaustively audited, although no such cases were observed.Source: Ethics Statement · p. 8

A Prompt

The appendix gives the shared emoji-only generation prompt and the decoding and analysis settings. It also specifies grapheme segmentation, which affects measured output-length ordering.

A Prompt

  • All systems and languages received the same single-turn prompt, with no system prompt.Source: A Prompt · p. 9
  • The prompt required one to five emojis representing the sentence’s emotion and prohibited words or explanations.Source: A Prompt · p. 9
  • Decoding used temperature 0.7 and a 50-token limit, with provider defaults for top_p and stop sequences and no exposed seed.Source: A Prompt · p. 10
  • Responses without any non-ASCII character triggered up to three retries, with observed retry rates of 0.001–0.064 per system.Source: A Prompt · p. 10

B Analysis Settings

  • Analyses ran on CPU from stored outputs with fixed random seeds; agreement and EAD used 2,000 bootstrap replicates and MOS confidence intervals used 5,000.Source: B Analysis Settings · p. 10
  • Probes used ℓ2-regularised multinomial logistic regression with sublinear TF-IDF emoji features or ℓ2-normalised embeddings, under five-fold stratified cross-validation.The emoji and embedding probes used C = 4 and C = 1, respectively, with class-balanced weights.Source: B Analysis Settings · p. 10
  • Emoji were segmented into graphemes, retaining joined sequences and skin-tone modifiers while discarding stray ASCII characters.Source: B Analysis Settings · p. 10
  • Naive per-codepoint segmentation inflated mean emoji counts by 15.0% overall and unevenly across systems, distorting their length ordering.The reported inflation ranged from +5.4% for Claude-3-Haiku to +31.5% for DeepSeek-V3.2.Source: B Analysis Settings · p. 10

Related papers

Loading 2609.29445v1…