PaperlayerView exact revision on arXiv

Source-linked AI summary

The Failure Is in the Readout: Fine-Grained Emotion Recognition Benchmarks Measure Elicitation, Not Perception

Tobias Hallmen, Fabian Deuser, Robin-Nico Kampa, Norbert Oswald, Elisabeth André

arXiv:2610.08162v1cs.CVcs.CL

TL;DR

Fine-grained facial-emotion recognition needs richer labels, but real-face collection raises privacy concerns, and EmoNet-Face-HQ benchmarks VLMs with generated portraits. The paper keeps its data and ratings fixed while reading answers from category logits rather than generated text. All eleven tested VLMs then significantly exceed the expert agreement anchor, with three also significantly surpassing EIF.

  • Problem

    Fine-grained emotion recognition needs more than the six to eight basic emotions, but collecting finer facial data carries privacy and data-protection burdens.

    Source: §1 · p. 1

  • Method

    The authors keep EmoNet-Face-HQ’s images, taxonomy, and ratings fixed, and compare generated answers with one binary category query scored from first-position logits.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    All eleven VLMs significantly exceed the κw = 0.468 expert anchor at κw = 0.507–0.586, and three significantly beat EIF.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Benchmark scores can describe the readout procedure more than model capability when the procedure shapes the score more than the model does.

    Source: §8 · p. 8

    The P(yes)-to-rating calibration uses the gold marginal, so reported figures are an upper bound rather than a deployable procedure.

    Source: §7 · p. 8

Abstract

from arXiv · show

Fine-grained emotion recognition supports therapy tools and social robots, but it needs facial data, which raises privacy and data-protection concerns. EmoNet-Face-HQ answers that with generated portraits, expert-rated over a $40$-category taxonomy far finer than the usual six to eight basic emotions. Under the protocol it ships with, vision-language models (VLMs) score poorly on that taxonomy, and the benchmark concludes that a dedicated fine-tuned model is necessary: Empathic-Insight-Face (EIF; Small/Large). We show that off-the-shelf VLMs match or beat that fine-tuned model when the answer is not generated but read from the logits, as one binary query per category. We keep the benchmark's images, taxonomy and ratings, and change only how the answer is read. Experts agree at $\kappa_w = 0.468$ on the five categories they measure most reliably. Generatively, no interval among eleven open-weight VLMs lies entirely above that anchor ($\kappa_w=0.268$-$0.486$). Under verification all eleven clear it, each of them significantly better at $\kappa_w=0.507$-$0.586$. Three also significantly beat EIF sitting at $\kappa_w = 0.551$ (Small; $0.534$ Large). The gain comes from the graded probability and not from asking a yes/no question: as a control, thresholding those same probabilities to yes/no costs 142% of the average gains and drops binarization below generative elicitation to $\kappa_w=0.254$-$0.423$. A replication on real photographs (FACES) is weaker and mixed: of the ten models that pass a validity gate, six gain, three are neutral to positive and one is negative, so the effect is not confined to synthetic data.

1. Introduction

Fine-grained emotion recognition needs richer facial labels, but collecting real-face data raises privacy concerns; EmoNet-Face-HQ uses generated portraits and expert ratings across 40 categories. This paper argues that the benchmark’s generative readout, rather than VLM capability, explains poor scores, and tests the same models with a logit-based readout.

1. Introduction

  • EmoNet-Face-HQ addresses privacy concerns with 2,500 generated portraits rated by experts on a 40-category taxonomy, beyond the usual six to eight basic emotions.Multiple raters per image also make inter-rater agreement measurable.Source: §1 · p. 1
  • The benchmark’s generative protocol demands exactly 25 category pairs, so it can force a model to name emotions that are absent from the face.Those invented categories are scored against zeros, making the count uninformative about the image.Source: §1 · p. 1
  • The authors keep the images, taxonomy, and ratings fixed while changing how answers are read, and report that the readout recovers more than training did.They reproduce poor scores under the benchmark protocol and find unstable rankings under generative elicitation.Source: §1 · p. 1
  • Verification asks one binary question per category and reads P(yes) from the first-position logits; thresholding those probabilities to yes/no costs 0.194 κw against a +0.137 verification gain.The binarized arm falls below generative elicitation on average, supporting a role for graded probabilities rather than binary wording alone.Source: §1 · p. 2
  • All eleven VLMs significantly exceed the κw = 0.468 expert anchor under verification, scoring κw = 0.507–0.586; three also significantly beat EIF.EIF scores κw = 0.551 (Small) and 0.534 (Large) on the five most reliable categories.Source: §1 · p. 2

2. Related Work

Prior work shows that emotion benchmarks, scoring protocols, and human agreement all shape how facial-affect models are evaluated. This paper situates its readout comparison among established probability-based scoring approaches while distinguishing its use on an existing benchmark.

2. Related Work

  • EmoNet-Face combines 40 categories, synthetic balanced images, multi-expert ratings, and API-served VLMs that score far below experts using parsed generated text.The paper reuses its images, taxonomy, and ratings, and uses FACES as a transfer set.Source: §2 · p. 2
  • Evaluation rankings can shift with query format and answer readout, and probability-based methods require calibration to account for bias.Prior work reports substantial leaderboard movement under different prompting and scoring procedures.Source: §2 · p. 2
  • Per-class Yes probabilities have lifted a generative baseline from 1.56% to over 20% in 200-way bird classification.Other work applies token-probability readouts to vision scoring and language-model judging.Source: §2 · p. 2
  • Prior affect work uses hidden-state readouts, while concurrent speech work separates emitted answers from option logits; probability readouts can also disagree with generated responses.Unlike those studies, this paper applies a readout to an already published evaluation without training.Source: §2 · p. 3
  • Expert agreement is κw = 0.468 on the five most reliably measured EmoNet-Face categories and 0.166 on the remaining 35.The paper treats human–human agreement as a reference rather than a ceiling.Source: §2 · p. 3

3. Protocol

The protocol compares generative and verification elicitation on EmoNet-Face-HQ and FACES, using shared scoring and calibration procedures plus checks against invalid answers. Verification reads category probabilities directly, while the generative arm is subject to parsing and validity constraints.

3. Protocol

  • EmoNet-Face-HQ contains 2,500 synthetic images rated by psychology experts on 40 categories; FACES contains 2,052 photographs of 171 people posing six expressions.The HQ results include all 40 categories and the five with strongest rater agreement; FACES is a six-way forced choice.Source: §3 · p. 3
  • Generative elicitation parses integer category ratings from model-written JSON, with the benchmark’s zero-shot prompt requiring exactly 25 pairs and retries for invalid counts or zeros.The paper also compares its own wording and reports the better of the two prompts per model, favoring generative elicitation.Source: §3 · p. 3
  • Generative coverage varies because unparseable replies are excluded per image, with five models covering all 2,500 images and six ranging down to 1,931.A parsed reply naming three categories remains in the sample, with the other 37 categories scored as zero.Source: §3 · p. 3
  • Verification exceeds the expert anchor and three models significantly outperform EIF, while generative scores do not reliably clear that anchor.This section’s protocol uses eleven locally run open-weight VLMs; the benchmark-trained reference is EIF.Source: §3 · pp. 3–4
  • Verification asks one binary question per category and reads P(yes) from first-position logits in a single forward pass, giving coverage of 2,500/2,500 by construction.An Answer: prefill is applied because some reasoning models place a chain-of-thought token at the read position.Source: §3 · p. 4
  • Models and raters are scored with quadratic-weighted Cohen’s κ against individual raters; κw penalizes disagreements by the square of their distance on the 0–7 scale.Per-category rank calibration maps P(yes) to the benchmark’s 0–7 scale using the gold marginal, making the procedure an oracle.Source: §3 · p. 4
  • Confidence intervals resample images and refit calibration in each replicate, while paired comparisons recompute both systems on each resample.The protocol also checks whether verification has sufficient yes/no answer mass and whether generative scores pass a single-answer validity gate.Source: §3 · p. 4
  • The single-answer gap is defined for FACES, not EmoNet’s multi-rating task; ten of eleven FACES generative runs pass with a gap of exactly 0.One model fails the FACES guard and is excluded from its aggregate.Source: §3 · pp. 4–5

4. Results

All eleven VLMs cleared the human-agreement anchor under verification, whereas no generative interval lay entirely above it. The benchmark’s own systems split sharply, verification compressed model separations, and thresholding its probabilities removed the gains.

4.1. The anchor

  • All eleven verification models significantly exceeded the κw = 0.468 expert anchor, while no generative interval lay entirely above it.The weakest verification model exceeded the anchor by +0.039 [+0.026, +0.051].Source: §4.1 · p. 5
  • All five trivial baselines scored κw = 0.001–0.008, and prompt-matched tuning left κw statistically unchanged on seven of eight models.The tuning placed 85–88% of answer-token mass on the correct answer.Source: §4.1 · p. 5

4.2. The benchmark’s own baselines

  • Rescoring the benchmark’s 14 systems split them sharply: seven scored 0.375–0.551 and seven at or below 0.011; Gemini 2.0 Flash ZS scored 0.499.Gemini’s score was +0.030 [+0.017, +0.043] above the anchor; Hume Face* at 0.466 was indistinguishable from it.Source: §4.2 · p. 5
  • On the five reliable categories, three VLMs significantly exceeded EIF Small (κw = 0.551), and two also exceeded EIF Large (κw = 0.534).Across all 40 categories, six VLMs exceeded EIF Small and two exceeded EIF Large; GLM-4.6V-Flash and InternVL3 5 exceeded both sizes on both sets.Source: §4.2 · p. 5

4.3. The spread collapses inside the measurement

  • Across eleven models, verification reduced between-model sd from 0.073 to 0.024 and median interval width from 0.043 to 0.034, leaving 18 of 55 pairs separated.The spread fell inside measurement precision even as the intervals narrowed, rather than because verification was noisier.Source: Figure 2 · p. 6, §4.3 · p. 6
  • Across 25 systems, 17 changed rank by up to eight places when the 35 unmeasurable categories were dropped, without using verification scores.The total ordering is not identified at the available precision, although a partial order remains and the extremes are distinguishable.Source: §4.3 · p. 6

4.4. The gain is the readout, not the question

  • Thresholding verification probabilities at 0.5 reduced κw to 0.254–0.423, below the generative mean of 0.410 and the continuous range of 0.507–0.586.The same answers’ mAP fell to 0.554 from 0.773; binarization creates two-level predictions that calibration expands across the eight-level scale, breaking ties arbitrarily.Source: §4.4 · pp. 6–7, Figure 3 · p. 7

5. What the result depends on

The section tests whether phrasing, category choice, and the Answer: prefill explain the verification result. The headline applies with the prefill included, while the category-set result and most phrasing tests remain robust.

5. What the result depends on

  • None of the six measured protocol choices produces the result, although the headline is explicitly for verification with its prefill.The section measures three factors directly and reports supplementary checks for the other three.Source: §5 · p. 7, §5.2 · p. 7

5.1. Phrasing and category set

  • Every phrasing cell clears the anchor (minimum 0.472), and mean spread is 0.016 against a CI width of 0.034.Only the two lowest-accuracy, most saturated predictors exceed the CI.Source: §5.1 · p. 7
  • Models clear the human anchor on five categories (κw = 0.468), 35 excluded categories (0.166), and all 40 (0.204).Expert agreement is low across much of the taxonomy: 11 categories fall below α = 0.10 and three below zero.Source: §5.1 · p. 7

5.2. How much of the anchor result is the prefill?

  • Without the prefill, four of eleven models depend on it: three lack answer mass at the read position, while Gemma-3-4b-it loses 0.120 κw.Seven of the eight models whose no-prefill answers can be read still clear the anchor.Source: §5.2 · p. 7
  • The all-model headline is specific to this protocol with prefill; it is not recoverable under arbitrary read-position placement.The generative arm has no prefill analogue because it does not fix a logit read position.Source: §5.2 · p. 7

6. Replication on Real Photographs

On FACES photographs, the elicitation contrast persists but is weaker and mixed. The replication uses categorical accuracy against posed expressions because per-rater judgments are unavailable for constructing the expert anchor.

  • The replication uses 2,052 photographs of 171 people to test whether the contrast extends beyond synthetic images.FACES is introduced as a categorical replication.Source: §6 · p. 8
  • FACES replication scores accuracy over six posed expressions (chance 1/6), not graded emotion, because released ratings lack per-rater judgments for an anchor.Intervals resample the 171 people, each contributing twelve photographs.Source: §6 · p. 8
  • Six of ten valid models gain detectably on FACES, three are neutral to positive, and one reverses, for a cohort mean of +0.035 accuracy.Macro-F1 splits the same ten models the same way; the validity gate excludes MiMo-VL-7B-RL, whose accuracy delta is −0.020.Source: §6 · p. 8

7. Limitations

The limitations constrain how broadly to interpret the results: calibration uses gold-label marginals, the primary corpus is synthetic with prompt-derived demographics, and both corpora are cropped single-face portraits.

  • Quantile calibration uses the gold marginal, making reported figures an upper bound rather than a deployable procedure.Held-out calibration adds +0.026 κw on Gemma-3-4b-it, +0.023 on MiniCPM-V-4.6, and ≤0.003 on the other nine.Source: §7 · p. 8
  • EmoNet-Face-HQ demographics come from generation prompts, not observed subjects, and raters did not verify those labels.The authors therefore report no HQ demographic breakdown; comparing HQ with FACES cannot settle bias because label provenance, image type, and taxonomy differ.Source: §7 · p. 8
  • Both corpora are cropped single-face portraits, leaving in-the-wild evaluation with context and multi-label classification for future work.The FACES replication is categorical rather than graded, and its validity gate excludes one model in a direction favoring the hypothesis.Source: §7 · p. 8

8. Conclusion

The conclusion argues that benchmark scores depend on the elicitation procedure, and that deliberate answer reading can change the apparent result. It recommends reporting protocol details and human and trivial-predictor anchors, while identifying in-the-wild evaluation and label quality as open limits.

  • The same eleven VLMs move from no interval entirely above the human anchor to all eleven significantly exceeding it when only the answer readout changes.Several off-the-shelf VLMs also outperform the benchmark’s fine-tuned EIF model on its own data.Source: §8 · p. 8
  • A fixed-count generative prompt can measure compliance with the requested category count rather than which emotions the face expresses.A generative answer can fail silently, so its accuracy is uninformative unless the protocol checks that an answer was actually given.Source: §8 · p. 8
  • The authors recommend reporting the query, answer-reading method and position, and decoding settings, alongside human anchors and trivial predictors scored through the same pipeline.They also recommend showing per-category agreement beside per-category performance.Source: §8 · p. 8
  • Whether continuous readout transfers to in-the-wild multi-label affect remains open, and low expert agreement across most categories makes labels a limit on measurement.The conclusion also asks whether readout can be trained into a model and whether annotation changes can make the fine taxonomy measurable.Source: §8 · pp. 8–9

A. Supplementary: training data, not architecture

On FACES-trained frozen features, the residual gap over EIF is attributed to training data rather than head scaling. The supplements also test taxonomy coverage, prompt phrasing, run settings, retry-loop exclusion, and readout validity.

A. Supplementary: training data, not architecture

  • +0.195 of the raw +0.276 gap remains after head scaling and is attributed to training data; the FACES-trained probe is an in-domain ceiling, not a like-for-like rival.The probe uses person-disjoint splits on frozen features, separating training data from architecture.Source: A. Supplementary: training data, not… · p. 12, Table 2 · p. 12

B. Supplementary: the full taxonomy breakdown

  • The fifth and sixth categories are nearly tied in expert agreement, and Table 4 shows category-level misses despite set-level agreement.By α, the fifth category is 0.300 and the sixth 0.298; seven of 55 model-category cells within the five fall below the anchor, all on Anger.Source: Figure 4 · p. 12, Table 4 · p. 13

C. Supplementary: the verification query phrasings

  • All 33 verification results across three query phrasings clear the 0.468 human anchor, with the lowest at 0.472.The mean spread across phrasings is 0.016, against a median bootstrap interval width of 0.034.Source: Table 5 · p. 13

D. Supplementary: the generative prompts

F. Supplementary: why the retry arm is not scored

  • The retry arm is excluded because its loop changes κw by −0.153 to +0.061 across two models while driving outputs toward 25 categories.The authors characterize this arm as measuring compliance with the prompt demand rather than the model.Source: F. Supplementary: why the retry arm is… · p. 14

G. Supplementary: the no-prefill arm

  • Without prefill, the answer may not occupy the first read position, so extremely low yes/no probability mass can expose a misleading score.For MiMo and Qwen3.5, the first-position mass on {yes, no} is 2.7×10−14 and 1.5×10−17, respectively.Source: G. Supplementary: the no-prefill arm · p. 14

I. Supplementary: the contrast with coverage held fixed

  • All eleven models are in the matched contrast; restricting both arms to each model’s parsed images shifts the cohort mean from +0.1365 to +0.1380.Verification remains ahead on all eleven models after coverage is held fixed.Source: Table 7 · p. 14, §I · pp. 14–15, Table 8 · p. 15

J. Supplementary: per-model FACES results

On FACES photographs, verification improves accuracy for six of ten valid models, but the result is mixed rather than uniform. A no-image control and a five-class protocol probe the interpretation and scope of that replication.

J. Supplementary: per-model FACES results

K. Supplementary: the no-image control

  • With every photograph replaced by a gray rectangle, all 22 arms predict neutral throughout, yielding accuracy 1/6 and macro-F1 0.048.The authors interpret the result as showing that neither elicitation has priors to score without the face.Source: K. Supplementary: the no-image control · pp. 16–17

L. Supplementary: the five-class FACES protocol

  • Under the five-class FACES protocol, the ten gated-in models show two gains, five indistinguishable results, and three reversals, with mean change −0.018.Both arms are offered the same five options over the same 1,710 images; neutral is removed from the prompt rather than dropped after scoring.Source: Table 11 · p. 17, L. Supplementary: the five-class FACES… · p. 17

M. Supplementary: the benchmark’s published baselines

Rescoring the benchmark’s published baselines reveals heterogeneous outputs and a nonuniform comparison set. Separately, generative-only systems also show substantial rank changes when low-agreement categories are removed.

M. Supplementary: the benchmark’s published baselines

  • The fourteen published systems are not comparable as one group: five emit nearly constant 25-category outputs, and three fall below the benchmark’s Random Baseline.The table also reports that three baselines exceed the human anchor; the systems differ in prompting and fine-tuning status.Source: Table 12 · p. 17

N. Supplementary: rank instability without the verification arm

  • Seventeen of 25 generative-only systems change rank by up to eight places when the 35 low-measurability categories are dropped.Per-category performance tracks rater agreement with median Spearman ρ = 0.46, although the range includes negative values.Source: N. Supplementary: rank instability… · p. 17

O. Supplementary: can the model be trained into the protocol?

Training that demonstrably learned the yes/no task generally did not improve calibrated κw, while shuffled-label controls could be damaged enough to exaggerate the apparent training effect. The experiment is limited to one adapter family, rank, epoch, and learning rate.

Design

  • The test uses byte-identical one-image, one-category, yes/no training examples, with 21,612 balanced queries from 10,806 unanimously judged images.Eight models receive LoRA adapters; checkpoints and untrained bases are scored on the same fixed 400-image subset.Source: O. Supplementary: can the model be… · p. 18

The training took

  • Real-label arms put 85–88% probability on the correct held-out answer, while every permuted-label arm remains at its 50.0% chance floor.The authors use this contrast to rule out undertraining as an explanation for the subsequent null results.Source: O. Supplementary: can the model be… · p. 18

Result

  • Seven of eight models show statistically unchanged calibrated κw; Gemma-3-12b gains +0.027 [+0.008, +0.056], below the +0.011 calibration-only gain at its lower bound.Training changes emission toward the training marginal on every model, but the metric is indifferent to that shift.Source: O. Supplementary: can the model be… · p. 18

A control that can be destroyed is not a floor

  • On five of eight models, the permuted-label control scores significantly below the untrained base, by up to −0.277.Qwen3-VL-8B has the largest arm-minus-control gap, +0.275 [+0.235, +0.331], yet no gain over its base (−0.002); a shuffled control bounds an effect only if it is unharmed.Source: O. Supplementary: can the model be… · p. 18

Scope

  • The study tests one adapter family, rank, epoch, and learning rate, and claims only that a run that learned the task did not move this metric.Training images are 512 × 512 while evaluation images are 1024 × 1024; the authors say this mismatch explains neither the nulls nor Gemma-3-12b’s exception.Source: O. Supplementary: can the model be… · p. 18

P. Supplementary: tested against the benchmark’s trained model

The supplementary comparison reports paired verification differences against EIF and cautions against interpreting the cohort’s size-score correlations as evidence that parameters do not matter. The authors restrict their claim to how answers are read under the benchmark’s task, taxonomy, and rater pool.

P. Supplementary: tested against the benchmark’s trained model

  • Table 13 reports paired verification differences against EIF, with Small leading on the reliable five categories and Large on all 40.The 95% intervals use paired image resampling; the table reports no detectable relationship between parameter count and κw in eleven models, not evidence of none.Source: Table 13 · p. 19

Q. Supplementary: tools, and scope of the claim

  • EmoNet-Face’s demographic attributes come from generation prompts rather than observed properties of subjects, and the images are synthetic.This is a property of the corpus, not the authors’ pipeline.Source: Q. Supplementary: tools, and scope of… · p. 19
  • The claim is limited to answer readout under the benchmark’s task, taxonomy, and rater pool, defined by the benchmark [49].The authors note that quantile calibration uses the gold marginal and is not deployable as reported; category-level agreement tracks whatever signal the labels carry.Source: Q. Supplementary: tools, and scope of… · p. 19
Loading 2610.08162v1…