Source-linked AI summary
Predicting Human Disagreement for Calibrated Dynamic Facial Expression Recognition
Yiming Wang, Frederick W. B. Li, Jingyun Wang
TL;DR
DFER models often collapse multiple annotator votes into majority labels, losing information about human disagreement. The paper trains on annotator count vectors with a Dirichlet–Multinomial likelihood, predicts ambiguity without votes, and applies a monotone reject rule. It preserves recognition accuracy while improving calibration and selective prediction across DFEW and transfer settings.
Problem
Most DFER methods collapse multiple annotator votes into majority labels, limiting representation of human disagreement at inference time.
Method
The framework trains on raw annotator count vectors with a Dirichlet–Multinomial likelihood, predicts annotation entropy with a separate head, and combines four signals in a monotone Chow-style reject rule.
Results
The method preserves recognition accuracy while improving calibration and selective prediction on DFEW, with gains transferring to FERV39k and stricter identity- and movie-disjoint splits.
Takeaways & Limitations
Disagreement-aware count-likelihood training provides concentration-dependent uncertainty for calibrated selective DFER prediction.
Takeaways & Limitations
The uncertainty signal is not an unbiased estimate of intrinsic annotator precision with ten votes and can also reflect model mismatch, noisy clips, or limited panel size.
Abstract
from arXiv · showhide
Dynamic facial expression recognition (DFER) benchmarks such as DFEW provide multiple annotator votes per clip, yet most models collapse them to a majority label and cannot represent human disagreement at inference time. We propose a disagreement-aware DFER framework that trains directly on the raw annotator count vector using a Dirichlet-Multinomial likelihood. Unlike mean-only soft-label objectives, the proposed likelihood provides scale-sensitive supervision for the Dirichlet concentration while preserving the predictive mean. A separate ambiguity head predicts annotation entropy for unseen clips, and a monotone Chow-style reject rule combines predicted ambiguity, vacuity, temporal instability, and input quality for selective prediction. On DFEW, the method preserves recognition accuracy while reducing ECE by 30% and AURC by 15%, and predicted ambiguity reaches a Spearman correlation of 0.52 with the annotation entropy of test clips. The calibration and selective-prediction gains transfer to FERV39k and remain under identity- and movie-disjoint DFEW splits.
1. INTRODUCTION
DFER benchmarks preserve annotator disagreement, but majority-label training discards it and leaves models overconfident on ambiguous clips. The proposed framework models count vectors directly and adds inference-time ambiguity and selective prediction.
- Motivation: DFEW records ten annotator votes per clip, but most DFER methods collapse them to majority labels and evaluate only WAR/UAR.This discards information about how consistently an expression is perceived across annotators.
- Contribution: The framework trains directly on annotator counts with a Dirichlet–Multinomial likelihood that supervises predictive mean and concentration-dependent uncertainty.This contrasts with mean-only objectives that are invariant to Dirichlet scale.
- Contribution: A separate ambiguity head predicts annotation entropy without votes at inference, while a monotone Chow-style reject rule combines ambiguity, vacuity, temporal instability, and input quality.The framework evaluates calibration and selective prediction under stricter identity- and movie-disjoint splits.
- Motivation: Calibration degrades as human disagreement increases, with the largest overconfidence on high-entropy clips.Public DFER checkpoints are well calibrated when annotators agree but increasingly overconfident as disagreement grows.
2. RELATED WORK
Prior work addresses evidential learning, label-distribution learning, self-disagreement, and calibration, but generally does not train count likelihoods for selective DFER prediction. This paper distinguishes itself by supervising Dirichlet concentration directly from annotator counts.
- Evidential learning: Existing evidential-learning methods typically fit Dirichlet evidence to one-hot labels, whereas this formulation supervises concentration through the likelihood.The distinction concerns the objective rather than a different output head.
- Label-distribution learning in FER: FER label-distribution methods primarily optimize predictive means with scale-blind KL or cross-entropy objectives, unlike the proposed DM likelihood on un-blended count vectors.The KL-to-mean arm provides a controlled mean-only reference on the true vote distribution.
- Model self-disagreement: RDFER measures prediction disagreement across temporally resampled views, whereas this work uses human disagreement as inference-time supervision for a video model.Its disagreement output is retained as an inference-time quantity rather than used only to reweight or clean training.
- Accuracy, calibration, and abstention: High-WAR DFER methods target recognition accuracy, while related calibration and abstention methods are generally post hoc and do not supervise Dirichlet concentration.The paper positions count-likelihood-based selective prediction as the missing combination.
3. METHOD
The method models annotator counts with a Dirichlet–Multinomial objective, then combines learned ambiguity with vacuity, temporal instability, and input quality for calibrated selective prediction.
- 3.1. Mechanism: modelling the annotation process: The model assumes annotator labels are conditionally exchangeable and does not represent annotator-specific bias or dependence.With ten votes, concentration is useful as predictive uncertainty but is not an unbiased estimate of intrinsic annotator precision.
- 3.1. Mechanism: modelling the annotation process: The Dirichlet–Multinomial likelihood supervises both predictive means and concentration-dependent uncertainty, unlike scale-blind mean-only objectives.For fixed predictive means, it distinguishes Dirichlet scales and supplies vacuity as an uncertainty signal.
- 3.2. Verifiable consequence: predicting disagreement without annotators: A separate ambiguity head regresses normalized vote entropy from concatenated temporal-view features, while test votes are used only for scoring.The method compares this estimate with predictive-mean entropy and Dirichlet mutual information using rank, error, and discrimination metrics.
- 3.3. Two engineering signals: instability and input quality: Temporal instability is the normalized generalized Jensen–Shannon divergence across fixed-view predictive means, whereas input quality is estimated from eight frozen clip-quality features.The instability signal measures prediction variation rather than epistemic uncertainty; the quality estimator is trained on synthetic corruptions.
- 3.4. Use: abstention as a reject option: A monotone Chow-style rule fuses predicted ambiguity, vacuity, input quality, and temporal instability to estimate validation-error risk and decide whether to abstain.Nonnegative weights ensure that increasing a signal cannot reduce estimated risk, and the threshold is selected for target coverage.
- 3.4. Use: abstention as a reject option: High predicted ambiguity alone need not trigger rejection because a clean, high-evidence clip may represent a genuinely mixed expression.The reject risk is learned from validation errors rather than derived from a specified abstention cost.
- 3.5. Calibration term and training objective: Calibration adds a stop-gradient Platt alignment term, while the full objective combines DM loss, ambiguity loss, and calibration loss with validation-selected weights.The calibration term aligns confidence with expected correctness without directly pushing entropy down on ambiguous clips.
4. EXPERIMENTS
Experiments show that the Dirichlet–Multinomial framework preserves recognition while improving calibration, selective prediction, and ambiguity estimation across DFEW protocols and transfer settings.
- Main results on DFEW: ECE falls from 0.052 to 0.036 while WAR remains 71.8 versus 71.6, and the full system reaches AURC 0.108 versus 0.128 for the KL-to-mean arm.The comparison uses each method’s available ranking score in Table 2.
- Ablations and protocol audit: The ambiguity head reaches ρ = 0.52 against test-clip annotation entropy, compared with 0.45 for H(ˆp) and 0.38 for mutual information.The correlation remains 0.49 and 0.47 under identity- and movie-disjoint splits.
- Ablations: Replacing the mean-only objective with DM raises misclassification-detection AUROC from 0.51 to 0.64 while changing NLL by at most 0.02 nats.Under the same reject rule, the objective lowers ECE by 0.016 and AURC by 0.009.
- Ablations and qualitative cases: The fused reject rule outperforms individual signals because evidence, ambiguity, temporal instability, and quality capture different failure modes.The interpretable gates recover 45% of the gap, while ambiguity alone can reject clean but reliably modelled mixed clips.
- Transfer and open set: On FERV39k, Eq. (2) lowers AURC by 5 to 7% in every reported setting, including 0.412 to 0.390 zero-shot and 0.296 to 0.276 fine-tuned.FERV39k provides single labels, so this transfer evaluates calibration and risk ranking rather than count-based disagreement.
- Transfer and open set: On MAFW, predicted ambiguity separates compound- from single-label clips with AUROC 0.71, while abstention is higher for DFEW-unseen classes than shared classes.The passage identifies compound labels as a coarse proxy rather than direct vote-count validation.
5. CONCLUSION
The framework models annotator count vectors with a Dirichlet–Multinomial likelihood, improving calibration and selective prediction without sacrificing recognition accuracy across stricter evaluation settings.
- The framework improves calibration and selective prediction under shift and stricter splits without sacrificing recognition accuracy.