Predicting Human Disagreement for Calibrated Dynamic Facial Expression Recognition

Motivation

• DFEW records ten annotator votes per clip, but most DFER methods collapse them to majority labels and evaluate only WAR/UAR.This discards information about how consistently an expression is perceived across annotators.Source: §1 · p. 1

Contribution

• The framework trains directly on annotator counts with a Dirichlet–Multinomial likelihood that supervises predictive mean and concentration-dependent uncertainty.This contrasts with mean-only objectives that are invariant to Dirichlet scale.Source: §1 · p. 1
• A separate ambiguity head predicts annotation entropy without votes at inference, while a monotone Chow-style reject rule combines ambiguity, vacuity, temporal instability, and input quality.The framework evaluates calibration and selective prediction under stricter identity- and movie-disjoint splits.Source: §1 · p. 1

Motivation

• Calibration degrades as human disagreement increases, with the largest overconfidence on high-entropy clips.Public DFER checkpoints are well calibrated when annotators agree but increasingly overconfident as disagreement grows.Source: Figure 1 · p. 1, §1 · p. 1

Evidential learning

• Existing evidential-learning methods typically fit Dirichlet evidence to one-hot labels, whereas this formulation supervises concentration through the likelihood.The distinction concerns the objective rather than a different output head.Source: §2 · p. 1

Label-distribution learning in FER

• FER label-distribution methods primarily optimize predictive means with scale-blind KL or cross-entropy objectives, unlike the proposed DM likelihood on un-blended count vectors.The KL-to-mean arm provides a controlled mean-only reference on the true vote distribution.Source: §2 · p. 1

Model self-disagreement

• RDFER measures prediction disagreement across temporally resampled views, whereas this work uses human disagreement as inference-time supervision for a video model.Its disagreement output is retained as an inference-time quantity rather than used only to reweight or clean training.Source: §2 · p. 1

Accuracy, calibration, and abstention

• High-WAR DFER methods target recognition accuracy, while related calibration and abstention methods are generally post hoc and do not supervise Dirichlet concentration.The paper positions count-likelihood-based selective prediction as the missing combination.Source: §2 · pp. 1–2

3.1. Mechanism: modelling the annotation process

• The model assumes annotator labels are conditionally exchangeable and does not represent annotator-specific bias or dependence.With ten votes, concentration is useful as predictive uncertainty but is not an unbiased estimate of intrinsic annotator precision.Source: §3.1 · p. 2
• The Dirichlet–Multinomial likelihood supervises both predictive means and concentration-dependent uncertainty, unlike scale-blind mean-only objectives.For fixed predictive means, it distinguishes Dirichlet scales and supplies vacuity as an uncertainty signal.Source: §3.1 · p. 2

3.2. Verifiable consequence: predicting disagreement without annotators

• A separate ambiguity head regresses normalized vote entropy from concatenated temporal-view features, while test votes are used only for scoring.The method compares this estimate with predictive-mean entropy and Dirichlet mutual information using rank, error, and discrimination metrics.Source: §3.2 · p. 2

3.3. Two engineering signals: instability and input quality

• Temporal instability is the normalized generalized Jensen–Shannon divergence across fixed-view predictive means, whereas input quality is estimated from eight frozen clip-quality features.The instability signal measures prediction variation rather than epistemic uncertainty; the quality estimator is trained on synthetic corruptions.Source: §3.3 · p. 2

3.4. Use: abstention as a reject option

• A monotone Chow-style rule fuses predicted ambiguity, vacuity, input quality, and temporal instability to estimate validation-error risk and decide whether to abstain.Nonnegative weights ensure that increasing a signal cannot reduce estimated risk, and the threshold is selected for target coverage.Source: §3.4 · pp. 2–3
• High predicted ambiguity alone need not trigger rejection because a clean, high-evidence clip may represent a genuinely mixed expression.The reject risk is learned from validation errors rather than derived from a specified abstention cost.Source: §3.4 · p. 3

3.5. Calibration term and training objective

• Calibration adds a stop-gradient Platt alignment term, while the full objective combines DM loss, ambiguity loss, and calibration loss with validation-selected weights.The calibration term aligns confidence with expected correctness without directly pushing entropy down on ambiguous clips.Source: §3.5 · p. 3

Main results on DFEW

• ECE falls from 0.052 to 0.036 while WAR remains 71.8 versus 71.6, and the full system reaches AURC 0.108 versus 0.128 for the KL-to-mean arm.The comparison uses each method’s available ranking score in Table 2.Source: §4.2 · p. 3, Table 2 · p. 4

Ablations and protocol audit

• The ambiguity head reaches ρ = 0.52 against test-clip annotation entropy, compared with 0.45 for H(ˆp) and 0.38 for mutual information.The correlation remains 0.49 and 0.47 under identity- and movie-disjoint splits.Source: §4.1 · p. 3, §4.3 · p. 4

Ablations

• Replacing the mean-only objective with DM raises misclassification-detection AUROC from 0.51 to 0.64 while changing NLL by at most 0.02 nats.Under the same reject rule, the objective lowers ECE by 0.016 and AURC by 0.009.Source: §4.2 · p. 3, §4.3 · p. 4, Table 3 · p. 4

Ablations and qualitative cases

• The fused reject rule outperforms individual signals because evidence, ambiguity, temporal instability, and quality capture different failure modes.The interpretable gates recover 45% of the gap, while ambiguity alone can reject clean but reliably modelled mixed clips.Source: §4.3 · p. 4, §4.4 · p. 4

Transfer and open set

• On FERV39k, Eq. (2) lowers AURC by 5 to 7% in every reported setting, including 0.412 to 0.390 zero-shot and 0.296 to 0.276 fine-tuned.FERV39k provides single labels, so this transfer evaluates calibration and risk ranking rather than count-based disagreement.Source: §4 · p. 3, §4.4 · p. 4, Table 4 · p. 4
• On MAFW, predicted ambiguity separates compound- from single-label clips with AUROC 0.71, while abstention is higher for DFEW-unseen classes than shared classes.The passage identifies compound labels as a coarse proxy rather than direct vote-count validation.Source: §4.4 · p. 4

• The framework improves calibration and selective prediction under shift and stricter splits without sacrificing recognition accuracy.Source: §5 · p. 4