A Deeper Look at Facial Expression Dataset Bias

I. INTRODUCTION

• The paper identifies a gap in understanding how facial-expression datasets differ and how those differences affect classification, against a background of increasingly diverse real-world data and varying test domains.Earlier evaluation often focused on a single database, while real-world collections include factors such as pose, occlusion, illumination, background, and subject identity.Source: §I · p. 1
• ECAN aligns marginal and conditional distributions while using a learnable class-wise weighting parameter to make re-sampled source and target class distributions more similar.The approach is unsupervised and uses target-domain label information; it also learns domain-invariant, class-discriminative representations.Source: §I · pp. 1–2
• The bias analysis attributes dataset differences to capture preferences and inconsistent annotator perceptions, which challenge the assumption that conditional distributions remain unchanged across domains.Capture bias is associated with marginal-distribution differences, while category bias is associated with conditional-distribution differences.Source: §I · p. 2
• ECAN yields comparable or superior performance to state-of-the-art methods across cross-dataset facial-expression recognition experiments on lab-controlled and real-world databases.The evaluated datasets include CK+, JAFFE, MMI, Oulu-CASIA, AffectNet, FER2013, RAF-DB 2.0, and SFEW 2.0.Source: §I · p. 2

II. RELATED WORK

• Prior domain-adaptation work includes instance-, model-, and feature-based approaches, while many feature-based methods neglect conditional-distribution bias.Some model-based methods require labeled target data; feature-based methods generally seek domain-invariant representations.Source: §II · pp. 2–3
• The paper motivates unsupervised adaptation because target-domain annotation is difficult and positions ECAN as addressing conditional-distribution bias and class imbalance without target labels.It uses pseudo labels to explore target-domain semantic information while considering both marginal and conditional distributions.Source: §II · p. 3

III. MEASURING FACIAL EXPRESSION DATASET BIAS

• Earlier cross-database evaluations reported substantial performance drops, motivating investigation of dataset differences beyond within-database assessment.Examples cited include decreases of around 36% on MMI and 40% on JAFFE, and results above 90% within a database falling to 50–60% on another database.Source: §III · p. 3
• Dataset recognition and class-specific cross-dataset generalization tests are used to quantify bias across lab-controlled and in-the-wild databases.The evaluated datasets include AffectNet, CK+, FER2013, MMI, Oulu-CASIA, RAF-DB 2.0, and SFEW 2.0.Source: §III · p. 3

A. Dataset Recognition

• The dataset-recognition experiment uses seven datasets, aligned and normalized face images, two feature representations, and person-independent evaluation averaged over ten data splits.The features are handcrafted HOG descriptors and deep features; 500 images per dataset are sampled for training and 200 remaining images per dataset for classification.Source: A. Dataset Recognition · pp. 3–4
• 79% dataset-recognition accuracy, against 14% chance, is reached with a small training sample, with no evident saturation as training data increases.The figure compares two feature representations as a function of training-set size.Source: A. Dataset Recognition · p. 4, Figure 2 · p. 4
• HOG separates lab-controlled datasets, especially CK+ and Oulu-CASIA, from others, while real-world datasets form confusion groupings and AffectNet and RAF-DB 2.0 are most confused.The authors attribute the latter confusion mainly to the datasets' diverse and extensive image samples.Source: A. Dataset Recognition · p. 4
• The authors attribute dataset distinctions to capture preferences, including acquisition devices, environments, participant identities, age, and ethnicity, and link capture bias to differing marginal distributions.Examples include laboratory cameras versus Internet snapshots, and a wider range of age and ethnicity in Internet-sourced datasets.Source: A. Dataset Recognition · p. 4

B. Cross-dataset Generalization

• The experiments compare expression detection and classification across datasets, focusing on happiness and sadness and evaluating cross-dataset performance against in-dataset performance.The setup follows person-independent protocols; the reported transfer measure emphasizes the percent drop from in-dataset results.Source: B. Cross-dataset Generalization · p. 4
• In seven-expression classification with deep features, average in-dataset performance across datasets was 63.83%, falling to 39.40% for average cross-dataset performance, indicating a significant transfer-performance drop.Source: B. Cross-dataset Generalization · p. 5
• Lab-controlled datasets, especially CK+, are easier across tasks, while AffectNet and RAF-DB 2.0 are more diverse and sometimes transfer better to lab data than in-dataset.The latter pattern holds for several tasks, not all tasks.Source: B. Cross-dataset Generalization · p. 5
• Category bias—including differing annotator interpretations and images showing multiple expressions—makes annotation uncertain and conditional distributions differ across datasets.The paper notes that prior adaptation methods often assume these conditional distributions remain unchanged.Source: B. Cross-dataset Generalization · p. 5

IV. EMOTION-CONDITIONAL ADAPTION NETWORK

• Although deep features show generalization ability, dataset bias remains evident, motivating the proposed adaptation approach.The paper introduces ECAN to address cross-dataset facial expression recognition.Source: §IV · p. 6

A. Problem Formulation

• The adaptation problem is to reduce discrepancy between related source and target datasets, using source supervision to minimize target error.The source has labels, whereas the target does not; the domains differ due to acquisition conditions.Source: A. Problem Formulation · p. 6

MMD background

• MMD measures distribution divergence in an RKHS, and an unbiased empirical estimator with linear-time complexity adapts it to CNN stochastic-gradient training.The paper describes MMD as a non-parametric metric for comparing domain embeddings.Source: B. Preliminary and Motivation · p. 6

Motivation

• Marginal alignment alone does not address conditional-distribution discrepancies, so the method also matches conditional distributions across domains.The paper states that facial-expression dataset conditional distributions differ.Source: B. Preliminary and Motivation · p. 6
• Domain invariance does not guarantee class discriminability; target fear features could align near source surprise features while satisfying global invariance.This example illustrates why global alignment alone may leave features poorly separated by class.Source: B. Preliminary and Motivation · p. 6
• Class imbalance can mislead judgments about source relevance because common expressions dominate while less common expressions are harder to capture and annotate.The paper identifies this bias as a bottleneck for cross-dataset facial-expression analysis.Source: B. Preliminary and Motivation · pp. 6–7

Class re-weighting

• Unequal class priors can shift marginal distributions even when class-conditional distributions match, so marginal-only alignment may misguide class predictions.The paper illustrates this with target samples incorrectly classified as fear under strict source-marginal matching.Source: C. Emotion-Conditional Adaption… · p. 7
• A learnable weight ratio reweights source classes toward the target class distribution, estimated from target pseudo-labels, to reduce class-distribution bias.The method amends the MMD distance with this ratio.Source: C. Emotion-Conditional Adaption… · p. 7

Class-conditional invariant features

• Class-conditional MMD matches each class across domains using source labels and target pseudo-labels, learning class-wise invariant features.The method explicitly minimizes distribution distance separately for each class.Source: C. Emotion-Conditional Adaption… · p. 7
• Aligning class centroids can suppress false pseudo-label effects through correct labels and bring same-label samples nearby, enhancing feature discrimination.The paper says correct and wrongly labeled samples’ effects are neutralized together.Source: C. Emotion-Conditional Adaption… · p. 8

MMD layer

• The last feature-extraction layer serves as the MMD RKHS because deeper features become more task-aimed and harder to transfer directly.The layer is used to regularize representations against domain shift.Source: D. Optimization of the Network · p. 8

Pseudo-labeling

• Periodic target pseudo-labels replace unavailable ground-truth labels, while minimum-entropy confidence weights regulate each tentative label’s contribution.The confidence parameter is intended to make use of unlabeled target data while adding robustness to learning.Source: D. Optimization of the Network · p. 8

Joint optimization

• The MMD loss weights γ and λ are kept small early in training to suppress noisy pseudo-label information.These parameters weight the loss functions in the objective.Source: D. Optimization of the Network · pp. 8–9
• ECAN combines source softmax supervision with MMD regularization to learn discriminative source features while matching marginal and conditional distributions across domains.The network is updated using mini-batch SGD.Source: D. Optimization of the Network · pp. 8–9

• Four lab-controlled datasets and four real-world databases are used to examine the method’s adaptation power.Source: §V · p. 9

• The experiments use AffectNet, CK+, FER2013, JAFFE, MMI, Oulu-CASIA, RAF-DB, and SFEW 2.0, described as widely used facial-expression datasets; Fig. 4 shows their experimental class distributions.Source: A. Databases and Setups · p. 9
• Around 280,000 AffectNet training images and 3,500 validation images are used, with neutral and six basic-expression labels.Source: A. Databases and Setups · p. 9
• CK+ contributes 1,236 images from 309 labeled sequences, while MMI contributes 704 images sampled from video sequences.For both, neutral and peak-expression frames are selected.Source: A. Databases and Setups · p. 10
• FER2013 contributes 35,887 labeled images, while JAFFE contributes all 213 samples from ten Japanese women.Source: A. Databases and Setups · p. 10
• RAF-DB 2.0’s experimental subset contains 14,216 images after rare-class additions, and its class distribution is more balanced than the original.Source: A. Databases and Setups · p. 10

• The model fine-tunes VGG-Face, appending two MMD-based regularization layers and changing the prediction layer to seven classes.Aligned grayscale images are resized to 256 × 256, randomly cropped to 224 × 224, and flipped horizontally with 50% probability.Source: B. Implementation Details · p. 10
• No labeled target samples are used; pseudo-label regularization weights are initially suppressed and increase with training progress.The trade-off parameters are grid-searched, and stochastic gradient descent uses momentum 0.9 with a base learning rate of 0.001.Source: B. Implementation Details · p. 10

• ECAN outperforms comparison methods on most datasets and is competitive on difficult transfers, especially when source and target domains differ more.Source: C. Experimental Results · p. 11, Table III · p. 11, Table IV · p. 12
• 89.69% is ECAN’s best CK+ cross-dataset performance using RAF-DB 2.0 with three lab-controlled datasets as a multisource domain.ECAN also slightly improves on CK+’s 85.40% in-dataset result.Source: C. Experimental Results · p. 11
• On JAFFE, ECAN achieves the best performance, while the fine-tuning method that scored highly on CK+ is ineffective on this gender- and ethnicity-biased dataset.The method matches marginal, conditional, and class distributions across domains.Source: C. Experimental Results · p. 11
• 69.89% cross-dataset accuracy on MMI exceeds all compared methods and the reported in-dataset performance.The authors attribute ECAN’s results to re-weighted MMD and class-conditional MMD addressing class-distribution bias and learning more discriminative features.Source: C. Experimental Results · p. 12
• On Oulu-CASIA, ECAN outperforms baselines and in-dataset recognition when its two MMD regularizations match class distributions and reduce marginal and conditional distribution distances.Source: C. Experimental Results · p. 12
• On AffectNet, original MMD slightly degrades performance under a very different class distribution, whereas ECAN mitigates the discrepancy; ECAN also significantly outperforms comparison methods on FER2013 and SFEW 2.0.FER2013 results show gains even when the target domain is much larger than the source.Source: C. Experimental Results · p. 12
• The authors report that fine-tuning declines substantially with obvious domain discrepancy, while ECAN can improve accuracy when marginal-only MMD gains little or performs worse.They relate ECAN’s gains to considering marginal and conditional distributions and class-distribution bias.Source: C. Experimental Results · pp. 12–13

• The analysis evaluates ECAN from several different aspects.Source: D. Empirical Analysis · p. 13
• γ and λ are tunable hyperparameters representing the degree of domain transfer.Source: 1) Parameter sensitivity · p. 13
• The sensitivity study tests parameter values on CK+ and FER2013, with the baseline CNN shown as dashed lines in Fig. 6.Source: 1) Parameter sensitivity · p. 13

1) Parameter sensitivity:

• On CK+, accuracy peaks at λ = 0.3, while on FER2013 it peaks at λ = 0.1; performance then falls as λ increases further.ECAN performs better than original MMD as λ varies; with γ fixed, increasing λ from the reweighted-MMD-only case further improves results, supporting the conditional MMD term.Source: 1) Parameter sensitivity · p. 13, Figure 6 · p. 13

2) Ablation study:

• On RAF-DB 2.0→CK+, all tested ECAN variants outperform baselines, and the variant combining reweighting with conditional MMD performs best.The authors report that the two MMD regularization terms cooperate to achieve desirable adaptation behavior.Source: 2) Ablation study · p. 13, Table VIII · p. 13

3) Feature visualization:

• On CK+, ECAN alleviates bias-related misclassification of anger and fear and separates differently labeled samples more clearly.The comparison visualizes learned features in a two-dimensional t-SNE embedding; the displayed class labels identify seven expressions.Source: 3) Feature visualization · p. 13, Figure 7 · p. 14

VI. CONCLUSION

• ECAN outperforms previous cross-dataset facial-expression recognition results across a series of transfer tasks.The conclusion attributes its strength to bridging marginal and conditional distribution discrepancies, deep-network discriminative power, and class reweighting.Source: §VI · p. 13