PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning

Tianshui Chen, Tao Pu, Hefeng Wu, Yuan Xie, Lingbo Liu, Liang Lin

arXiv:2008.00923v8cs.CV

TL;DR

Inconsistent dataset and feature-extractor choices make cross-domain FER methods difficult to compare fairly. The paper builds a unified evaluation benchmark and proposes AGRA, which combines graph propagation with adversarial learning to adapt holistic and local features. AGRA outperforms previously identified best performers in the reported unified benchmark comparison.

  • Problem

    Cross-domain FER comparisons use inconsistent source-target datasets and feature extractors, making actual method improvements difficult to assess amid dataset domain shifts.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs a unified evaluation benchmark and proposes AGRA, which uses intra- and inter-domain graphs with adversarial learning to co-adapt holistic and local features.

    Source: §1 · pp. 2–3, §4.1 · p. 8

  • Results

    4.02% higher accuracy averaged across CK+, JAFFE, FER2013, SFEW2.0, and ExpW than previously identified best performers, using RAF-DB as source and ResNet-50.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The unified benchmark supports fair comparison, while the 54,901-sample AFE dataset can be used to promote cross-culture FER evaluation.

    Source: §1 · p. 3

    RAF-DB 2.0 was unavailable, so the re-implementation used RAF-DB and compared against reported results obtained with RAF-DB 2.0.

    Source: Credibility of Re-implementation · p. 19

Abstract

from arXiv · show

To address the problem of data inconsistencies among different facial expression recognition (FER) datasets, many cross-domain FER methods (CD-FERs) have been extensively devised in recent years. Although each declares to achieve superior performance, fair comparisons are lacking due to the inconsistent choices of the source/target datasets and feature extractors. In this work, we first analyze the performance effect caused by these inconsistent choices, and then re-implement some well-performing CD-FER and recently published domain adaptation algorithms. We ensure that all these algorithms adopt the same source datasets and feature extractors for fair CD-FER evaluations. We find that most of the current leading algorithms use adversarial learning to learn holistic domain-invariant features to mitigate domain shifts. However, these algorithms ignore local features, which are more transferable across different datasets and carry more detailed content for fine-grained adaptation. To address these issues, we integrate graph representation propagation with adversarial learning for cross-domain holistic-local feature co-adaptation by developing a novel adversarial graph representation adaptation (AGRA) framework. Specifically, it first builds two graphs to correlate holistic and local regions within each domain and across different domains, respectively. Then, it extracts holistic-local features from the input image and uses learnable per-class statistical distributions to initialize the corresponding graph nodes. Finally, two stacked graph convolution networks (GCNs) are adopted to propagate holistic-local features within each domain to explore their interaction and across different domains for holistic-local feature co-adaptation. We conduct extensive and fair evaluations on several popular benchmarks and show that the proposed AGRA framework outperforms previous state-of-the-art methods.

1 INTRODUCTION

The introduction identifies domain shift and inconsistent evaluation settings as barriers to reliable cross-domain FER comparisons. It presents a unified benchmark, an Asian-face dataset, and AGRA, which combines graph propagation with adversarial learning to adapt holistic and local features.

1 INTRODUCTION

  • Domain shifts arise from subjective annotations, collection environments, and racial differences, while inconsistent source-target datasets and feature extractors hinder fair algorithm comparisons.Using different source datasets alone produced a performance disparity of more than 8.0% on CK+ for ICID [24].Source: §1 · pp. 1–2
  • The benchmark re-implements leading methods and compares them using unified source-target datasets and feature extractors; the accompanying AFE dataset contains 54,901 labeled samples mainly from Asian individuals.AFE is used as a source dataset to evaluate cross-culture FER performance.Source: §1 · pp. 2–3
  • Local facial features are presented as more transferable across datasets and more informative for fine-grained adaptation than holistic features alone.The introduction cites mouth-corner and eye regions as examples of local cues.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The feature-distribution illustration reports that AGRA gathers same-category samples from different domains more closely than the holistic-only adversarial baseline [31].The authors interpret this pattern as indicating more discriminative domain-invariant features.Source: Figure 2 · p. 3
  • AGRA builds intra-domain and inter-domain graphs, initializes nodes with extracted holistic-local features and learnable per-class distributions, then uses stacked GCNs for feature interaction and co-adaptation.The propagation is intended to progressively mitigate holistic-local feature shifts between source and target domains.Source: §1 · p. 3
  • AGRA improves accuracy averaged across CK+, JAFFE, FER2013, SFEW2.0, and ExpW by 4.02% over previously identified best performers using RAF-DB as source and ResNet-50.The comparison uses the unified benchmark settings described by the authors.Source: §1 · p. 3

2 RELATED WORKS

The related-work review covers cross-domain FER, adversarial domain adaptation, and graph representation learning. It positions AGRA against approaches that often use inconsistent evaluation settings or adapt holistic features without modeling local-feature relations.

2 RELATED WORKS

  • The review is organized around cross-domain FER, adversarial domain adaptation, and graph representation learning.Source: §2 · p. 3

2.1 Cross-Domain Facial Expression Recognition

  • Prior CD-FER approaches include subspace transfer, auxiliary target data, sample generation, and conditional adaptation; Yan et al. require target annotations, unlike Zheng et al..The section also notes that differing datasets and feature extractors make comparisons difficult, motivating a unified benchmark.Source: §2.1 · p. 4

2.2 Adversarial Domain Adaptation

  • Adversarial adaptation methods learn domain-invariant features but mainly focus on holistic content, whereas AGRA models holistic-local correlations with graphs and graph propagation.The cited methods include Tzeng et al. and Long et al..Source: §2.2 · p. 4

2.3 Graph Representation Learning

  • Graph neural networks learn graph-structured representations through iterative message propagation, and prior visual applications model relationships to contextualize features or predictions.This motivates using graph neural networks to capture holistic-local interactions within and across domains.Source: §2.3 · p. 4

3 UNIFIED EVALUATION BENCHMARK

The section presents a unified benchmark to compare cross-domain FER methods under consistent dataset and feature-extractor choices. It also analyzes how those choices affect measured accuracy and describes the datasets, methods, and evaluation protocols.

3.1 Datasets for CD-FER

  • The benchmark covers lab-controlled and in-the-wild datasets, with AFE adding 54,901 well-labeled samples from Asian individuals for cross-culture evaluation.The seven datasets can each serve as a source or target; the paper highlights RAF-DB and AFE as representative sources.Source: §3.1 · p. 5, §3.4 · p. 6
  • More than 70% accuracy gaps arise from different source datasets on the same target, while different targets with the same source produce gaps above 30%.The comparison trains a ResNet-50 baseline on each dataset and tests across datasets without fine-tuning.Source: §3.1 · p. 5, Figure 4 · p. 6

3.2 Feature Extractors for CD-FER

  • Different backbones produce an accuracy gap of approximately 10% with the same source and target datasets.The cross-dataset evaluation trains on RAF-DB and tests on CK+, JAFFE, SFEW2.0, FER2013, and ExpW.Source: §3.2 · p. 6

3.3 Algorithms for CD-FER

  • The benchmark re-implements six CD-FER methods, adapts six domain-adaptation methods, and adds direct-transfer and pseudo-label fine-tuning baselines.The paper reports fourteen competing algorithms, including AGRA.Source: §3.3 · p. 6, §3.4 · p. 6

3.4 Unified Evaluation Protocols

  • Unified protocols align source/target datasets and feature extractors, and competing methods receive concatenated holistic and local features for fair comparison.The framework illustration shows two graphs and stacked GCNs propagating holistic-local information within and across domains.Source: §3.4 · p. 6, Figure 6 · p. 7

4.1 Overview

Cross-domain FER uses labeled source data to classify unlabeled target data, where dataset distributions differ. The section outlines adversarial adaptation and AGRA’s graph-based holistic-local feature co-adaptation.

4.1 Overview

  • The task has labeled source samples and unlabeled target samples drawn from different distributions, and the learned model must perform well on the target.The framework builds on adversarial learning to learn domain-invariant features.Source: §4.1 · p. 7
  • Adversarial adaptation trains the feature extractor to fool a domain discriminator, while the discriminator distinguishes source from target samples.A classifier trained on labeled source samples is then used to classify samples from both domains.Source: §4.1 · p. 7
  • Local features can be more transferable than holistic features and provide detail for fine-grained FER; experiments report lower MMD for local features and generally lower still for holistic-local features.The comparison uses RAF-DB features against CK+, JAFFE, FER2013, SFEW2.0, and ExpW.Source: §4.1 · pp. 7–8
  • AGRA extracts holistic and local-region features, initializes graph nodes with input features or learned per-class distributions, then propagates messages through two stacked GCNs.The intra-domain graph models feature interaction, while the inter-domain graph transfers information for holistic-local co-adaptation.Source: §4.1 · p. 8

4.2 Graph Construction

AGRA represents holistic and local facial-region relationships with intra-domain and inter-domain graphs. The graphs share nodes but encode different connection types.

4.2 Graph Construction

  • The node set represents each domain’s holistic face and five local regions: left eye, right eye, nose, and both mouth corners.The regions are selected based on facial landmarks and FACS-defined action-unit locations.Source: §4.2 · p. 8
  • The intra-domain adjacency encodes holistic-to-local and local-to-local links, while inter-domain adjacency also includes holistic-to-holistic links across domains.Different values denote different connection types.Source: §4.2 · p. 8
  • Both graphs use the same node set; their adjacency matrices distinguish within-domain from cross-domain connections.The two graph structures correlate holistic-local features within and across domains.Source: §4.2 · p. 8

4.3 Graph Representation Adaptation

AGRA uses intra- and inter-domain graphs to propagate holistic and local features, while class-aware feature distributions initialize graph nodes. Two stacked GCNs model within-domain interactions and cross-domain co-adaptation.

4.3 Graph Representation Adaptation

  • Two stacked GCNs propagate messages through intra-domain graphs for holistic-local interactions and inter-domain graphs for holistic-local co-adaptation.The resulting features feed the expression classifier for source samples and the domain discriminator for domain estimation.Source: §4.3 · p. 8, §4.3.2 · pp. 8–9

4.3.1 Class-aware two-stage updating mechanism

  • Class-aware distributions are initialized from per-domain feature clusters, then updated each iteration and reclustered every ten epochs.The updating mechanism combines iteration-level moving averages with epoch-level reclustering.Source: §4.3.1 · p. 8

4.3.2 Stacked graph convolutional networks

  • Source-domain graph nodes start from extracted holistic and local features, while target-domain nodes use distributions from the closest target cluster.Intra-domain propagation produces features that initialize nodes in the inter-domain graph before further propagation.Source: §4.3.2 · p. 9

4.4 Implementation Details

AGRA extracts holistic and landmark-centered local features, trains the feature extractor, classifier, and domain discriminator in two stages, and uses nearest class distributions to initialize graph nodes at inference.

4.4.1 Network architecture

  • The intra-domain GCN uses two layers and the inter-domain GCN one, which the ablation study identifies as the best layer setting.Holistic and local representations are each 64-dimensional, forming a 384-dimensional feature vector for the seven-class classifier.Source: §4.4.1 · p. 9

4.4.2 Training details

  • Training first fits the feature extractor and classifier with source-label cross-entropy, then adversarially trains the domain discriminator and fine-tunes the feature extractor and classifier.Backbones start from MS-Celeb-1M pretraining; the initial stage runs approximately 15 epochs.Source: §4.4.2 · p. 9

4.4.3 Inference details

  • At inference, AGRA matches the input to the nearest source-domain class distribution, initializes source graph nodes with it, and predicts from the propagated feature.The input's holistic and local images initialize the target-domain nodes.Source: §4.4.3 · p. 9

5 EXPERIMENTS

The experiments present results for re-implemented algorithms and AGRA under a fair evaluation setting, alongside an ablation study. Figures 7 and 8 compare average accuracies across source-dataset and backbone choices.

5 EXPERIMENTS

  • The experiment section presents AGRA and re-implemented algorithms under a fair evaluation setting and includes an ablation study.The supplied section passages do not state the ablation findings.Source: §5 · pp. 9, 11
  • Table 2 covers accuracies on CK+, JAFFE, SFEW2.0, FER2013, and ExpW, generated with the same source dataset and backbone for the methods.The caption does not provide individual accuracy values.Source: Table 2 · p. 10
  • Figures 7 and 8 present average accuracy comparisons across source-dataset and backbone choices.Figure 7 compares RAF-DB and AFE sources across three backbones; Figure 8 compares the three backbones for RAF-DB and AFE sources.Source: Figure 7 · p. 11, Figure 8 · p. 11

5.1 Performance Evaluation and Analysis

AGRA outperforms competing methods on nearly all evaluated source-target and backbone combinations, and achieves the best mean accuracy across all six reported source-backbone choices. The analysis also finds performance varies with dataset and backbone selection.

5.1.1 Effect of the inconsistent choice

  • AFE as source performs better on JAFFE, whereas RAF-DB performs well on CK+, SFEW2.0, FER2013, and ExpW.The authors suggest the pattern may reflect greater distribution similarity between Japanese-captured JAFFE and AFE, and between the other datasets and RAF-DB.Source: §5.1.1 · p. 11
  • For RAF-DB with ResNet-50, SWD accuracies range from 52.06% to 75.19% across target datasets; average performance also decreases from ResNet-50 to ResNet-18 to MobileNet-v2.The authors attribute the backbone pattern to successively weaker discrimination and generalization abilities.Source: §5.1.1 · p. 11

5.1.2 Comparison of the AGRA with current state-of-the-art algorithms

  • AGRA consistently outperforms current methods for almost all source-target datasets and backbones, and has the best mean accuracy for all six reported source-backbone choices.With RAF-DB and ResNet-18, its accuracies are 77.52% on CK+, 61.03% on JAFFE, 52.75% on SFEW2.0, 54.94% on FER2013, and 69.70% on ExpW.Source: §5.1.2 · p. 11

5.2 Ablation Study

Ablations test the contributions of holistic-local co-adaptation, statistical distributions, graph design, adversarial learning, and computational cost under a shared evaluation setup.

5.2.1 Analysis of holistic-local feature co-adaptation

  • Removing holistic-local co-adaptation lowers mean accuracy from 66.13% to 59.68%, a 6.45% decrease across the five datasets.The comparison removes the co-adaptation module while keeping the other components unchanged.Source: Table 3 · p. 11, §5.2.1 · p. 12
  • Concatenating holistic-local features improves mean accuracy by 0.31%, but remains 6.14% below AGRA’s mean accuracy.The comparison covers CK+, JAFFE, SFEW2.0, FER2013, and ExpW.Source: Table 3 · p. 11, §5.2.1 · p. 12
  • Using one GCN instead of two lowers mean accuracy by 4.64%; removing the inter-domain or intra-domain GCN lowers it by 2.19% or 4.79%, respectively.The authors attribute the single-GCN drop to the difference between within-domain and cross-domain message propagation.Source: §5.2.1 · p. 12, Table 4 · p. 12

5.2.2 Analysis of the per-class statistical distributions

  • Replacing per-class statistical distributions with dataset-level distributions lowers mean accuracy from 66.13% to 63.03%; updating every iteration or reclustering every ten epochs lowers it by 2.50% or 3.48%.The authors report that same-category samples from CK+ and SFEW2.0 gather together across training stages.Source: §5.2.2 · pp. 12–13, Table 5 · p. 12

5.2.3 Analysis of the adjacency matrix

  • Random and all-one adjacency matrices lower mean accuracy by 6.65% and 3.32%; fixing the prior matrices lowers it from 66.13% to 56.01%.The authors report that jointly fine-tuning the matrices learns dataset-specified matrices.Source: §5.2.3 · p. 13, Table 6 · p. 13

5.2.4 Analysis of GCN

  • Two intra-domain GCN iterations and one inter-domain iteration perform best; deeper intra-domain propagation or neural-tensor and transformer fusion perform worse than GCN-based fusion.Increasing intra-domain iterations from one to two boosts performance, but increasing them to three reduces it.Source: §5.2.4 · p. 13, Table 7 · p. 13, Table 8 · p. 13

5.2.5 Contribution of adversarial learning

  • Without adversarial learning, accuracies fall to 68.22%, 49.30%, 52.98%, 56.46%, and 63.93% on the five datasets.The ablation trains the model using classification loss alone.Source: §5.2.5 · p. 13, Table 9 · p. 14

5.2.6 Efficiency analysis

  • Adding GCNs increases ResNet-50 inference MACs from 6.7700G to 6.7702G (0.00295%) and adds 2.54 ms; training time per epoch rises about 5% to 20%.The inference-time comparison uses a batch size of one on a GTX 1070 Ti; training-time measurements use RAF-DB and CK+ on the same GPU.Source: §5.2.6 · p. 14, Table 10 · p. 14, Table 11 · p. 14

6 CONCLUSION

The conclusion presents a unified benchmark and a new Asian-face dataset alongside AGRA, which combines graph propagation and adversarial learning for holistic-local co-adaptation.

6 CONCLUSION

  • The benchmark unifies source and target datasets and feature extractors to support fair comparisons among competing CD-FER methods.The work also analyzes how inconsistent choices affect CD-FER performance.Source: §6 · p. 14
  • The new AFE dataset contains 54,901 well-labeled samples captured mainly from Asian individuals.The conclusion says AFE is added to the benchmark.Source: §6 · p. 14
  • AGRA integrates graph propagation with adversarial learning to co-adapt holistic and local representations across domains, and experiments demonstrate its effectiveness on the unified benchmark.Source: §6 · p. 14

SUPPLEMENTARY MATERIALS

Supplementary experiments examine implementation credibility, training choices, AFE as a source for JAFFE, and AGRA’s performance with additional source datasets.

Credibility of Re-implementation

  • Re-implementations of ICID, DFA, LPL, DETN, FTDNN, and ECAN achieve comparable or better reported results, with some comparisons constrained by unavailable original datasets.FTDNN is re-implemented using CK+ and MMI rather than its original six source datasets; RAF-DB 2.0 was unavailable for ECAN.Source: Credibility of Re-implementation · pp. 18–19
  • For ECAN, using RAF-DB instead of unavailable RAF-DB 2.0 produces slightly worse results than the original reported comparison.RAF-DB 2.0 supplements RAF-DB with additional anger, disgust, and fear images.Source: Credibility of Re-implementation · p. 19

Analysis of Different Optimizers

Contribution of Pretraining

Effectiveness of using AFE source dataset on JAFFE

Performance Analyses Using Different Source Datasets

Loading 2008.00923v8…