PaperlayerView exact revision on arXiv

Source-linked AI summary

Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognition

Zhengyao Wen, Wenzhong Lin, Tao Wang, Ge Xu

arXiv:2109.07270v6cs.CV

TL;DR

FER must distinguish expressions with similar appearances while integrating cues from multiple facial regions. DAN combines feature clustering, multi-head attention, and attention fusion to address these challenges. It achieves state-of-the-art performance across AffectNet, RAF-DB, and SFEW 2.0.

  • Problem

    FER is difficult because different expressions may look similar, while expression evidence appears simultaneously across multiple facial regions.

  • Method

    DAN uses FCN to improve class separability, MAN to attend to multiple facial regions, and AFN to enforce diverse attentions and fuse their features.

  • Results

    62.09% on AffectNet-8, 65.69% on AffectNet-7, 89.70% on RAF-DB, and 53.18% on SFEW 2.0 are reported as state-of-the-art FER performance.

  • Takeaways & Limitations

    Experiments on three benchmark datasets demonstrate DAN’s superiority for facial expression recognition and provide insights into feature clustering and diverse attention learning.

Abstract

from arXiv · show

We present a novel facial expression recognition network, called Distract your Attention Network (DAN). Our method is based on two key observations. Firstly, multiple classes share inherently similar underlying facial appearance, and their differences could be subtle. Secondly, facial expressions exhibit themselves through multiple facial regions simultaneously, and the recognition requires a holistic approach by encoding high-order interactions among local features. To address these issues, we propose our DAN with three key components: Feature Clustering Network (FCN), Multi-head cross Attention Network (MAN), and Attention Fusion Network (AFN). The FCN extracts robust features by adopting a large-margin learning objective to maximize class separability. In addition, the MAN instantiates a number of attention heads to simultaneously attend to multiple facial areas and build attention maps on these regions. Further, the AFN distracts these attentions to multiple locations before fusing the attention maps to a comprehensive one. Extensive experiments on three public datasets (including AffectNet, RAF-DB, and SFEW 2.0) verified that the proposed method consistently achieves state-of-the-art facial expression recognition performance. Code will be made available at https://github.com/yaoing/DAN.

1 Introduction

The paper identifies two central FER challenges: subtle differences between expressions and the need to model multiple facial regions simultaneously. DAN addresses them with feature clustering, multi-head attention, and attention fusion, achieving state-of-the-art results on three benchmarks.

  • Facial expression categories can share similar underlying facial appearance, making their differences difficult to distinguish.
  • DAN combines FCN, MAN, and AFN to improve feature separability, attend to multiple facial regions, and fuse non-overlapping attentions.FCN optimizes intra-class variation and inter-class margins; MAN attends to multiple regions; AFN encourages distinct locations before fusion.
  • A single attention module cannot capture all subtle and complex appearance variations across different expressions.
  • 62.09% on AffectNet-8, 65.69% on AffectNet-7, 89.70% on RAF-DB, and 53.18% on SFEW 2.0 represent state-of-the-art FER performance.

2 Related Work

Related work frames FER as image classification supported by attention mechanisms and discriminative losses. Prior attention-based FER methods generally use one attention head, whereas DAN uses multiple non-overlapping regions and cascaded spatial-channel attention.

  • FER systems identify human emotional states from facial images, commonly involving face detection, feature extraction, and expression classification.
  • Attention mechanism: Attention mechanisms help neural networks retain valuable spatial or channel information from complex visual inputs.
  • Attention mechanism: DAN cascades spatial and channel attention units within each attention head.
  • Attention mechanism: Earlier FER attention methods generally use one head and attend to rough facial areas, while DAN activates multiple non-overlapping regions.
  • Discriminative loss: Discriminative losses regulate deep-feature distributions by increasing class separability or clustering features around class centers.
  • Discriminative loss: FER studies combine center-based and classification losses, or use cosine-based center objectives, to increase distances between expression categories.

3 Our Approach

DAN combines feature clustering, multi-head attention, and attention fusion to improve facial-expression representations. FCN separates classes, MAN attends to multiple regions, and AFN diversifies and combines those attentions for classification.

  • Overview: The complete DAN pipeline extracts and clusters backbone features, learns diverse attention maps, and outputs class confidence.Its three subnetworks are FCN, MAN, and AFN.
  • Feature Clustering Network (FCN): FCN uses affinity loss to reduce intra-class variation while increasing inter-class margins in backbone features.The loss pushes features toward their class centers while encouraging class centers to remain farther apart than with standard center loss.
  • Multi-Head Attention Network (MAN): MAN uses parallel attention heads, each combining sequential spatial and channel attention to capture multiple facial regions.The spatial unit extracts spatial features, the channel unit extracts channel features, and both are combined into an attention vector.
  • Multi-Head Attention Network (MAN): Each spatial attention unit uses convolution kernels at multiple scales, while its channel unit encodes channel information through pooling and linear layers.The spatial unit uses 1 × 1, 1 × 3, 3 × 1, and 3 × 3 kernels; the channel unit uses global average pooling and two linear layers.
  • Attention Fusion Network (AFN): AFN applies log-softmax scaling and partition loss to emphasize informative regions and reduce overlap among attention heads.The normalized attention feature vectors are then merged for final classification.

4 Experimental Evaluation

The evaluation verifies DAN quantitatively on three benchmark datasets and uses ablation studies to assess both overall performance and component contributions.

  • DAN is evaluated on AffectNet, RAF-DB, and SFEW 2.0 against strong baselines.The experiments assess both benchmark performance and the contribution of individual model components.
  • Ablation studies demonstrate that the model’s various components contribute to its final performance.

4.1 Datasets

The evaluation uses AffectNet, RAF-DB, and SFEW 2.0, which differ in label structure, scale, and dataset construction.

  • AffectNet provides AffectNet-7 and AffectNet-8 branches, with AffectNet-8 adding contempt as an expression category.AffectNet-7 contains 287,401 images, while AffectNet-8 contains 287,568 training images and 4,000 test images.
  • RAF-DB contains more than 29,670 Internet images with seven basic and eleven compound emotion labels.The expression-classification split contains 12,271 training images and 3,068 testing images, aligned and cropped to 100 × 100 pixels.
  • SFEW 2.0 contains seven expression categories across train, validation, and test sets totaling 1,766 samples.Its splits contain 958 training, 436 validation, and 372 test samples, making it smaller than AffectNet and RAF-DB.

4.2 Implementation Details

The experiments use standardized image preparation and training settings, while ablations examine loss functions and attention-head counts on RAF-DB.

  • Implementation Details: All datasets use 224 × 224 input images, with aligned samples and selective augmentation to reduce over-fitting.RAF-DB and AffectNet use official aligned images, while SFEW 2.0 images are manually aligned using RetinaFace.
  • Implementation Details: ResNet-18 is used as the FCN backbone, and models are trained for 40 epochs with batch size 256.The MAN uses four attention heads by default, and the code is implemented in PyTorch.
  • Implementation Details: RAF-DB training uses SGD with learning rate 0.1, whereas AffectNet-7 and AffectNet-8 use ADAM with learning rate 0.0001.Training also balances categories by upsampling low-volume classes and downsampling high-volume classes.
  • Ablation Studies: The affinity loss improves FCN performance over standard center loss, while partition loss is crucial to AFN performance.These loss-function ablations are conducted on RAF-DB with cross-entropy included in all cases.
  • Ablation Studies: Four attention heads outperform a single attention head on RAF-DB and maximize the performance gain in the reported ablation.The four-head setting is therefore used throughout the experiments.

4.3 Quantitative Performance Comparisons

DAN achieves strong accuracy across AffectNet, RAF-DB, and SFEW 2.0, with state-of-the-art results on the two AffectNet branches and RAF-DB.

  • 62.09% accuracy on AffectNet-8 and 65.69% on AffectNet-7 are both superior to existing methods.
  • 89.70% accuracy on RAF-DB is reported as state of the art.
  • 53.18% accuracy on SFEW 2.0 is competitive but not the best reported result.The paper suggests multi-head attention may require larger datasets for effective learning.
  • The reported results demonstrate competitive performance and effective components across multiple datasets.

4.4 Ablation Studies and Computational Complexity

Ablation studies show that the affinity and partition losses improve DAN performance, while four attention heads maximize performance. The four-head model maintains state-of-the-art performance with moderate computational cost.

  • Effects of Loss Functions for FCN and AFN: The affinity loss improves the standard center loss in FCN, while the partition loss is crucial to AFN performance.Both losses contribute to the model’s superior performance on RAF-DB.
  • Effects of the Number of Attention Heads: Four attention heads maximize the performance gain on RAF-DB and outperform a single attention module.The paper therefore uses four attention heads throughout its experiments.
  • Computational Complexity: The method provides competitive performance while maintaining a manageable computational cost.Table 6 compares model size and inference time across methods.
  • Computational Complexity: 19.72 M parameters and 2.23 G FLOPs provide state-of-the-art performance with moderate resource consumption.The comparison concerns model size and inference time against state-of-the-art methods.

4.5 Confusion Matrix

Confusion matrices reveal that DAN performs best on relatively easy expression classes, while disgust, fear, and anger remain more difficult because of class similarities and skewed training distributions.

  • Confusion Matrix: Happy is easiest on AffectNet-8 and AffectNet-7, while happiness, sadness, surprise, neutral, and anger achieve high accuracy on RAF-DB.On SFEW 2.0, happy and neutral perform relatively well, whereas disgust and fear are very challenging.
  • Confusion Matrix: Appearance similarities among expression categories and skewed class distributions are possible reasons for performance gaps across classes.The paper identifies disgust, fear, and anger as classes requiring further effort to reduce confusion.

4.6 Precision–Recall Analysis

On RAF-DB, DAN achieves near-perfect precision for happiness and strong precision for several other classes, exceeding 80% precision at 50% recall for all classes.

  • Precision–Recall Analysis: Over 80% precision at 50% recall is achieved for all RAF-DB classes.The per-class precision–recall curves show particularly strong performance for happiness, neutral, surprise, sadness, and anger.
  • Precision–Recall Analysis: The happiness class achieves near-perfect precision on RAF-DB.Neutral, surprise, sadness, and anger also perform well.

5 Conclusions

DAN combines feature clustering with multiple diverse attentions to improve facial expression recognition. Its experiments demonstrate superiority on three benchmark datasets and suggest relevance for related vision tasks.

  • Conclusions: DAN comprises FCN, MAN, and AFN, which respectively maximize class separability, capture diverse attentions, and penalize overlap while fusing learned features.Together, these sub-networks form the proposed facial expression recognition method.
  • Conclusions: Experiments on three benchmark datasets demonstrate the superiority of DAN for facial expression recognition.The paper presents feature clustering and diverse-attention learning as potentially informative for future related vision research.
Loading 2109.07270v6…