Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion

Fuyan Ma; Bin Sun; Shutao Li

Source-linked AI summary

Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion

Fuyan Ma, Bin Sun, Shutao Li

arXiv:2103.16854v3Mar 31, 2021cs.CV

TL;DR

FER in the wild is difficult because unconstrained conditions create occlusions, pose variation, deformation, blur, and information-deficient regions, while many earlier methods targeted lab-controlled data. VTFF fuses LBP and CNN features with attentional selective fusion, converts them into visual words, and applies Transformer global self-attention; the authors report superior performance across three in-the-wild datasets and promising CK+ cross-dataset generalization.

Problem
FER in the wild remains difficult under occlusions, variant poses, deformation, motion blur, and other unconstrained conditions, despite progress on lab-controlled data.
Method
VTFF fuses LBP and CNN features with attentional selective fusion, converts fused maps into visual words, and models their relationships with a multi-layer Transformer encoder.
Results
VTFF exceeds other state-of-the-art methods on RAF-DB, FERPlus, and AffectNet, with promising generalization in cross-dataset evaluation on CK+.
Takeaways & Limitations
Global self-attention shifts attention toward discriminative visual words, supporting expression recognition from a global perspective under challenging conditions.
Takeaways & Limitations
Directly adding LBP features provides limited benefit because simple addition fusion is unsatisfactory for combining LBP and CNN features.

Abstract

from arXiv · show

Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging two kinds of feature maps generated by two-branch CNNs. The ASF captures discriminative information by fusing multiple features with the global-local attention. The fused feature maps are then flattened and projected into sequences of visual words. Second, inspired by the success of Transformers in natural language processing, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Under the same settings, extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%. The cross-dataset evaluation on CK+ shows the promising generalization capability of the proposed method.

1 INTRODUCTION

FER in the wild remains difficult because unconstrained images contain occlusions, pose variation, deformation, blur, and irrelevant backgrounds. VTFF addresses these challenges by fusing complementary features and modeling facial images as visual-word sequences.

Unconstrained FER is challenged by occlusions, variant head poses, face deformation, motion blur, and complex backgrounds.
Lab-controlled FER methods often degrade dramatically on real-world datasets such as RAF-DB, FERPlus, and AffectNet.Controlled datasets typically contain frontal images with limited illumination changes and occlusions.
VTFF translates facial images into visual-word sequences and recognizes expressions from a global perspective.The approach is motivated by modeling relationships among visual components rather than relying only on local regions.
The method combines LBP and CNN features using global-local attention, then applies global self-attention to focus on discriminative visual words.
Experiments evaluate VTFF on RAF-DB, FERPlus, AffectNet, including occlusion and pose subsets, with cross-dataset evaluation on CK+.

2 RELATED WORK

Related work progresses from handcrafted and deep features toward feature fusion and Transformer-based representations for FER. Existing fusion methods combine complementary cues, while Transformers provide a global modeling perspective.

Deep learning methods have surpassed traditional handcrafted features for FER performance.Earlier approaches used HOG, LBP, NMF, and sparse representation, whereas later methods learned representations with deep networks.
Feature Fusion for FER: Feature fusion methods combine handcrafted, deep, dynamic-texture, geometric, or acoustic features to improve FER representations.
Feature Fusion for FER: Prior LBP-CNN fusion approaches used concatenation or dense connections but did not fully exploit complementary information or remove redundancy.
Transformers for Vision: Transformers have achieved strong results in NLP and have been applied to diverse computer-vision tasks.

Multi-layer Transformer Encoder×𝑁𝑙

VTFF extracts and fuses visual features, converts them into visual words, and uses Transformer self-attention to model relationships among feature components before classification.

Visual words extraction: VTFF extracts feature maps with pre-trained ResNet18 backbones and fuses them through attentional selective fusion.
Visual words extraction: Fused feature maps are flattened across spatial dimensions and projected into visual words.
Relationship modeling: A multi-layer Transformer encoder models relationships between visual feature components using self-attention.
Expression classification: The network produces expression probabilities with a softmax classifier after relationship modeling.

3 METHOD

VTFF combines RGB and LBP features through attentional selective fusion, then represents the fused map as position-aware visual words processed by a multi-layer Transformer encoder for expression prediction.

Overview: VTFF uses two pre-trained ResNet18 branches to extract feature maps from RGB and LBP facial images.The branches process complementary input representations before feature fusion.
Attentional Selective Fusion: ASF combines RGB and LBP features with global and local attention to emphasize discriminative information.Global average pooling captures inter-channel relationships, while pixel-wise convolution preserves subtle local details.
Visual Word Embeddings: The fused feature map is flattened, linearly projected into a visual embedding sequence, and augmented with a learnable classification token and positional embeddings.The projection maps 512-dimensional features to 768-dimensional embeddings, with sequence length H_dW_d before adding [cls].
Multi-Layer Transformer Encoder: The Transformer encoder models interactions among all visual words using multi-head self-attention across multiple encoder layers.Queries, keys, and values project the sequence into multiple representation subspaces, while layer normalization and an MLP are applied within each encoder block.
Prediction: The output [cls] token represents the whole feature sequence and is passed through a fully connected layer to produce facial-expression probability scores.The predicted expression is selected using arg max during inference.

4 EXPERIMENTS

Experiments show that VTFF achieves strong performance across standard, occlusion, pose-variant, and cross-dataset FER evaluations. Ablations attribute gains to the Transformer encoder, ASF, and LBP features, while larger Transformer configurations can overfit.

Standard benchmarks: 88.14% accuracy on RAF-DB establishes VTFF as the reported state-of-the-art, with gains of 18.8% over VGG and 1.11% over SCN.The method achieved the best results except for the surprise category.
Standard benchmarks: 88.81% accuracy on FERPlus exceeds RAN by 0.96% and SCN by 0.80% under the same experiment settings.VTFF also outperformed an aligned RAN variant without using face alignment.
Standard benchmarks: 61.85% accuracy on AffectNet was obtained with oversampling, while VTFF exceeded SCN by 1.62% using only AffectNet for training.On AffectNet-7, VTFF improved mean class accuracy over other methods by 0.74% to 5.94%.
Robustness and generalization: VTFF showed superior performance on occlusion- and pose-variant subsets, supporting visual-word recognition from a global perspective under these conditions.The evaluation covered six subsets derived from RAF-DB, FERPlus, and AffectNet.
Robustness and generalization: Cross-dataset evaluation on CK+ showed better performance than previous approaches when training separately on RAF-DB, FERPlus, and AffectNet.The AffectNet-trained model predicted one additional expression, contempt, compared with some seven-class baselines.
Ablation studies: 0.16%, 0.47%, and 0.20% improvements indicate that LBP features provide useful texture information for FER across RAF-DB, FERPlus, and AffectNet.The paper notes that simple addition is unsatisfactory for combining LBP and CNN features.
Ablation studies: 0.61%, 0.38%, and 1.23% gains over element-wise addition show that ASF improves fusion of LBP and CNN features across the evaluated datasets.ASF aggregates global and local contexts and also surpassed concatenation under the reported condition.
Ablation studies: 1.23%, 0.85%, and 2.05% improvements over the baseline on RAF-DB, FERPlus, and AffectNet respectively were obtained by integrating the multi-layer Transformer encoder.The encoder contributed more to accuracy improvements than the LBP features and ASF in the reported ablation.

5 CONCLUSION

VTFF addresses in-the-wild facial expression recognition by combining adaptive feature fusion with Transformer-based global modeling of visual words. It outperforms state-of-the-art methods across three datasets and shows promising cross-dataset generalization on CK+.

5 CONCLUSION: VTFF translates facial images into visual-word sequences and recognizes expressions from a global perspective.The approach combines attentional selective fusion with global self-attention over the resulting visual words.
5 CONCLUSION: Attentional selective fusion dynamically combines LBP and CNN features to improve recognition accuracy.The fused feature maps are flattened and projected into visual words.
5 CONCLUSION: The Transformer encoder uses global self-attention to shift attention toward discriminative visual words.This models relationships among visual words from a global perspective.
5 CONCLUSION: VTFF exceeds other state-of-the-art methods on RAF-DB, FERPlus, and AffectNet.The conclusion reports superior performance across all three frequently used facial-expression datasets.
5 CONCLUSION: Cross-dataset evaluation on CK+ demonstrates the method's promising generalization ability.The evaluation tests models trained on the in-the-wild datasets directly on CK+.