Source-linked AI summary
Frame attention networks for facial expression recognition in videos
Debin Meng, Xiaojiang Peng, Kai Wang, Yu Qiao
TL;DR
Video-based facial expression recognition must integrate features from individual frames despite variable video length, while existing aggregation can ignore frame importance. FAN embeds each face with a CNN and adaptively aggregates frames using self- and relation-attention; it achieves state-of-the-art results on CK+ and performance on par with state-of-the-art single models on AFEW.
Problem
Video-based FER needs a fixed video representation from variable-length facial-frame sequences, but existing aggregation methods ignore frame importance.
Method
FAN embeds each face image with a deep CNN and adaptively aggregates frame features using self-attention and relation-attention weights.
Results
FAN achieves state-of-the-art results on CK+ and performance on par with state-of-the-art methods on AFEW.
Takeaways & Limitations
FAN can automatically capture frame importance while producing a single discriminative video representation for FER.
Abstract
from arXiv · showhide
The video-based facial expression recognition aims to classify a given video into several basic emotions. How to integrate facial features of individual frames is crucial for this task. In this paper, we propose the Frame Attention Networks (FAN), to automatically highlight some discriminative frames in an end-to-end framework. The network takes a video with a variable number of face images as its input and produces a fixed-dimension representation. The whole network is composed of two modules. The feature embedding module is a deep Convolutional Neural Network (CNN) which embeds face images into feature vectors. The frame attention module learns multiple attention weights which are used to adaptively aggregate the feature vectors to form a single discriminative video representation. We conduct extensive experiments on CK+ and AFEW8.0 datasets. Our proposed FAN shows superior performance compared to other CNN based methods and achieves state-of-the-art performance on CK+.
1. INTRODUCTION
Video-based facial expression recognition must convert variable-length facial-frame sequences into representations for classifying basic emotions. Existing approaches use several feature-extraction families, but frame aggregation can overlook which frames are most informative, motivating adaptive frame weighting.
- Task and motivation: FER in the wild remains challenging because of head-pose variation, illumination variance, occlusion, and motion blur.The paper motivates improved video representation under these uncontrolled conditions.
- Task and motivation: Video-based FER classifies videos into basic emotions after preprocessing faces, extracting features, and classifying a compact video representation.The pipeline includes face detection, alignment, illumination normalization, feature extraction, and prediction.
- Existing approaches: Feature extraction methods are broadly categorized as static-based, spatial-temporal, and geometry-based approaches.Spatial-temporal methods include LSTM and C3D, while geometry-based methods model facial-landmark motion.
- Existing approaches: Static-based methods require frame aggregation to produce video-level results from varied numbers of frames.Prior methods average or expand frame probabilities, or aggregate feature statistics such as mean, variance, minimum, and maximum.
- Research gap: Existing aggregation methods ignore frame importance, although some frames may be representative of an emotion while others are not.The paper proposes FAN to adaptively aggregate frame features using self-attention and relation-attention kernels learned end-to-end.
2. FRAME ATTENTION NETWORKS
FAN processes a variable-length facial video by embedding each face image and adaptively aggregating frame features. It first estimates self-attention weights, then refines them using relations between individual frames and a global video representation.
- Architecture: FAN uses a deep CNN to embed each face image into a feature vector and produces a fixed-dimension video representation.Its input is a facial video containing a variable number of face images.
- Architecture: The frame attention module learns self-attention and relation-attention weights to aggregate frame features into one discriminative video representation.The two-level weighting is the central aggregation mechanism of FAN.
- Self-attention: Self-attention weights are assigned from individual frame features, then used to aggregate the frames into a global video-level anchor feature.A fully connected layer and sigmoid function produce the coarse weights before aggregation.
- Relation-attention: Relation-attention weights refine frame importance by modeling relations between each frame feature and the global anchor representation.The relation module concatenates local and global features and applies another fully connected layer to estimate the new weights.
- Relation-attention: The final aggregation uses both self-attention and relation-attention weights to form a compact feature.This combines frame-level information with its relation to the video-level representation.
3. EXPERIMENTS
Experiments on CK+ and AFEW 8.0 evaluate FAN across datasets, baselines, attention variants, and backbone strategies. FAN improves the baseline, reaches state-of-the-art on CK+, and performs comparably to the best prior single models on AFEW.
- Datasets and setup: CK+ contains 593 sequences from 123 subjects, including 327 labeled sequences from 118 subjects across seven expressions; evaluation uses 10-fold person-independent cross-validation.The authors construct folds by sampling subject IDs in ascending order with a step size of 10 and report overall accuracy across folds.
- Datasets and setup: AFEW 8.0 contains 1,809 movie and television clips split into Train, Val, and Test sets with 773, 383, and 653 samples, respectively.The splits use mutually exclusive movies and actors, and the test split is not publicly available.
- Evaluation on CK+: 94.8% accuracy from the ResNet18 score-fusion baseline rises to 99.08% with self-attention and 99.69% with added relation-attention on CK+.The baseline is 2.8 percentage points better than the cited latent ordinal model, while the full FAN sets a new state of the art on CK+.
- Evaluation on AFEW 8.0: FAN improves the AFEW baseline by 2.36% and achieves performance comparable to the best previous single model, although one static method performs slightly better.That method uses DenseNet-161, additional pretraining, complicated post-processing, and an SVM classifier.
- Visualization and hyper-parameters: On CK+, final FAN weights assign higher weights to more obvious face frames, whereas self-attention alone can emphasize obscure frames.This visualization is presented as an explanation for the performance gain from adding relation-attention.
- Visualization and hyper-parameters: CK+ accuracy is 99.39% for K={2, 5} and 99.69% for K=8, indicating limited sensitivity to K; fine-tuning VGGFace layers outperforms freezing them.The authors attribute the VGGFace difference to domain discrepancy between FERPlus and CK+ and report further gains from both attention stages.
4. CONCLUSION
The paper concludes that FAN adaptively aggregates frames through self-attention and relation-attention. Experiments show improvements over the baseline, state-of-the-art CK+ results, and performance on par with state-of-the-art single models on AFEW.
- Conclusion: FAN combines self-attention and relation-attention to automatically capture frame importance for video-based facial expression recognition.Its self-attention variant improves the baseline, while relation-attention further boosts performance.