Source-linked AI summary
TransFER: Learning Relation-aware Facial Expression Representations with Transformers
Fanglei Xue, Qiangchang Wang, Guodong Guo
TL;DR
FER must distinguish expressions with similar features while handling substantial variation and invisible facial parts. TransFER combines diverse local-patch discovery with ViT-based relation modeling and attention dropping, and it outperforms state-of-the-art methods on several FER benchmarks.
Problem
FER remains challenging because different expressions can look similar, same-class expressions can vary substantially, and facial parts may be invisible under pose variations or occlusions.
Method
TransFER combines MAD for diverse local patches, ViT-FER for relations among patches, and MSAD for richer self-attention relations.
Results
TransFER outperforms state-of-the-art methods on several FER benchmarks.
Takeaways & Limitations
The model learns rich, diverse relation-aware local representations for facial expression recognition.
Takeaways & Limitations
In the reported experiments, MAD drops only one branch because dropping two or more self-attentions simultaneously did not improve performance.
Abstract
from arXiv · showhide
Facial expression recognition (FER) has received increasing interest in computer vision. We propose the TransFER model which can learn rich relation-aware local representations. It mainly consists of three components: Multi-Attention Dropping (MAD), ViT-FER, and Multi-head Self-Attention Dropping (MSAD). First, local patches play an important role in distinguishing various expressions, however, few existing works can locate discriminative and diverse local patches. This can cause serious problems when some patches are invisible due to pose variations or viewpoint changes. To address this issue, the MAD is proposed to randomly drop an attention map. Consequently, models are pushed to explore diverse local patches adaptively. Second, to build rich relations between different local patches, the Vision Transformers (ViT) are used in FER, called ViT-FER. Since the global scope is used to reinforce each local patch, a better representation is obtained to boost the FER performance. Thirdly, the multi-head self-attention allows ViT to jointly attend to features from different information subspaces at different positions. Given no explicit guidance, however, multiple self-attentions may extract similar relations. To address this, the MSAD is proposed to randomly drop one self-attention module. As a result, models are forced to learn rich relations among diverse local patches. Our proposed TransFER model outperforms the state-of-the-art methods on several FER benchmarks, showing its effectiveness and usefulness.
1. Introduction
TransFER addresses facial expression recognition's difficulty in distinguishing similar expressions and handling varied appearances by learning diverse, relation-aware local representations. It combines MAD, ViT-FER, and MSAD, and reports state-of-the-art performance on several FER benchmarks.
- FER is challenging because different expression classes can share similar facial features, while the same class can vary across races, genders, ages, and cultural backgrounds.
- Global-based methods may neglect critical facial regions, while landmark-based methods can fail under illumination changes, pose variations, and occlusions.
- Attention-based methods may repeatedly emphasize similar facial parts and overlook other discriminative regions, motivating diverse local representations.
- MAD randomly drops attention maps to encourage local CNNs to explore comprehensive and diverse facial patches, including when some parts are invisible.
- ViT-FER models connections among local patches, while MSAD randomly drops one self-attention module to encourage richer relations among them.
- TransFER combines MAD and MSAD and achieves state-of-the-art performance on several FER benchmarks.
2. Related Work
Prior FER research includes handcrafted features, deep-learning losses, region-attention methods, and Transformer-based vision models. TransFER extends this landscape by locating diverse local patches and modeling relations among them, while MSAD regularizes Transformer representations.
- Traditional FER uses handcrafted features such as LBP, HOG, and SIFT, but these features lack generalization under challenging illumination conditions.
- Deep-learning FER methods enhance feature discrimination, weight facial regions, and adaptively capture region importance for occlusion- and pose-variant recognition.
- TransFER differs by locating diverse local patches and capturing relations among them, described as a first attempt for FER.
- Vision Transformers treat images as sequences of patches, and the paper applies Transformers to FER to study relations among local patches.
- MSAD is proposed as a Transformer regularization method that explores rich relations among different local patches.
3. TransFER
TransFER combines local CNNs, Multi-Attention Dropping, a Transformer encoder, and self-attention dropping to learn diverse local patches and rich relations among them for FER.
- Multi-Attention Dropping: MAD randomly zeros one attention branch during training, encouraging local branches to explore complementary facial parts rather than redundant regions.The dropped branch retains the input shape but is not activated in subsequent layers.
- Local CNNs: Local CNNs use multiple LANet branches to generate attention maps that locate diverse facial patches, aggregate them by maximum, and reweight feature maps.Each LANet produces one attention map; the aggregated map is multiplied element-wise with the original feature map.
- ViT-FER: The Transformer converts projected feature maps into a sequence of patch vectors, adds a class token and position embeddings, and processes them with encoder blocks.The encoder uses stacked multi-head self-attention and MLP layers with skip connections, followed by an MLP classification head.
- Multi-head Self-Attention: Multi-head self-attention models relations among local patches, but its heads can form redundant projections without explicit guidance.The MSA runs self-attention operations in parallel and linearly embeds their concatenated outputs.
- Multi-head Self-Attention Dropping: MSAD randomly drops one self-attention module during training, forcing the remaining modules to learn complementary relations among local patches.The paper reports that dropping two or more self-attention modules did not improve performance, so experiments drop only one branch.
4. Experiments
Experiments show that MAD and MSAD improve TransFER’s diverse local representations and performance across FER benchmarks. Ablations identify suitable architectural settings, while visualizations show broader attention to discriminative facial areas.
- Datasets: RAF-DB contains 15,339 single-label expression images, with 12,271 used for training and the remainder for testing.The dataset includes six basic expressions and neutral.
- Datasets: FERPlus contains 28,709 training, 3,589 validation, and 3,589 test images across eight emotion categories, with test accuracy reported.The labels include six basic expressions, neutral, and contempt.
- Module ablation: Adding local CNNs improves performance by (0.1%, 0.11%), while adding MAD yields gains of (0.42%, 0.30%) on RAF-DB and AffectNet.The tuple reports RAF-DB and AffectNet performance, respectively.
- Module ablation: 90.91% and 66.23% are achieved after adding MSAD, improving performance by (0.56%, 0.29%) over the baseline.The reported values correspond to RAF-DB and AffectNet, respectively.
- Stem CNN depth: Stage 3 of IR-50 achieves 90.91% on RAF-DB, while stage 4 reaches 90.32% with more parameters and stage 2 reaches 84.94%.The study seeks both semantic information for localization and detailed information for later extraction.
- Branch number: On RAF-DB, performance peaks at B = 2 with 90.91%; increasing branch number beyond B = 5 reduces performance, while AffectNet performs best at B = 4.Small B limits robust feature-part localization, whereas large B can produce nearly identical branch outputs.
- Drop rates: With p1 fixed at 0.6, performance rises from 89.80% to 90.91% at p2 = 0.3, then falls to 89.24% as p2 increases.The best p2 is smaller than p1, which the authors associate with MSAD’s eight self-attention heads versus MAD’s two branches.
- Dropping strategies: MAD outperforms Dropout, Drop Block, and Spatial Dropout because it drops whole attention maps rather than independently dropping feature-map elements, regions, or channels.A 0.6 Dropout rate yields 39.05% on RAF-DB but 65.51% on AffectNet, whereas MAD remains the strongest method in the comparison.
5. Conclusion
TransFER is a Transformer-based FER architecture designed to learn rich, diverse relation-aware local representations. It combines MAD, ViT-FER, and MSAD, and outperforms state-of-the-art methods on three public FER datasets.
- TransFER uses MAD to guide local CNNs toward diverse local patches, improving robustness to pose variations or occlusions.
- ViT-FER builds rich connections among multiple local patches, assigning higher weights to important facial parts and smaller weights to useless ones.
- MSAD explores richer relations among diverse facial parts by addressing redundancy in multi-head self-attention.
- TransFER outperforms state-of-the-art methods in extensive experiments on three public FER datasets.