Source-linked AI summary
Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition
Fanglei Xue, Qiangchang Wang, Zichang Tan, Zhongsong Ma, Guodong Guo
TL;DR
FER in the wild is difficult, while existing ViT approaches face limited-data, noise, and computation challenges. The paper introduces parameter-free APP and ATP attentive-pooling modules to retain discriminative patches and tokens, reporting state-of-the-art performance across six in-the-wild FER datasets.
Problem
FER in the wild is challenging, and ViT-based FER methods can be limited by noisy or occluded features and small training datasets.
Method
APViT combines APP, which selects informative CNN patches, with ATP, which progressively keeps relevant Transformer tokens without extra learnable parameters.
Results
APViT outperforms state-of-the-art methods on six major in-the-wild FER datasets.
Takeaways & Limitations
Attentive pooling focuses APViT on discriminative features while discarding less relevant ones and reducing computation.
Takeaways & Limitations
The reported comparison includes poor performance for the fear and disgust categories, which the authors attribute to few training samples.
Abstract
from arXiv · showhide
Facial Expression Recognition (FER) in the wild is an extremely challenging task. Recently, some Vision Transformers (ViT) have been explored for FER, but most of them perform inferiorly compared to Convolutional Neural Networks (CNN). This is mainly because the new proposed modules are difficult to converge well from scratch due to lacking inductive bias and easy to focus on the occlusion and noisy areas. TransFER, a representative transformer-based method for FER, alleviates this with multi-branch attention dropping but brings excessive computations. On the contrary, we present two attentive pooling (AP) modules to pool noisy features directly. The AP modules include Attentive Patch Pooling (APP) and Attentive Token Pooling (ATP). They aim to guide the model to emphasize the most discriminative features while reducing the impacts of less relevant features. The proposed APP is employed to select the most informative patches on CNN features, and ATP discards unimportant tokens in ViT. Being simple to implement and without learnable parameters, the APP and ATP intuitively reduce the computational cost while boosting the performance by ONLY pursuing the most discriminative features. Qualitative results demonstrate the motivations and effectiveness of our attentive poolings. Besides, quantitative results on six in-the-wild datasets outperform other state-of-the-art methods.
1 INTRODUCTION
FER in the wild is difficult because expressions vary substantially within and across identities, while ViT-based FER methods often struggle with limited data, noisy features, and excessive computation. The paper proposes APP and ATP to retain discriminative patches and tokens while discarding irrelevant features.
- 1 INTRODUCTION: FER is challenging because the same identity can show different expressions, while different identities, ages, genders, and poses can express the same emotion.
- 1 INTRODUCTION: ViT-based FER methods generally underperform state-of-the-art CNNs because they require substantial training data and lack inductive bias.Existing FER datasets are much smaller than general image-classification datasets.
- 1 INTRODUCTION: The proposed APViT combines APP and ATP to progressively remove noisy features before downstream Transformer computation.ATP decreases token number after a specific layer, offering more controllable reduction than CNN max pooling.
- 1 INTRODUCTION: APP selects distinguishable local patches from CNN feature maps, while ATP retains top-k relevant Transformer tokens to reduce noise, occlusion effects, and computation.The two modules are designed to focus recognition on informative features without adding learnable parameters.
- 1 INTRODUCTION: The paper reports that APViT outperforms state-of-the-art methods on several challenging in-the-wild FER datasets.
2 RELATED WORK
Prior FER and visual-transformer methods use attention, masks, or token selection to handle informative regions, but often continue computing background or occluded features. APViT instead directly discards unimportant features through parameter-free attentive pooling while using CNN and Transformer components.
- 2 RELATED WORK: The model combines APP for CNN feature maps with ATP for ViT tokens, progressively pooling both representations.Figure 2 contrasts CNN max pooling, fixed-token ViT, gradually shrinking ATP, and the combined APViT design.
- 2 RELATED WORK: Earlier attention-based FER methods use landmarks, local attention, or separate masking and classification Transformers to address occlusion and complex backgrounds.
- 2 RELATED WORK: APViT directly discards unimportant background and occlusion features, unlike methods that merely reduce their weights while retaining them in subsequent computation.The method uses the original top-K operation instead of differentiable selection and reports that a hand-designed mask generator performs better than a learning-based one.
- 2 RELATED WORK: Standard ViT applies Transformer attention to visual tokens and has achieved strong image-classification results after large-scale pre-training and downstream fine-tuning.
- 2 RELATED WORK: Unlike prior ViT-based approaches, the proposed attentive pooling methods add no extra training parameters and use pre-trained weights while reducing computation time.
3 PROPOSED METHOD
The method introduces attentive pooling modules that select informative CNN patches and Transformer tokens while discarding noisy features. APP uses attention-based patch selection, whereas ATP progressively reduces patch-token counts using Transformer attention to preserve discriminative information and reduce computation.
- 3.2 Attentive Patch Pooling: APP selects the top-k patch tokens from CNN feature maps using an attention map, dropping uninformative regions before Transformer processing.The attention map assigns importance to spatial positions, and the corresponding highest-scoring features are retained.
- 3.2 Attentive Patch Pooling: APP defines pooling criteria that reduce feature-map channels to one, using hand-designed operations such as absolute-value sums or learned generators.The hand-designed FABS criterion emphasizes locations where multiple channels have high activations.
- 3.2 Attentive Patch Pooling: Multiplying feature maps by attention weights does not remove noisy patches, so APP directly selects relevant features instead of merely shrinking irrelevant activations.Low-weight locations remain computed by downstream modules under conventional attention weighting.
- 3.3 Attentive Token Pooling: ATP uses Transformer attention to rank patch tokens by their relevance to the [class] token, then retains only the top-k patch tokens while always preserving [class].The pooling criterion aggregates attention across heads, and pooling acts only on patch tokens.
- 3.3 Attentive Token Pooling: ATP is applied in the second half of Transformer blocks with a shared keep rate, progressively reducing token counts to limit redundant or noisy representations.With M blocks, the output token count is approximately k · r^M/2, ignoring the [class] token.
4 EXPERIMENTS
APViT improves facial-expression recognition while reducing computation, with attentive pooling benefits supported by ablations and state-of-the-art comparisons across multiple in-the-wild datasets.
- Ablation Studies: 66.94% on AffectNet improves the baseline from 65.91% using only 83% of its FLOPs.Combining APP and ATP produces the strongest result in the ablation study.
- Ablation Studies: APP and ATP individually improve performance, while their combination yields the best results and APP reduces FLOPs earlier than ATP.APP gains 0.39% on RAF-DB and 0.60% on AffectNet; ATP gains 0.26% and 0.74%, respectively.
- Ablation Studies: 45.12% of baseline FLOPs raises AffectNet performance from 65.91% to 66.48% with only five retained tokens.The minimal keep configuration still outperforms the unpooled baseline, although overly aggressive reduction eventually lowers performance.
- Ablation Studies: Pre-training is especially important for CNN features, increasing RAF-DB performance from 81.85% to 90.84%, while ViT pre-training helps when CNN pre-training is also used.The reported effects depend on whether CNN pre-trained weights are loaded.
- Ablation Studies: Hand-designed attention generation outperforms FLANet, with FABS reaching 91.98% on RAF-DB and 66.94% on AffectNet.The authors attribute the advantage to avoiding feature-magnitude changes from multiplying attention maps with features.
- Comparison with the State-of-the-Art Methods: 91.98% on RAF-DB outperforms state-of-the-art methods, including TransFER at 90.91%.The comparison uses the proposed APViT and reports the highest recognition performance among the cited methods.
5 CONCLUSION
The paper proposes APViT, combining two attentive pooling modules to retain discriminative features and discard less relevant ones in FER. Across six major in-the-wild FER datasets, APViT outperforms state-of-the-art methods, while visualizations support the robustness of the pooling strategy.
- APViT combines two attentive pooling modules to focus on discriminative features and discard less relevant ones, reducing attention to occlusions and noise.
- APViT outperforms state-of-the-art methods on six major in-the-wild FER datasets.
- Visualizations demonstrate the intuition and robustness of the proposed attentive poolings, while decreasing token numbers is identified as a promising direction.