PaperlayerView exact revision on arXiv

Source-linked AI summary

Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition

Kai Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, Yu Qiao

arXiv:1905.04075v2cs.CV

TL;DR

Real-world occlusion and pose variation make FER difficult, while this setting has received relatively limited attention and lacks suitable annotated datasets. The paper builds six test datasets and proposes RAN with RB-Loss to weight facial regions adaptively. Experiments report improved performance under occlusion and pose variation, plus state-of-the-art results on four popular datasets.

  • Problem

    Occlusion and pose variations significantly alter facial appearance, but real-world FER has received less attention partly because annotated datasets are scarce.

  • Method

    The paper builds six real-world test datasets and combines a Region Attention Network with Region Biased Loss to learn important facial-region weights.

  • Results

    The method improves FER under occlusion and pose variation and achieves state-of-the-art results on FERPlus, AffectNet, RAF-DB, and SFEW.

  • Takeaways & Limitations

    RAN and RB-Loss provide a tested approach for FER in real-world occlusion and pose-variant conditions across newly built and established datasets.

Abstract

from arXiv · show

Occlusion and pose variations, which can change facial appearance significantly, are two major obstacles for automatic Facial Expression Recognition (FER). Though automatic FER has made substantial progresses in the past few decades, occlusion-robust and pose-invariant issues of FER have received relatively less attention, especially in real-world scenarios. This paper addresses the real-world pose and occlusion robust FER problem with three-fold contributions. First, to stimulate the research of FER under real-world occlusions and variant poses, we build several in-the-wild facial expression datasets with manual annotations for the community. Second, we propose a novel Region Attention Network (RAN), to adaptively capture the importance of facial regions for occlusion and pose variant FER. The RAN aggregates and embeds varied number of region features produced by a backbone convolutional neural network into a compact fixed-length representation. Last, inspired by the fact that facial expressions are mainly defined by facial action units, we propose a region biased loss to encourage high attention weights for the most important regions. We validate our RAN and region biased loss on both our built test datasets and four popular datasets: FERPlus, AffectNet, RAF-DB, and SFEW. Extensive experiments show that our RAN and region biased loss largely improve the performance of FER with occlusion and variant pose. Our method also achieves state-of-the-art results on FERPlus, AffectNet, RAF-DB, and SFEW. Code and the collected test data will be publicly available.

I. INTRODUCTION

Real-world occlusion and pose variation substantially alter facial appearance and remain underexplored in FER, partly because suitable annotated datasets are scarce. The paper addresses this gap with six test datasets, a Region Attention Network, and a region-biased loss.

  • Occlusions and variant poses significantly change facial appearance, creating major challenges for real-world facial expression recognition.
  • Real-world FER has received less attention partly because datasets with occlusion and pose annotations are lacking.
  • The authors build six real-world test datasets from FERPlus, AffectNet, and RAF-DB, with manual occlusion annotations and automatically labeled pose variation.
  • The Region Attention Network learns region importance end-to-end, aggregates CNN features into a fixed-length representation, and improves FER under occlusion and pose variation.
  • The Region Biased Loss encourages important facial regions to receive higher attention than the original face and further improves FER without additional computation cost.

II. RELATED WORK

Prior FER research spans engineered and learned features, supervised classification, transfer learning, and attention mechanisms. The paper positions RAN as a region-focused attention approach that aggregates facial features into a compact representation.

  • FER systems commonly comprise face detection, feature extraction, and expression recognition, using engineered or learned representations.
  • Engineered FER features include texture-based local descriptors, geometry-based global features, and hybrid combinations.
  • Learned FER systems typically feed extracted features into supervised classifiers such as SVMs, softmax layers, or logistic regression.
  • Pretraining on face-recognition datasets is used to reduce overfitting when facial-expression datasets are small.
  • Real-world occlusion and pose variation remain relatively underexplored compared with standard FER and artificial-occlusion settings.
  • Attention mechanisms have been applied to image classification, translation, face detection, and video representation, including feature aggregation into compact representations.
  • RAN differs from related attention aggregation by adaptively capturing the importance of facial regions for pose- and occlusion-robust FER.

III. METHODOLOGY

The paper proposes a Region Attention Network that adaptively weights facial regions, refines those weights using global context, and aggregates region features into a compact representation. A region biased loss further encourages discriminative facial crops to receive higher attention than the original face.

  • Region Attention Network: RAN combines region cropping and feature extraction, self-attention, and relation-attention modules to address occlusion and pose variation.The network is designed to reduce the influence of occluded or irrelevant regions while balancing regional and global facial information.
  • Region generation: RAN accepts facial regions generated through fixed-position, random, or landmark-based cropping before attention-based aggregation.The figure illustrates fixed-position, random, and landmark-based region generation strategies.
  • Self-attention module: Self-attention estimates coarse weights from individual region features and summarizes them into a compact global representation.The weighted representation can serve as the classifier input.
  • Relation-attention module: Relation-attention refines regional weights by modeling relationships between each region feature and the aggregated global representation.This stage produces the final compact RAN representation from region and global features.
  • Region biased loss: RB-Loss imposes a margin constraint requiring the maximum crop attention weight to exceed the original-face weight.It is jointly optimized with classification loss and further improves FER slightly without additional computation cost.

C. Region Generation

The RAN generates multiple facial regions using fixed-position, random, or landmark-based cropping. The design balances feature diversity against the discriminative strength of each crop.

  • Region generation: The paper evaluates fixed-position, random, and landmark-based cropping as three region-generation schemes.These schemes are illustrated in Figure 2.
  • Design trade-off: Region size requires a trade-off because oversized crops reduce feature diversity, whereas undersized crops weaken discrimination.The paper identifies crop sizing as a fundamental design issue for RAN.
  • Fixed position cropping: Fixed-position cropping uses five aligned regions, including top-left, top-right, and center-down face regions at a 0.75 scale ratio.Two additional center regions follow a prior smile-classification setup.
  • Random cropping: Random cropping samples N regions with sizes ranging from 0.7 to 0.95 of the original face scale.The regions are generated with random positions and sizes.
  • Landmark-based cropping: Landmark-based cropping uses five MTCNN facial landmarks to define regions around the eyes, nose, and mouth corners.Regions extending beyond the original image are removed.

D. Occlusion and Pose Variant Dataset

The authors construct six real-world occlusion and pose-variant test datasets from FERPlus, AffectNet, and RAF-DB. Occlusion categories are manually assigned, while pose subsets use pitch and yaw thresholds above 30 degrees.

  • Dataset construction: Six test datasets cover occlusion and pose variation across FERPlus, AffectNet, and RAF-DB.The datasets are intended to evaluate FER under real-world conditions where publicly available annotations were limited.
  • Pose datasets: Pose subsets include faces whose pitch or yaw angle exceeds 30 degrees, using OpenFace estimates while excluding roll after alignment.The pose criterion is applied to FERPlus, AffectNet, and RAF-DB.
  • Occlusion datasets: Occlusion subsets are manually annotated for masks, glasses, left/right objects, upper-face objects, bottom-face objects, and non-occlusion.Images with at least one occlusion type are selected.
  • Dataset statistics: Upper-face occlusion has the fewest samples among the occlusion types across FERPlus, AffectNet, and RAF-DB.Table I reports the statistics of the collected test datasets.
  • Dataset statistics: 605, 682, and 735 occlusion samples come from FERPlus, AffectNet, and RAF-DB, representing 16.86%, 17.05%, and 23.9% of their original sets.The source splits are FERPlus test, AffectNet validation, and RAF-DB test.

IV. EXPERIMENTS

The experiments evaluate the collected occlusion and pose-variant datasets, analyze RAN components on established FER datasets, and compare the method with state-of-the-art approaches.

  • Experiments: The experimental section evaluates RAN on collected occlusion and pose-variant test datasets, studies its components, and compares it with state-of-the-art methods.Component analyses use FERPlus, AffectNet, and SFEW.

A. Datasets

The study evaluates four popular in-the-wild FER datasets and constructs additional occlusion- and pose-variant test subsets from three of them.

  • The evaluation uses FERPlus, AffectNet, RAF-DB, and SFEW, covering different dataset scales and challenging facial conditions.
  • Additional occlusion and pose-variant subsets are built from FERPlus, AffectNet, and RAF-DB, using annotated challenging conditions and a standardized 224×224 aligned-face preprocessing pipeline.
  • FERPlus contains 28,709 training, 3,589 validation, and 3,589 test images, with eight expression labels and majority-voting or label-distribution supervision.
  • AffectNet contains more than one million Internet images, including 450,000 manually annotated images with eight basic expression labels.
  • SFEW provides 958 training, 436 validation, and 372 test images from unconstrained video frames with varied poses, occlusions, resolution, and illumination.
  • RAF-DB contains 30,000 facial images, with experiments using 12,271 training and 3,068 test images labeled for basic emotions.

C. FER with occlusion and variant pose in the wild

On real-world occlusion and pose-variant test sets, RAN consistently outperforms CNN baselines, with especially large gains under occlusion and severe pose changes.

  • RAN improves baseline accuracy by 10.3%, 10.02%, and 2.53% on Occlusion-FERPlus, Occlusion-AffectNet, and Occlusion-RAF-DB, respectively.
  • For poses larger than 30 degrees, RAN gains 4.12%, 3.09%, and 2.70% on Pose-FERPlus, Pose-AffectNet, and Pose-RAF-DB, respectively.
  • For poses larger than 45 degrees, the gains increase to 4.9%, 5.4%, and 2.05% on Pose-FERPlus, Pose-AffectNet, and Pose-RAF-DB, respectively.
  • RAN consistently boosts happiness, surprise, and sadness categories across the test sets, which have facial action units represented by clear regional features.
  • On FED-RO, RAN achieves 67.98%, exceeding the reported 66.5% result of the comparison method.
  • RAN outperforms naive score fusion and augmentation training, indicating that its integrated regional aggregation provides more than simple region combination.
  • With RB-Loss, attention shifts toward visible or informative regions, including a center crop or up-right region when corresponding areas are occluded.

D. Ablation study on FERPlus and AffectNet

Ablations show that RAN benefits from its attention modules, region-biased loss, and aggregation strategy, while region generation, region size, and inference cost impose practical trade-offs.

  • Attention modules: With face alignment, the attention modules improve baselines by 0.83% on FERPlus and 1.85% on AffectNet.
  • Attention modules: Without face alignment, adding relation attention improves the baseline by 1.13% on FERPlus and 3.05% on AffectNet.
  • Region biased loss: RB-Loss consistently improves FERPlus and AffectNet performance, including a 0.92% improvement on AffectNet without face alignment and 59.5% with oversampling.
  • Region biased loss: Increasing RB-Loss margin α from 0 to 0.02 improves performance, whereas larger margins degrade it quickly, underscoring the importance of the original face image.
  • Fusion schemes: On FERPlus without alignment, score fusion improves the baseline by 0.54%, while RAN improves it by 1.35%; individual crops and simple feature fusion do not improve similarly.
  • Region generation: Fixed cropping consistently outperforms landmark-based and random cropping, while increasing random crops eventually degrades performance after 30 crops.
  • Region size: Reducing region size to 0.4 of default causes significant degradation, whereas increasing it to 1.1 is evaluated as a near-default alternative.
  • Inference time: Average inference time is 0.025s for RAN versus 0.006s for the baseline, despite GPU parallelism preventing a linear increase with region count.

E. Comparison with the state-of-the-art methods

The RAN achieves strong results across FERPlus, AffectNet, SFEW, and RAF-DB, including state-of-the-art performance on the reported benchmarks. Results include gains over baselines and improvements from RB-Loss and score fusion.

  • Comparison on FERPlus: 89.16% accuracy is achieved on FERPlus with RAN, establishing a new state of the art to the authors’ knowledge.
  • Comparison on AffectNet: 59.5% validation accuracy is achieved on AffectNet with oversampling, compared with 52.97% without oversampling.
  • Comparison on SFEW: 56.4% validation accuracy is obtained on SFEW by averaging ResNet18 and VGG16 scores, while the single model achieves 54.19%.
  • Comparison on RAF-DB: 86.9% overall accuracy is achieved on RAF-DB’s basic emotion categories, exceeding DLP-CNN by 2.77% and another method by 1.83%.

V. CONCLUSION

The paper targets facial expression recognition under real-world occlusion and pose variation by introducing new test datasets, RAN, and RB-Loss.

  • The authors build several new FER test datasets for real-world occlusion and pose-variant conditions.
  • RAN adaptively adjusts the importance of facial parts for occlusion- and pose-variant FER.
  • RB-Loss encourages high attention weights for the most important facial regions.
Loading 1905.04075v2…