PaperlayerView exact revision on arXiv

Source-linked AI summary

Facial Expression Recognition Using Residual Masking Network

Luan Pham, The Huynh Vu, Tuan Anh Tran

arXiv:2603.05937v1cs.CVcs.AI

TL;DR

In-the-wild FER must handle variation across people and imaging conditions, while landmark-based localization can struggle outside lab-controlled settings. This paper embeds U-net-based masking blocks in residual CNNs to refine feature maps, reporting higher FER2013 accuracy than comparison systems.

  • Problem

    In-the-wild FER must handle inter-subject differences and intra-subject changes, while landmark localization often works mainly in lab-controlled conditions.

    Source: §I · p. 1, §II · p. 2

  • Method

    The paper embeds U-net-based localization and masking blocks in residual CNNs to refine feature maps and focus on relevant facial regions.

    Source: §I · p. 1, B. Residual Masking Block · p. 3

  • Results

    76.82% with a six-CNN ensemble was 1% above ensemble-based FER2013 methods; without ensembling, 74.14% exceeded CNN-SIFT’s 73.14%.

    Source: D. Evaluation and Analysis · p. 5

  • Takeaways & Limitations

    The Masking Block can be integrated into existing networks, and the authors created VEMO to evaluate their network.

    Source: §I · p. 1

    Generalization remains a future check, with the authors proposing evaluation on ImageNet.

    Source: §V · p. 7

Abstract

from arXiv · show

Automatic facial expression recognition (FER) has gained much attention due to its applications in human-computer interaction. Among the approaches to improve FER tasks, this paper focuses on deep architecture with the attention mechanism. We propose a novel Masking idea to boost the performance of CNN in facial expression task. It uses a segmentation network to refine feature maps, enabling the network to focus on relevant information to make correct decisions. In experiments, we combine the ubiquitous Deep Residual Network and Unet-like architecture to produce a Residual Masking Network. The proposed method holds state-of-the-art (SOTA) accuracy on the well-known FER2013 and private VEMO datasets. The source code is available at https://github.com/phamquiluan/ResidualMaskingNetwork.

I. INTRODUCTION

In-the-wild facial expression recognition faces substantial variation across people and imaging conditions, despite its applications in human-computer interaction. The paper proposes a Masking Idea and Residual Masking Network to focus CNNs on relevant facial information, and introduces VEMO for evaluation.

I. INTRODUCTION

  • The FER task faces inter-subject differences and intra-subject changes such as occlusions, illumination, and head-pose variation.The paper notes applications including sociable robotics, advertising, consumer behavior, and medical treatment.Source: §I · p. 1
  • Facial cues are concentrated in regions such as the eyes and mouth, while landmark detection can perform poorly in noisy conditions.The proposed Masking Idea uses a U-net-based localization network to refine feature maps and focus attention on selected areas.Source: §I · p. 1
  • Each Masking Block is a U-net variant intended to focus the Residual Masking Network on crucial spatial information for expression classification.The authors describe this as enabling the network to make correct expression classifications.Source: §I · p. 1
  • The paper proposes a Masking Idea as an attention mechanism that can be embedded into CNNs to boost performance.Source: §I · p. 1
  • The authors build a Residual Masking Network for facial expression recognition and create VEMO alongside FER2013 for evaluation.They also state that the Masking Block can be integrated into existing networks.Source: §I · p. 1

II. RELATED WORKS

Prior FER work spans handcrafted features, deep CNNs, and ensemble strategies, while landmark-based localization faces difficulties in uncontrolled conditions. The paper motivates masking as a way to refine feature maps with learned importance weights.

II. RELATED WORKS

  • Traditional FER uses handcrafted features and geometric or appearance descriptors, but landmark detection can be difficult in natural or noisy settings.Traditional methods remain under development, even as deep-learning approaches have become efficient for automatic FER.Source: §II · pp. 1–2
  • Combining traditional methods with deep learning can produce good results, but often requires technical manipulation and is optimized for particular datasets.Source: §II · p. 2
  • Deep FER uses large-scale datasets and varied CNN architectures; ensemble approaches combine networks with different data, architectures, parameters, filters, or layers.The paper notes that deep learning-based networks require many training images to prevent overfitting.Source: §II · p. 2
  • Landmark methods often work best on lab-controlled data, whereas CNN intermediate layers can reveal facial localization and attention can focus on relevant information.The paper identifies eyes, nose, and mouth as facial regions whose combination informs expression recognition.Source: §II · p. 2
  • The paper proposes scoring trunk-branch activations by importance and using a deeper localization network to refine feature tensors with learned weights.The proposed intuition is likened to U-net producing a segmentation mask, with loss tracking the refinement.Source: §II · p. 2

A. Overview

The Residual Masking Network combines residual layers with masking blocks operating at multiple feature-map sizes. Its image pipeline reduces spatial resolution before producing seven expression-state outputs.

A. Overview

  • The figure shows landmark detections and feature maps before and after the third Masking Block.Source: Figure 1 · p. 3
  • The network contains four Residual Masking Blocks, each pairing a Residual Layer with a Masking Block and operating at a different feature size.Source: A. Overview · p. 3
  • A 224 × 224 input is reduced to 56 × 56, then passes through four blocks at 56 × 56, 28 × 28, 14 × 14, and 7 × 7.Average pooling and a seven-way softmax fully connected layer produce outputs for six emotions and neutral.Source: A. Overview · p. 3

B. Residual Masking Block

The Residual Masking Block combines residual feature processing with a masking module that generates weights for feature maps. Its U-Net-based mask refines spatial information, and the block can also serve as an activation in other segmentation architectures.

B. Residual Masking Block

  • The block pairs a Residual Layer for feature processing with a Masking Block that produces corresponding feature-map weights.The method reuses ResNet34 as its backbone and removes the trunk branch.Source: B. Residual Masking Block · p. 3
  • The Masking Block computes a same-size activation map from the residual feature map, which is used to refine that map element-wise.The mask values are specified to lie in [0, 1].Source: B. Residual Masking Block · p. 3
  • The Masking Block uses a U-Net structure with contracting and expansive paths, and its pooling and upsampling depth varies with input spatial size.The authors distinguish this mask structure from other attention modules,.Source: B. Residual Masking Block · p. 3
  • The Masking Block can serve as an activation in segmentation architectures beyond the U-Net-like design used in this method.Source: B. Residual Masking Block · p. 3

C. Ensemble Method

The ensemble procedure combines prediction results from seven different CNNs using a simple unweighted average. Model selection is based on validation accuracy, and the procedure is documented on GitHub.

C. Ensemble Method

A. Dataset

The experiments use the public FER2013 dataset and the VEMO2020 dataset, which the paper describes as private and expected to become public. The passages report FER2013's image count and split, and VEMO2020's sources and labeling process.

A. Dataset

  • FER2013 contains 35,887 grayscale 48×48 images, split into 28,709 training, 3,859 validation, and 3,589 testing images.Source: A. Dataset · p. 3
  • FER2013 has seven emotion categories, with more than 6,000 happiness images but approximately 500 disgust images.The dataset also contains invalid samples, including non-face images and incorrectly cropped faces.Source: A. Dataset · pp. 3–4
  • VEMO2020 contains 36,470 multi-resolution images, including 6,470 collected photos and 30,000 images labeled by professionals.The 6,470-photo portion includes YouTube frames sampled at five frames per second and images from Google Image and Flickr; a group of ten members voted on emotion labels for images in that portion.Source: A. Dataset · p. 4
  • The paper points to Figure 4 for training, validation, and testing statistics and category distributions for the datasets.Source: Figure 4 · p. 4, A. Dataset · p. 5

B. Experiment setup

The experiment setup resizes and converts training images for ImageNet-pretrained models, applies augmentation, and uses shared evaluation settings across networks. The paper also reports the proposed network's processing speed on a specified laptop setup.

B. Experiment setup

  • Training images are resized to 224×224, converted to RGB, and augmented with horizontal flips and rotations from −30° to 30°.The preprocessing adapts images to ImageNet-pretrained models, and augmentation is used to prevent overfitting.Source: B. Experiment setup · p. 5
  • Experiments run for at most 50 epochs with batch size 48 and initial learning rate 0.0001, stopping after eight steps without validation-accuracy improvement.The learning rate is reduced tenfold when validation accuracy does not improve for two consecutive epochs.Source: B. Experiment setup · p. 5
  • Networks are evaluated using the same settings for hyperparameters, preprocessing, augmentation, and evaluation metrics.The stated classification evaluation metric is accuracy; the passage also specifies momentum 0.9 and weight decay 0.001.Source: B. Experiment setup · p. 5
  • The proposed network processes 100 frames per second, each containing a single face, on a laptop with an i7-8750H CPU and GTX 1050Ti.The paper presents this as supporting real-time application.Source: B. Experiment setup · p. 5

C. Visual result explanation

The paper uses Grad-CAM to visualize class-relevant regions in test images, with high activations often appearing near the mouth or eyes. Table II is identified as a FER2013 classifier-performance evaluation.

C. Visual result explanation

  • Grad-CAM uses class gradients routed to the final convolutional layers to generate maps showing which image regions predict a class.The paper applies this approach to explain classification decisions.Source: C. Visual result explanation · p. 5
  • Figure 6 shows Grad-CAM visualizations for images from the testing sets.Source: Figure 6 · p. 5, C. Visual result explanation · p. 5
  • Table II is described as a performance evaluation of classification networks and the proposed method on FER2013.Source: Table II · p. 5
  • High-activation regions often lie near the mouth or eyes, which the authors identify as essential areas for the network's classification decisions.Source: C. Visual result explanation · p. 5

D. Evaluation and Analysis

The Residual Masking Network reports leading FER2013 results in ensemble and non-ensemble settings, and the highest listed accuracy on VEMO. Confusion-matrix scores vary by emotion, while visualizations indicate attention to facial regions including the eyes, nose, and mouth.

D. Evaluation and Analysis

  • The network had higher accuracy than the compared well-known classifiers on FER2013, despite having the highest number of parameters.The comparison used the same training and testing environment and dataset.Source: D. Evaluation and Analysis · p. 5
  • Happy scores were 0.91/0.81 and Surprise 0.85/0.71, while Fear (0.56/0.44) and Sad (0.59/0.63) were among the lowest across the two testing sets.Figure 7's confusion matrices report predicted classes by row and true labels by column; the passage notes that the datasets are unbalanced across emotions.Source: D. Evaluation and Analysis · p. 6, Figure 7 · p. 6
  • Visualized activations before and after the third Masking Block suggest that the Resnet34 heat map favors the eyes, nose, and mouth over other facial regions.The paper also reports that the Masking Block boosts Resnet34 accuracy on both FER2013 and VEMO.Source: D. Evaluation and Analysis · p. 6
  • 76.82% accuracy for the six-CNN ensemble exceeded all ensemble-based FER2013 methods by 1%; non-ensemble accuracy was 74.14% versus CNN-SIFT's 73.14%.Table III covers reported FER2013 accuracy with and without ensemble.Source: D. Evaluation and Analysis · p. 5, Table III · p. 6
  • 65.94% accuracy on VEMO was the highest listed, versus 64.84% for Resnet34, 63.94% for Resnet18, and 60.82% for ResAttNet56.Table IV presents the VEMO comparison.Source: D. Evaluation and Analysis · p. 6, Table IV · p. 6
  • Figure 8 presents examples of incorrect emotion predictions on FER2013 and VEMO.Source: Figure 8 · p. 7

V. CONCLUSION

The paper presents the Residual Masking Network, which applies Masking Blocks across residual layers to improve attention to important information. It reports higher accuracy than well-known systems and FER2013 state-of-the-art results, while identifying generalization and model reduction as future work.

V. CONCLUSION

  • The Residual Masking Network implements the Masking Idea through several Masking Blocks applied across Residual Layers to improve attention to important information.Source: §V · p. 7
  • Experiments showed higher accuracy than well-known classification systems and state-of-the-art reported results on FER2013.Source: §V · p. 7
  • Future work includes evaluating generalization on ImageNet, exploring network and model-parameter reduction, and testing a complete system in an open rehearsal environment.Source: §V · p. 7
Loading 2603.05937v1…