PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Region Ensemble Convolutional Neural Network for Facial Expression Recognition

Yingruo Fan, Jacqueline C. K. Lam, Victor O. K. Li

arXiv:1807.10575v1cs.CVcs.HC

TL;DR

Facial-expression recognition is challenged by limited training data and visual variation. MRE-CNN combines whole-face and local-region CNN inputs through weighted prediction aggregation, and outperforms state-of-the-art methods on AFEW 7.0 and RAF-DB.

  • Problem

    FER databases lack sufficient training data compared with large-scale visual-recognition datasets, resulting in overfitting, while pose, resolution, deformation, and illumination vary.

    Source: §1 · pp. 1–2

  • Method

    MRE-CNN pairs whole-face images with facial subregions in subnetworks and combines their prediction scores by weighted sum.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    MRE-CNN outperforms state-of-the-art facial-expression methods on AFEW 7.0 and RAF-DB.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The study investigates three facial regions, each providing different local information for expression recognition.

    Source: §5 · p. 9

    Grid searching to determine individual subnetwork contribution weights is relatively computationally expensive.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Facial expressions play an important role in conveying the emotional states of human beings. Recently, deep learning approaches have been applied to image recognition field due to the discriminative power of Convolutional Neural Network (CNN). In this paper, we first propose a novel Multi-Region Ensemble CNN (MRE-CNN) framework for facial expression recognition, which aims to enhance the learning power of CNN models by capturing both the global and the local features from multiple human face sub-regions. Second, the weighted prediction scores from each sub-network are aggregated to produce the final prediction of high accuracy. Third, we investigate the effects of different sub-regions of the whole face on facial expression recognition. Our proposed method is evaluated based on two well-known publicly available facial expression databases: AFEW 7.0 and RAF-DB, and has been shown to achieve the state-of-the-art recognition accuracy.

1 Introduction

The introduction frames facial-expression recognition as data-limited and visually challenging, then presents MRE-CNN as a way to combine whole-face and local-region information. It reports stronger performance than state-of-the-art methods on AFEW 7.0 and RAF-DB.

1 Introduction

  • FER databases lack the training scale of datasets such as ImageNet, resulting in overfitting; head pose, resolution, deformation, and illumination also make FER challenging.The introduction lists depression treatment, customer-satisfaction measurement, fatigue surveillance, and human-robot interaction as applications.Source: §1 · pp. 1–2
  • The paper investigates facial subregions as CNN inputs, addressing prior emphasis on whole-face processing.Each subnetwork processes a whole-face image together with a subregion image.Source: §1 · p. 2
  • The proposed MRE-CNN combines global whole-face information with local facial characteristics, and aggregates subnetwork prediction scores using a weighted sum.The authors say this weighted sum can improve recognition rate compared with the original single network.Source: §1 · p. 2
  • MRE-CNN outperforms state-of-the-art facial-expression methods on AFEW 7.0 and RAF-DB.The introduction presents this as the framework’s benchmark result.Source: §1 · p. 2

2 Related Work

Prior CNN-based FER work explored ensemble learning, transfer learning, and architectural changes, but generally processed the entire face rather than directly using facial subregions.

2 Related Work

  • Hu et al. added a Supervised Scoring Ensemble learning block to a CNN to improve prediction accuracy.This inspired incorporating other learning strategies into mainstream networks.Source: §2 · p. 3
  • Transfer learning in used two-stage supervised fine-tuning of an ImageNet-pretrained CNN to address limited FER training data.Source: §2 · p. 3
  • Inception layers and network-in-network theory were applied to FER in, focusing on network architecture.Source: §2 · p. 3
  • Most prior methods processed the whole face, while few directly cropped facial subregions as CNN inputs.Source: §2 · p. 3

3 The Proposed Method

MRE-CNN pairs whole-face and local-region inputs and combines subnet predictions for facial expression recognition. It outperforms some state-of-the-art methods on both RAF-DB and AFEW 7.0.

3.1 Data Pre-processing

  • Faces are detected and aligned using 68 landmarks; the method crops the whole face and left-eye, nose, and mouth regions, resizing them to 224×224 pixels.AFEW clips contribute 3–10 clear-face frames sampled at adaptive intervals.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Multi-Region Ensemble Convolutional Neural Network

  • Three subnetworks pair the whole face with the left eye, nose, or mouth, then combine their prediction scores using a weighted sum.Each subnet uses softmax loss; the framework produces a final emotion prediction from the three subnet predictions.Source: §3.2 · pp. 4–5, §3.3 · p. 6

3.3 The Sub-networks in MRE-CNN Framework

  • The VGG-16 subnetwork uses 13 convolutional and five max-pooling layers, concatenating two pool5 outputs before the first fully connected layer; a modified AlexNet is also evaluated.VGG-16 is fine-tuned on each dataset, whereas the modified AlexNet is trained without pretrained models and uses smaller fully connected layers.Source: §3.3 · pp. 5–6

4.2 Implementation Details

  • Training augmentation expands RAF-DB samples fifteen-fold, and the ensemble uses different region weights for VGG-16 and AlexNet.VGG-16 weights are 4/7 for left eye, 2/7 for mouth, and 1/7 for nose; AlexNet weights are 2/5, 2/5, and 1/5, respectively.Source: §3.2 · p. 5, §4.2 · p. 6

4.3 Results on RAF-DB

  • MRE-CNN with VGG-16 outperforms the existing state-of-the-art methods evaluated on RAF-DB.The comparison uses the mean diagonal value of the confusion matrix.Source: §4.3 · p. 7, Table 2 · p. 7
  • Face+LeftEye performs better than Face+Mouth, while Face+Nose is less effective but still outperforms VGG-FACE using the whole face alone.The paper says the nose result may reflect less emotion-related information in that region.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 Results on AFEW 7.0

  • MRE-CNN with VGG-16 achieves results superior to some state-of-the-art methods on AFEW 7.0 validation videos.Table 4 reports average accuracy across validation videos.Source: Table 4 · p. 8, §4.4 · p. 9

4.5 Discussions

  • Feature maps show shallower layers capturing profile information and deeper layers encoding semantic information; grid-searching subnet contributions is relatively computationally expensive.The authors suggest combining facial expression models with audio, physiological, or thermal infrared data as a research direction.Source: §4.5 · p. 9

5 Conclusion

The conclusion presents MRE-CNN as a three-subnetwork framework that combines predictions and examines different facial regions, reporting appealing performance on RAF-DB and AFEW 7.0 compared with state-of-the-art methods.

  • The framework combines predictions from three subnetworks by assigning each a weight.Source: §5 · p. 9
  • The study investigates three facial regions, each providing different local information.Source: §5 · p. 9
  • MRE-CNN achieves appealing performance on RAF-DB and AFEW 7.0 compared with other state-of-the-art methods.Source: §5 · p. 10
Loading 1807.10575v1…