PaperlayerView exact revision on arXiv

Source-linked AI summary

Facial Expression Recognition using Convolutional Neural Networks: State of the Art

Christopher Pramerdorfer, Martin Kampel

arXiv:1612.02903v1cs.CV

TL;DR

The paper examines why CNN-based facial expression recognition results vary and identifies bottlenecks in architectures and datasets. An ensemble of modern deep CNNs achieves 75.2% FER2013 test accuracy without auxiliary training data or face registration.

  • Problem

    Reliable facial expression recognition under naturalistic conditions remains unsolved because head pose, illumination, occlusions, and subtle unposed expressions complicate recognition.

  • Method

    The paper reviews CNN-based FER methods, empirically compares CNN architectures and learned features, and identifies bottlenecks affecting performance.

  • Results

    75.2% FER2013 test accuracy is achieved by an ensemble of 8 CNNs, with modern deep models outperforming the best shallow model by almost 2% under identical conditions.

  • Takeaways & Limitations

    Modern deep CNN architectures can substantially improve FER performance, while larger FER datasets remain a major bottleneck for further progress.

  • Takeaways & Limitations

    Results on FER2013 and other FER datasets are only indicative of real-world FER performance because of dataset bias.

Abstract

from arXiv · show

The ability to recognize facial expressions automatically enables novel applications in human-computer interaction and other areas. Consequently, there has been active research in this field, with several recent works utilizing Convolutional Neural Networks (CNNs) for feature extraction and inference. These works differ significantly in terms of CNN architectures and other factors. Based on the reported results alone, the performance impact of these factors is unclear. In this paper, we review the state of the art in image-based facial expression recognition using CNNs and highlight algorithmic differences and their performance impact. On this basis, we identify existing bottlenecks and consequently directions for advancing this research field. Furthermore, we demonstrate that overcoming one of these bottlenecks - the comparatively basic architectures of the CNNs utilized in this field - leads to a substantial performance increase. By forming an ensemble of modern deep CNNs, we obtain a FER2013 test accuracy of 75.2%, outperforming previous works without requiring auxiliary training data or face registration.

I. INTRODUCTION

Automatic facial expression recognition supports human-computer interaction and data analytics, but reliable recognition in naturalistic conditions remains unresolved. The paper reviews CNN-based methods, compares their methodological choices, and reports improved performance from modern deep CNN ensembles.

  • I. INTRODUCTION: Naturalistic FER remains challenging because head pose, illumination, occlusions, and subtle unposed expressions vary substantially.Controlled recognition of basic expressions is considered solved, whereas realistic conditions remain difficult.
  • I. INTRODUCTION: CNNs are used for feature extraction and inference in several recent FER methods, but their architectures, preprocessing, and training and test protocols differ significantly.These differences prevent performance impacts from being assessed from reported results alone.
  • I. INTRODUCTION: The paper reviews CNN-based FER methods, highlights their differences, and compares CNN architectures empirically under consistent settings.It uses this analysis to identify bottlenecks and directions for improving FER performance.
  • I. INTRODUCTION: The study focuses on predicting basic expressions from single images using CNNs.The paper notes that image-based methods can be adapted to image sequences by integrating per-frame results with graphical models.
  • I. INTRODUCTION: FER2013 contains 35,887 grayscale 48 × 48 face crops with basic-expression labels and realistic variation in age, pose, and other factors.The dataset is split into training, validation, and test sets of 28,709, 3,589, and 3,589 samples, respectively.

A. Overview

Prior CNN-based FER work uses diverse ensembles, preprocessing, data sources, and feature-integration strategies. These methods include augmentation, heterogeneous datasets, registration, and pose-related features to improve recognition or generalization.

  • A. Overview: Early CNN-based FER methods used ensembles trained on FER data, while later methods added augmentation, learned ensemble weighting, or larger CNN committees.The reported approaches differ in how they diversify models and integrate predictions.
  • A. Overview: Some methods train on multiple posed and naturalistic datasets to increase available training data and improve cross-dataset generalization.One method uses data compiled from multiple datasets, while another uses heterogeneous labels from several datasets.
  • A. Overview: The review organizes methodological differences around preprocessing operations, CNN architecture, and CNN training and inference.Table I summarizes preprocessing operations across the compared methods.
  • A. Overview: Registration and pose information are used in some methods, including selective registration and deep-network-based pose representations.One study reports a small FER performance increase of about 0.4% from utilizing pose information captured by deep networks.

B. Methodological Differences

The reviewed FER methods differ in face handling and illumination correction before CNN processing. Face detection, landmark-based registration, and normalization are used inconsistently across methods.

  • B. Methodological Differences: Preprocessing typically includes face detection, registration to compensate for pose variation, and illumination correction.The standard pipeline applies these operations once to each image.
  • B. Methodological Differences: Only two reviewed methods perform face detection, while the others rely on dataset-provided face crops.This difference is summarized in Table I.
  • B. Methodological Differences: Landmark-based registration can improve FER when landmarks are reliably detected, but challenging poses and partial occlusions can make detection inaccurate.Methods address this problem with confidence-based or selective registration strategies.
  • B. Methodological Differences: Most reviewed methods apply illumination correction, using normalization, histogram equalization, linear plane fitting, or method-specific ensemble variations.The exact correction procedure differs across methods.

2) CNN Architecture:

The compared CNNs vary widely in layer composition, depth, parameter count, and training protocols, while most remain shallow relative to architectures used in related fields. The study therefore evaluates whether modern architectures can improve FER under consistent conditions.

  • 2) CNN Architecture:: Table II encodes CNN architecture using convolutional, pooling, response-normalization, Inception, and fully connected layer types.Parameter counts assume single-channel 48 × 48 FER2013 inputs.
  • 2) CNN Architecture:: Compared architectures differ substantially in layer composition, depth, and parameter count, and most are shallow relative to related fields.The source papers generally do not explain why these architectures were selected.
  • 2) CNN Architecture:: Deeper networks do not necessarily have more parameters and may impose a stronger prior that combats overfitting.The paper also notes that modern deep CNNs perform well on similarly sized datasets such as CIFAR10.
  • 2) CNN Architecture:: Training and inference differ in auxiliary data, additional features, augmentation, test-time transformations, and ensemble prediction integration.Table III summarizes these protocol differences, including registration and data augmentation choices.

C. Reported Results on FER2013

Reported FER2013 results suggest that shallow CNNs, face registration, data augmentation, and ensemble voting are associated with stronger performance, although methodological differences limit direct comparisons.

  • The three best-performing works use comparatively shallow CNNs with depths of 5 and 6.
  • The deepest and most modern CNN performs worst on this dataset, but direct comparison is not possible because it was trained on a superset of FER2013.
  • The three best-performing methods use face registration, suggesting registration is beneficial under challenging conditions.Facial landmark extraction is inaccurate for about 15% of FER images, according to.
  • Data augmentation and ensemble voting are identified as important for improving generalization performance.
  • Ensemble voting reportedly improves test accuracy by 2-3%.

III. EMPIRICAL COMPARISON

Reported FER2013 results show that top-performing methods use shallow, basic CNNs, but architecture cannot be isolated from other influential factors; the paper therefore compares architectures empirically under consistent settings.

  • The best-performing FER methods use shallow and basic CNNs, contrasting with deeper networks in related fields.
  • CNN architecture is one of many factors influencing FER performance, so reported results alone cannot establish its individual impact.
  • The empirical comparison trains and tests all compared CNN architectures using the same protocols.
  • The comparison adds batch normalization after convolutional and fully connected layers and dropout after the first fully connected layer.These additions are made for robustness to suboptimal network initialization.
  • Experiments use FER2013’s official training, validation, and test sets with provided face crops and histogram equalization.
  • Landmark-based face registration is omitted to avoid registration errors and require CNNs to compensate for pose variations.

3) CNN Training and Inference:

CNN training uses standardized optimization and augmentation protocols, while a shared MLP backend enables a more direct comparison of learned feature quality and representation depth.

  • Every architecture is trained for up to 300 epochs using SGD with momentum 0.9 and cross-entropy loss.The initial learning rate, batch size, and weight decay are 0.1, 128, and 0.0001, respectively.
  • Training augmentation uses horizontal mirroring and random 48 by 48 pixel crops after zero-padding.All CNNs receive the same augmented samples in the same order for fair comparison.
  • A grid search selects an optimal dropout rate for each architecture because their original regularization settings were designed for different training conditions.
  • The best validation model for each architecture is evaluated on the test set using standard ten-crop oversampling.
  • Feature quality is compared by replacing each trained CNN’s fully connected backend with a two-layer MLP containing 1,024 hidden units.The MLP learns FER from features extracted by each pretrained CNN frontend.
  • Fixing the backend removes backend differences and enables study of how network depth affects learned representations.

B. Results and Discussion

Measured results show that shallow CNNs generally outperform deeper ones, while modern deep architectures remain promising. The paper identifies limited, non-task-specific augmentation and insufficient dataset scale as major bottlenecks.

  • Results and Discussion: Auxiliary training data can significantly improve FER performance, but dataset bias must be addressed.Additional data explains some performance differences, while one reported case shows that it can negatively affect FER2013 performance.
  • Results and Discussion: About 3% lower accuracy was measured for method than its reported single-CNN result, despite testing both stochastic and max pooling.The authors could not reproduce the reported accuracy under their evaluation.
  • Results and Discussion: Shallower CNN architectures outperform deeper ones, including in learned feature quality.The comparison does not establish that modern deep networks are unsuitable because only one architecture qualifies as modern and its design choices are questioned.
  • Current Bottlenecks: Current FER architectures are basic and shallow compared with state-of-the-art architectures in related fields.The paper identifies comparatively simple CNN architecture as a bottleneck and tests modern deep architectures as a possible remedy.
  • Current Bottlenecks: General augmentation such as random crops and mirroring is not optimized for FER, whereas face-aware augmentation has potential to compensate for limited dataset size.Suggested directions include registration, frontalized samples, and synthesized poses during training and inference.
  • Current Bottlenecks: The largest stated bottleneck is the absence of a publicly available FER dataset large by current deep-learning standards.FER2013 contains 35,887 images, while related fields have benefited from datasets containing hundreds of thousands or millions of images.

IV. DEEP CNNS FOR FER

The paper tests whether replacing the comparatively shallow CNN architectures used in FER with modern deep architectures improves FER2013 accuracy. The comparison uses the same dataset, preprocessing, and training and testing protocols as earlier experiments.

  • Modern deep CNN architectures are evaluated to overcome the comparatively shallow and basic architectures of current FER methods.
  • The experiments use the exact same dataset, preprocessing, and training and testing protocols as Section III for a fair comparison.
  • Table IV reports ten-crop test accuracy on FER2013 for the tested deep architectures.

1) CNN Architectures:

The study evaluates three modern deep CNN designs inspired by architectures from related fields and examines their parameter efficiency and ensemble potential. The designs include VGG-, Inception-, and ResNet-based networks.

  • CNN Architectures:: Three CNNs are considered, with architectures summarized in Table IV and inspired by state-of-the-art designs from related fields.
  • CNN Architectures:: The VGG-based model removes one CCP block, adds dropout after each block, and uses a 1024-unit hidden backend layer.Dropout improved validation accuracy by around 1%.
  • CNN Architectures:: The Inception-based model uses a more consistent structure without initial strided convolutions or pooling because the input images are already small.
  • CNN Architectures:: The ResNet-based model removes the initial CP block, narrows the final residual group to 256 feature maps, and adds dropout after final pooling.
  • CNN Architectures:: VGG and Inception use fewer parameters than all pertinent literature architectures despite being significantly deeper, while the deep ResNet also uses fewer parameters than most.
  • CNN Architectures:: The study seeks to confirm that modern deep architectures generally perform well rather than optimize specifically for the absolute best FER2013 accuracy.
  • CNN Architectures:: An exhaustive search identifies optimal ensembles of up to eight models using FER2013 validation accuracy.

B. Results and Discussion

Under identical conditions, modern deep architectures outperform the best shallow model, and an eight-model ensemble reaches 75.2% FER2013 test accuracy without auxiliary data or face registration. The paper notes that FER2013 results remain indicative rather than fully representative of real-world performance because of dataset bias.

  • B. Results and Discussion: The best modern deep model outperforms the best shallow model by almost 2% under identical conditions, and every tested architecture exceeds the shallow baseline.Even Inception achieves this while using only half as many parameters.
  • B. Results and Discussion: An ensemble of eight deep CNNs achieves 75.2% FER2013 test accuracy, performing comparably to the current best known method.
  • B. Results and Discussion: The deep-model ensemble obtains state-of-the-art performance without additional training data or features, comprehensive data augmentation, or face registration.The authors expect auxiliary training data and comprehensive FER-specific augmentation could improve performance further.
  • B. Results and Discussion: FER2013 results are only indicative of real-world FER performance because of dataset bias, a limitation applying to this study and FER research generally.
  • B. Results and Discussion: The paper reviews CNN-based FER methods, identifies bottlenecks, and shows that modern deep CNNs substantially improve FER2013 performance without additional training data or face registration.
  • B. Results and Discussion: The authors expect that addressing remaining bottlenecks, especially FER-specific data augmentation, will yield further substantial performance improvements.
Loading 1612.02903v1…