PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Facial Expression Recognition: A Survey

Shan Li, Weihong Deng

arXiv:1804.08348v2cs.CV

TL;DR

Deep FER must address limited training data and expression-unrelated variation as recognition moves into challenging real-world conditions. This paper systematically surveys datasets, pipelines, architectures, training strategies, benchmarks, applications, and future directions for static and dynamic FER. It reports that temporal modeling has become increasingly prevalent and can improve performance on widely evaluated benchmarks.

  • Problem

    Deep FER faces insufficient training data, overfitting risk, and nuisance variation from illumination, pose, identity, and other real-world factors.

  • Method

    The paper provides a systematic survey of deep FER datasets, preprocessing and classification pipelines, neural architectures, training strategies, applications, challenges, and future directions.

  • Results

    Temporal dependency modeling improves performance on benchmarks such as CK+ and MMI, and 5 of 7 reviewed EmotiW 2017 systems used deep spatio-temporal networks.

  • Takeaways & Limitations

    Robust deep FER requires attention to data quantity and quality, nuisance variations, dataset bias, class imbalance, and both spatial and temporal information.

  • Takeaways & Limitations

    Large-scale expression datasets still face noisy or difficult annotation, and few datasets jointly cover real-world variation with detailed facial attributes.

Abstract

from arXiv · show

With the transition of facial expression recognition (FER) from laboratory-controlled to challenging in-the-wild conditions and the recent success of deep learning techniques in various fields, deep neural networks have increasingly been leveraged to learn discriminative representations for automatic FER. Recent deep FER systems generally focus on two important issues: overfitting caused by a lack of sufficient training data and expression-unrelated variations, such as illumination, head pose and identity bias. In this paper, we provide a comprehensive survey on deep FER, including datasets and algorithms that provide insights into these intrinsic problems. First, we describe the standard pipeline of a deep FER system with the related background knowledge and suggestions of applicable implementations for each stage. We then introduce the available datasets that are widely used in the literature and provide accepted data selection and evaluation principles for these datasets. For the state of the art in deep FER, we review existing novel deep neural networks and related training strategies that are designed for FER based on both static images and dynamic image sequences, and discuss their advantages and limitations. Competitive performances on widely used benchmarks are also summarized in this section. We then extend our survey to additional related issues and application scenarios. Finally, we review the remaining challenges and corresponding opportunities in this field as well as future directions for the design of robust deep FER systems.

1 INTRODUCTION

FER has shifted from laboratory-controlled settings toward challenging in-the-wild conditions, while deep learning addresses representation learning and persistent data and nuisance-variation problems. This survey reviews deep FER datasets, pipelines, architectures, training strategies, and future challenges.

  • FER modalities: FER systems use static image representations, dynamic sequence representations, or additional modalities such as audio and physiological signals.Static methods encode spatial information, whereas dynamic methods model temporal relations among contiguous frames.
  • Research transition: Since 2013, FER competitions and increased computing capacity helped shift research from handcrafted or shallow methods toward deep learning in real-world settings.FER2013 and EmotiW collected training data from challenging scenarios, supporting this transition.
  • Survey scope: The survey systematically reviews deep FER for static images and videos, including datasets, technical details, recent advances, and a standard framework for newcomers.It also examines methods addressing the field’s data and variation problems.
  • Core challenges: Deep FER is constrained by insufficient training data and inter-subject variation caused by attributes such as age, gender, ethnicity, and expressiveness.These factors can produce overfitting and identity bias in deep networks.

2 FACIAL EXPRESSION DATABASES

Deep FER research uses both laboratory-controlled and Internet-collected datasets, with differences in scale, labels, collection conditions, viewpoints, and temporal structure. The survey presents dataset characteristics and common selection or evaluation practices.

  • Dataset landscape: Public FER databases range from controlled collections with posed expressions to large-scale Internet datasets containing diverse real-world imagery.The survey distinguishes dataset size, collection environment, expression distribution, and related information.
  • Laboratory-controlled datasets: CK+ contains 593 sequences from 123 subjects, with 327 sequences labeled for seven basic expressions and common person-independent n-fold evaluation.Static methods commonly use the first neutral frame and one to three peak-expression frames.
  • Laboratory-controlled datasets: MMI provides 326 laboratory-controlled sequences from 32 subjects, including onset-apex-offset expression phases and more challenging recording conditions than CK+.The sequences begin neutrally, reach a peak near the middle, and return to neutral.
  • In-the-wild datasets: In-the-wild resources include AFEW’s spontaneous movie clips with pose, occlusion, illumination, audio, and video variation, and SFEW 2.0’s 958 training, 436 validation, and 372 test images.Both datasets use seven expression categories.

3 DEEP FACIAL EXPRESSION RECOGNITION

A deep FER system commonly comprises preprocessing, deep feature learning, and deep feature classification. The survey discusses alignment, augmentation, normalization, CNN-based architectures, and sequence-oriented spatio-temporal modeling.

  • Pipeline: The deep FER pipeline consists of preprocessing, deep feature learning, and deep feature classification.The survey summarizes algorithms and recommended implementations for each stage.
  • 3.1.1 Face alignment: Face detection removes background and non-face regions, while landmark-based alignment reduces face-scale and in-plane-rotation variation and can substantially improve FER performance.Multiple detectors may be combined for landmark estimation in challenging unconstrained environments.
  • 3.1.2 Data augmentation: Data augmentation is vital because most public FER databases lack sufficient training images, using on-the-fly or offline transformations to expand data size and diversity.Operations include cropping, flipping, rotation, shifting, scaling, noise, contrast, and color jittering.
  • 3.1.3 Face normalization: Illumination and pose normalization address nuisance variation that can impair FER, including contrast changes and non-frontal views in unconstrained settings.Pose normalization can synthesize frontal views after facial landmarks are localized.
  • 3.2 Deep feature learning: CNNs learn FER representations through convolutional, pooling, and fully connected layers, while derived 3D CNNs capture spatio-temporal features from adjacent frames.The survey compares CNN configurations and their reported achievements.

4 THE STATE OF THE ART

The survey’s state-of-the-art review groups deep FER systems by static images and dynamic image sequences, but the supplied passage provides only this organizational description.

  • Review organization: The literature review separates deep FER networks and training strategies for static images from those for dynamic image sequences.It also surveys network architecture and performance across these two data types.

4.1 Deep FER networks for static images

Static-image deep FER reviews strategies for overcoming limited training data and expression-unrelated variation, including pre-training, multistage fine-tuning, and feature-level guidance.

  • Pre-training and fine-tuning: Additional task-oriented data and pre-trained models are used to reduce overfitting when facial-expression datasets are too small for deep networks.Face-recognition and larger FER datasets are described as suitable auxiliary sources.
  • Pre-training and fine-tuning: Multistage fine-tuning first adapts pre-trained models with FER2013 and then refines them on the target EmotiW training data.The second stage specializes the models to the target dataset.
  • Performance summary: Table 4 summarizes representative static-image FER methods by network size, preprocessing, illumination normalization, ensembles, cascades, multitask learning, and evaluation protocol.LOSO denotes leave-one-subject-out evaluation.
  • Pre-training and fine-tuning: Figure 3 reports that the proposed two-stage fine-tuning strategy, Submission 3, achieved the best performance among the shown combinations.FER28 and FER32 denote different FER2013 partitions, while EmotiW is the target dataset.
  • Pre-training and fine-tuning: The two-stage FaceNet2ExpNet algorithm uses a fine-tuned face net to guide convolutional features before jointly training randomly initialized fully connected layers with expression labels.The face net is used as initialization and feature-level guidance, while the fully connected layers are trained from scratch.

4.1.2 Diverse network input

Diverse network inputs augment raw aligned-face RGB data with handcrafted, low-level, and part-based representations to emphasize expression-relevant information and reduce confounding variation.

  • Input representations: Raw aligned-face RGB inputs can lack homogeneous textures and invariance to scaling, rotation, occlusion, and illumination.These missing properties may act as confounding factors for FER.
  • Input representations: Low-level representations use local histograms and descriptors such as mapped LBP and SIFT to improve robustness to illumination, registration, scaling, and rotation variation.Mapped LBP was proposed for illumination-invariant FER, while SIFT was used for multiview FER.
  • Input representations: Part-based representations focus on task-sensitive facial regions, including eyebrows, eyes, and mouth, while some methods learn key parts automatically.These approaches remove noncritical regions from the whole image.

4.1.3 Auxiliary blocks & layers

Auxiliary blocks, layers, and loss functions extend CNNs to strengthen expression-related representations, supervise features at multiple depths, and address FER-specific class geometry.

  • Auxiliary architectures: HoloNet combines CReLU, residual structures, and an inception-residual block to increase depth and capture multiscale expression variation.The design targets deeper and more efficient FER representations.
  • Auxiliary architectures: Supervised Scoring Ensemble embeds shallow-, intermediate-, and deep-layer supervised blocks in a mainstream CNN.The three blocks increase supervision at different stages of the network.
  • Auxiliary architectures: Feature selection networks filter irrelevant features and emphasize correlated ones using learned facial-expression feature maps.The feature-selection mechanism is embedded inside AlexNet.
  • Loss functions: Novel loss layers complement softmax by compacting same-class features and emphasizing difficult examples under high inter-class similarity and intra-class variation.The reviewed approaches include center-based, local-cluster, and triplet-based loss variations.
  • Loss functions: Table 5 summarizes majority voting, simple averaging, and weighted averaging as primary decision-level ensemble rules.Weighted averaging accounts for the importance and confidence of individual networks.

4.1.4 Network ensemble

Network ensembles combine diverse FER models at the feature or decision level, using complementary representations and aggregation rules to form committee predictions.

  • Ensemble design: Effective network ensembles require both sufficiently diverse committee members and an aggregation method that combines them appropriately.Diversity can come from different training data, preprocessing, architectures, parameters, or initialization seeds.
  • Feature-level ensembles: Feature-level ensembles concatenate normalized features learned by different networks into a single feature vector describing the input image.The cited example combines VGG13 fc5, VGG16 fc7, and ResNet pool features.
  • Decision-level ensembles: Decision-level ensembles aggregate member predictions using majority voting, simple averaging, or weighted averaging.Weighted methods seek model weights that reflect each member’s importance and confidence.

4.1.5 Multitask networks

Multitask FER networks combine expression recognition with related tasks to learn expression-discriminative features while reducing identity and other nuisance variations.

  • Multitask networks: Multitask learning transfers knowledge from related tasks to disentangle expression information from head pose, illumination, and subject identity.Relevant auxiliary tasks include facial landmark localization, facial action unit detection, and face verification.
  • Multitask networks: IACNN uses two identical sub-CNNs with expression-sensitive and identity-sensitive contrastive losses for identity-invariant FER.
  • Multitask networks: MSCNN jointly trains expression recognition with cross-entropy loss and face verification with contrastive loss using paired images.The expression task increases between-expression variation, while face verification reduces within-expression variation.
  • Multitask networks: An all-in-one CNN branches task-specific subnetworks from different layers and uses domain-based regularization across multiple face-analysis datasets.For smile detection, lower convolutional layers are fused because the task relies more on local information.

4.1.6 Cascaded networks

Cascaded FER networks combine sequential modules so later stages use earlier outputs to learn hierarchical features and progressively filter expression-unrelated variation.

  • Cascaded networks: Cascaded networks combine modules for different tasks sequentially, with each later module consuming the preceding module’s output.This structure supports hierarchical feature learning.
  • Cascaded networks: DBNs can first detect faces and expression-related areas before a stacked autoencoder classifies the parsed face components.
  • Cascaded networks: GAN-based methods support pose-invariant FER by frontalizing faces or synthesizing expressions under arbitrary poses.These methods aim to preserve expression and identity characteristics while generating pose-diverse inputs.
  • Cascaded networks: Identity-invariant GAN-based FER generates neutral faces or different expressions while preserving identity information.

4.1.8 Discussion

The discussion compares deep FER strategies against limited data, nuisance variation, efficiency, accuracy, and training difficulty, highlighting distinct trade-offs and unresolved deployment boundaries.

  • Discussion: Deep FER methods primarily address insufficient diverse training data and expression-unrelated variations such as illumination, head pose, and identity.
  • Discussion: Pre-training and fine-tuning with auxiliary object or face-recognition data are mainstream strategies for insufficient data and overfitting.The recommended progression is from large to small datasets and from general to specific tasks.
  • Discussion: Pre-designed features can improve robustness to illumination, head pose, and occlusion, but identity bias is commonly ignored and heterogeneous inputs increase time and dimensionality.
  • Discussion: Network ensembles combine complementary models but increase computation and storage, while weights learned on training data can overfit unseen tests.
  • Discussion: GANs can address pose and identity variation while augmenting data size and diversity, but training instability and visual-quality–diversity trade-offs remain.

4.2 Deep FER networks for dynamic image sequences

Dynamic deep FER methods aggregate frame outputs or features, model expression intensity, or learn spatio-temporal dependencies; sequence-based modeling improves benchmark performance but remains computationally and temporally challenging.

  • Deep FER networks for dynamic image sequences: Dynamic FER methods use frame aggregation, intensity-invariant networks, or spatio-temporal networks to exploit information across expression sequences.
  • Deep FER networks for dynamic image sequences: Decision-level aggregation integrates per-frame probability vectors, while feature-level aggregation summarizes frame features with statistics or matrix-based encodings.
  • Deep FER networks for dynamic image sequences: Frame averaging uses 10 uniformly sampled frame groups for longer sequences, whereas frame expansion repeats frames to reach 10 frames for shorter sequences.
  • Deep FER networks for dynamic image sequences: Intensity-invariant networks train on samples with different expression intensities to exploit correlations across a sequence.PPDN uses peak and non-peak images of the same expression and subject, with an L2-norm loss during training.
  • Deep FER networks for dynamic image sequences: RNN, LSTM, IRNN, BRNN, and C3D provide foundational approaches for learning spatio-temporal features from facial-expression sequences.
  • Deep FER networks for dynamic image sequences: Frame aggregation ignores temporal information, intensity-invariant networks require unavailable intensity labels, and spatio-temporal models face limited data and computational costs.Current CNN and temporal-encoding networks are also often trained separately rather than end-to-end.
  • Deep FER networks for dynamic image sequences: Deep spatio-temporal networks improve performance on benchmarks such as CK+ and MMI by analyzing temporal dependencies between frames.Their use increased from 1 of 7 reviewed EmotiW 2015 systems to 5 of 7 reviewed EmotiW 2017 systems.

5 ADDITIONAL RELATED ISSUES

The survey extends deep FER beyond basic expression classification to challenges involving occlusion, pose, illumination, 3D data, synthesis, visualization, and related applications.

  • Occlusion and non-frontal head pose alter facial appearance and are major obstacles for FER in real-world scenarios.
  • Deep models address multi-view FER through landmark-weighted projection, pose-aware CNNs, and GAN-generated expressions under arbitrary poses.
  • Infrared and depth modalities offer alternatives to RGB or grayscale data because they are less sensitive to illumination and encode facial geometry.
  • Combining RGB and depth information is explored through jointly learned representations and multimodal fusion CNNs for 2D+3D FER.
  • Deep generative models support realistic facial-expression synthesis and data augmentation, while visualization methods analyze discriminative facial regions in learned CNN features.
  • Additional applications and challenges include dominant versus complementary emotions, real versus fake expressions, smartphone FER, and eye-tracking-based recognition.

6 CHALLENGES AND OPPORTUNITIES

The survey identifies data scarcity and dataset limitations, incomplete emotion models, dataset bias, class imbalance, and the need for multimodal information as continuing challenges for deep FER.

  • Deep FER requires substantial training data, while datasets lack comprehensive demographic, occlusion, and head-pose annotations.
  • Accurately annotating large image collections with varied natural expressions remains an impediment to constructing expression datasets.
  • Categorical prototypical expressions cover only a small portion of expressive behaviors and cannot capture the full repertoire of realistic interactions.
  • CNN representations show congruity with facial areas defined by action units, motivating filters that weight facial muscle-action regions by importance.
  • Different collection conditions and annotation practices create dataset bias and inconsistent labels, making cross-database evaluation important.
  • Class imbalance makes less common expressions harder to capture and lowers mean accuracy, especially on real-world datasets; augmentation or cost-sensitive losses are alternatives.
  • Visible-face FER can benefit from complementary information supplied by other modalities in high-level multimodal frameworks.
Loading 1804.08348v2…