PaperlayerView exact revision on arXiv

Source-linked AI summary

Facial Emotion Recognition: State of the Art Performance on FER2013

Yousif Khaireddin, Zhuofa Chen

arXiv:2105.03588v1cs.CVcs.AIcs.LG

TL;DR

Facial emotion recognition remains challenging under naturalistic conditions such as varied pose and subtle expression differences. This paper fine-tunes a VGG-based CNN on FER2013 and reports 73.28% testing accuracy, the highest single-network accuracy reported without extra training data.

  • Problem

    Facial emotion recognition remains challenging in naturalistic conditions because facial pose varies and expressions may differ subtly.

  • Method

    The paper fine-tunes a VGG-based CNN on FER2013 while exploring optimization algorithms and learning-rate schedulers.

  • Results

    73.28% testing accuracy is reported on FER2013, claimed as the highest single-network accuracy without extra training data.

  • Takeaways & Limitations

    The study reports state-of-the-art single-network performance on FER2013 without using extra training data.

Abstract

from arXiv · show

Facial emotion recognition (FER) is significant for human-computer interaction such as clinical practice and behavioral description. Accurate and robust FER by computer models remains challenging due to the heterogeneity of human faces and variations in images such as different facial pose and lighting. Among all techniques for FER, deep learning models, especially Convolutional Neural Networks (CNNs) have shown great potential due to their powerful automatic feature extraction and computational efficiency. In this work, we achieve the highest single-network classification accuracy on the FER2013 dataset. We adopt the VGGNet architecture, rigorously fine-tune its hyperparameters, and experiment with various optimization methods. To our best knowledge, our model achieves state-of-the-art single-network accuracy of 73.28 % on FER2013 without using extra training data.

Abstract · Introduction

FER2013 is a challenging benchmark for facial emotion recognition, motivating CNN-based optimization beyond existing single-network results. The study uses a tuned VGG network and reports 73.28% testing accuracy without extra training data.

  • Introduction: Facial emotion recognition supports human-computer interaction and applications including digital advertising, online gaming, customer feedback assessment, and healthcare.
  • Introduction: FER2013 captures difficult naturalistic conditions, became an emotion-recognition benchmark, and has an estimated human performance of 65.5%.
  • Introduction: 73.28% testing accuracy was achieved by adopting VGG, exploring optimization algorithms and learning-rate schedulers, and tuning model and training hyperparameters.
  • Introduction: The reported result is presented as the highest single-network FER2013 accuracy without extra training data.
  • Introduction: Learning-rate selection affects training: large rates can cause oscillation or divergence, while small rates slow convergence or trap models at nonoptimal local minima.
  • Introduction: Existing CNN variants achieved 65%–72.7% classification accuracy, while ensembles improved performance by 2.6% or 2.5%.The work therefore focuses on optimizing a single network as an ensemble building block.

Experiments

The experiments use FER2013’s official dataset splits and apply substantial randomized augmentation to address facial-expression variability. FER2013 contains 35,888 images spanning seven emotions, while reported human accuracy is 65–68%.

  • Dataset, Preprocessing, and Augmentation: FER2013 contains 35,888 images across seven emotions and follows the official ICML training, validation, and test splits.The emotions are anger, neutral, disgust, fear, happiness, sadness, and surprise.
  • Dataset, Preprocessing, and Augmentation: 65–68% is the reported human-accuracy range on FER2013 from a competition-organizer Kaggle forum discussion.
  • Dataset, Preprocessing, and Augmentation: Training augmentation randomly rescales, shifts, and rotates images by up to ±20%, ±20%, and ±10 degrees, respectively, with each technique applied at 50% probability.Images are then tencropped to 40×40.

B. Training and Inference

All experiments train for 300 epochs with cross-entropy loss while varying the optimizer and learning-rate scheduler and holding other parameters constant. Models use fixed momentum and weight decay, gradient scaling, validation-accuracy evaluation, and standard ten-crop testing.

  • Training: 300 epochs: All experiments optimize cross-entropy loss for 300 epochs.The training duration and objective are fixed across experiments.
  • Training: The experiments vary the optimizer and learning-rate scheduler while maintaining other parameters constant.This isolates the effects of the optimization choices.
  • Training: Momentum is fixed at 0.9 and weight decay at 0.0001.These settings remain constant during the experiments.
  • Training: Gradient scaling is used in all experiments to prevent gradient underflow.This stabilization procedure is applied throughout training.
  • Inference: Models are evaluated with validation accuracy and tested using standard ten-crop averaging.Validation accuracy is the evaluation measure, while ten-crop averaging is used for testing.

C. VGGNet Architecture

The proposed VGGNet variant uses four convolutional stages followed by three fully connected layers to extract facial features and classify emotions. Each convolutional stage contains two convolutional blocks and a max-pooling layer.

  • Architecture: The VGGNet variant comprises four convolutional stages and three fully connected layers for facial emotion classification.The convolutional stages extract high-level image features, while the fully connected layers classify the emotion.
  • Architecture: Each convolutional stage contains two convolutional blocks and a max-pooling layer.Each block consists of a convolutional layer, ReLU activation, and batch normalization layer.

D. Tuning · D.1 Optimizer · D.2 LR Schedule

The study first optimizes architecture, then compares optimizers and learning-rate schedulers, and finally fine-tunes trained weights. Scheduler and weight-tuning experiments also test whether added training data improves performance.

  • D. Tuning: Architecture optimization precedes comparisons of optimizers, learning-rate schedulers, and final weight fine-tuning.Initial experiments use SGD, with grid searches for batch size and post-fully-connected-layer dropout.
  • D.1 Optimizer: The optimizer experiment compares SGD, SGD with Nesterov Momentum, Average SGD, Adam, Adam with AMSGrad, Adadelta, and Adagrad.The paper describes these as 6 different algorithms despite listing seven names.
  • D.1 Optimizer: Two optimizer variations use either a fixed learning rate of 0.001 or a scheduler starting at 0.01 and reducing it by 0.75 after 5 validation-accuracy plateau epochs.Both learning-rate settings are selected through grid search.
  • D.2 LR Schedule: The scheduler experiment evaluates RLRP, Cosine, CosineWR, OneCycleLR, and StepLR against a constant learning-rate baseline.It uses the optimal optimizer selected in the preceding experiment.
  • D.2 LR Schedule: StepLR updates after a fixed epoch count, whereas RLRP responds to current model performance; Cosine and CosineWR use cosine-based learning-rate updates.The schedulers differ in how they update learning rates despite sharing similarities.
  • D.2 LR Schedule: Final weight tuning reloads the trained parameters and runs 50 epochs from an initial learning rate of 0.0001 using Cosine and CosineWR.The small learning rate is intended to prevent large updates from substantially shifting an already trained model.
  • D.2 LR Schedule: A second tuning variation merges the validation set into training while leaving the test set unchanged and keeping other parameters constant.The larger training set is intended to provide more samples for learning and improve performance.

A. Optimizer

The study compares optimizers under constant and decaying learning-rate settings. Excluding Adadelta, all optimizers achieve validation accuracy above 70%, with SGD using Nesterov momentum performing best in both experiment variations.

  • A. Optimizer: Above 70% validation accuracy is achieved by all optimizers except Adadelta.The comparison uses constant and decaying learning-rate variations.
  • A. Optimizer: SGD with Nesterov momentum performs best in both experiment variations.

B. LR Schedule

The study compares learning-rate schedulers using SGD with Nesterov momentum and finds that Reducing Learning Rate on Plateau performs best. Its validation accuracy is 73.59% and testing accuracy is 73.06%, surpassing prior single-network state-of-the-art performance.

  • B. LR Schedule: 73.59% validation accuracy and 73.06% testing accuracy are achieved by Reducing Learning Rate on Plateau, the best-performing scheduler.All runs use SGD with Nesterov momentum.
  • B. LR Schedule: The reported testing accuracy is used strictly for public benchmarking, so subsequent scheduler comparisons focus on validation accuracy.This comparison convention separates model-selection analysis from benchmarking results.
  • B. LR Schedule: The constant learning rate outperforms the OneCycleRL and StepLR schedulers on validation accuracy.OneCycleLR may be less applicable to FER2013 because it is typically intended for fast training with larger learning rates.

C. Fine Tuning · D. Confusion Matrix

The study further tuned its best-performing VGGNet with cosine annealing and evaluated the final model using a FER2013 confusion matrix. Fine-tuning improved testing performance, while the confusion matrix showed strongest recognition for happiness and surprise and frequent confusion between disgust and anger.

  • C. Fine Tuning: 50 extra epochs of tuning were applied to the best-performing model using cosine annealing schedulers at a small learning rate.The experiment reloaded the best model achieved thus far before tuning.
  • C. Fine Tuning: Training and validation data were combined for a separate run using the same extra-tuning procedure.This run was compared with the model being tuned.
  • C. Fine Tuning: Cosine annealing performed best in the extra-tuning experiment and improved the model’s testing performance.Table 1 reports the final testing accuracies benchmarked against the tuned model.
  • D. Confusion Matrix: The final VGGNet’s confusion matrix was evaluated on the FER2013 public test set.Figure 4 presents the model’s classification pattern across emotions.
  • D. Confusion Matrix: The model classified happiness and surprise most accurately among the evaluated emotions.These emotions showed the best classification in the final model’s confusion matrix.
  • D. Confusion Matrix: The most frequent mistakes occurred between disgust and anger, while disgust and fear also had low classification accuracy.The low accuracy for disgust and fear was attributed to their lower sample counts in the original training set.

E. Performance Comparison

On FER2013, most reported methods exceed estimated human performance of approximately 65.5%, while this work achieves the highest reported single-network accuracy at 73.28%.

  • E. Performance Comparison: 72.7% accuracy was the previous best-reported single-network result on FER2013.This work’s result improves on that prior benchmark.
  • E. Performance Comparison: 65.5% is the estimated human performance on FER2013, which most reported methods surpass.The passage reports the estimate as approximately 65.5%.

F. Saliency map · Conclusion

The paper uses saliency maps to show how its CNN identifies facial features for emotion classification, while concluding that a thoroughly tuned VGGNet achieves state-of-the-art single-network FER2013 accuracy. The visualizations also reveal background pixels being mistakenly emphasized, suggesting that better facial-feature localization could improve performance.

  • F. Saliency map: The black-box nature of deep learning motivates visualizing internal information to evaluate the CNN and understand how it differentiates facial emotions.An understandable visualization is presented as a way to describe the model’s processing and prediction.
  • F. Saliency map: Saliency maps visualize the pixels with the greatest impact on the loss and clarify which captured features influence the final classification.They are generated by propagating the loss back to pixel values.
  • F. Saliency map: The authors generate saliency maps with their best-performing network for each FER2013 emotion and superimpose them on the original images.Figure 5 presents the maps before and after superimposition.
  • F. Saliency map: The CNN effectively captures most critical regions and places substantial importance on nearly all facial features across the emotion images.This pattern is reported from judging the saliency visualizations.
  • F. Saliency map: The saliency maps sometimes highlight background pixels, including in images (b), (e), and the corner of (g).The authors suggest that removing useless information through better facial-feature identification could improve dataset performance.
  • Conclusion: 73.06 % is the best initial testing classification accuracy achieved by the VGGNet after thorough hyperparameter tuning, surpassing previously reported single-network accuracies.The conclusion also states that different optimizers and learning-rate schedulers were explored.
Loading 2105.03588v1…