PaperlayerView exact revision on arXiv

Source-linked AI summary

Facial Emotion Recognition: State of the Art Performance on FER2013

Yousif Khaireddin, Zhuofa Chen

arXiv:2105.03588v1cs.CVcs.AIcs.LG

TL;DR

Facial emotion recognition remains challenging because faces and image conditions vary. This paper fine-tunes a VGGNet for FER2013 and reports 73.28% single-network accuracy without extra training data.

  • Problem

    Although ensembles can improve FER2013 performance, optimizing a single network remains an important goal for improving their building blocks.

    Source: Related work · p. 2

  • Method

    The paper uses a VGGNet and tunes its hyperparameters by exploring optimizers and learning-rate schedules.

    Source: Conclusion · p. 7

  • Results

    73.28% single-network classification accuracy on FER2013 is the paper’s reported final result.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The authors position single-network optimization as a step toward improving ensemble building blocks for FER2013.

    Source: Related work · p. 2

    The study excludes auxiliary training data.

    Source: Related work · p. 2

Abstract

from arXiv · show

Facial emotion recognition (FER) is significant for human-computer interaction such as clinical practice and behavioral description. Accurate and robust FER by computer models remains challenging due to the heterogeneity of human faces and variations in images such as different facial pose and lighting. Among all techniques for FER, deep learning models, especially Convolutional Neural Networks (CNNs) have shown great potential due to their powerful automatic feature extraction and computational efficiency. In this work, we achieve the highest single-network classification accuracy on the FER2013 dataset. We adopt the VGGNet architecture, rigorously fine-tune its hyperparameters, and experiment with various optimization methods. To our best knowledge, our model achieves state-of-the-art single-network accuracy of 73.28 % on FER2013 without using extra training data.

Related work

Related work traces CNNs’ evolution into practical image-classification tools and reviews training techniques, optimizers, and learning-rate schedules. On FER2013, prior CNN results establish a benchmark that motivates optimizing single networks rather than relying on ensembles or auxiliary data.

Related work

  • CNNs became more viable for feature extraction and image classification as computing power and dataset sizes grew after 2010.Typical CNNs use convolutional, pooling, and fully connected layers, supporting efficient static-image manipulation.Source: Related work · p. 2
  • ReLU, pooling choices, dropout, regularization, data augmentation, and batch normalization are among techniques used to improve training, generalization, or overfitting control –.Source: Related work · p. 2
  • SGD is common, while AdaGrad adapts per-dimension learning rates, RMSProp diminishes the learning rate, and Adam combines adaptive scaling with gradient momentum –.The paper notes that no systematic theoretical guideline dictates optimizer choice, although empirical results show that a suitable optimizer can improve performance.Source: Related work · p. 2
  • Learning rates that are too large can cause oscillation or divergence, while rates that are too small slow convergence or trap models in nonoptimal minima; schedules address this tradeoff –.Examples include time-based decay, step decay, adaptive schedules, and cosine annealing, which periodically resets the rate and reuses good weights.Source: Related work · p. 2
  • FER2013 became a benchmark, with CNN variants reporting 65%–72.7% accuracy; VGG led VGG, ResNet, and Inception at 72.7%, 72.4%, and 71.6%, respectively –.Reported examples include Minaee et al.’s 70.02% accuracy, Tang et al.’s 71.2%, and Shi et al.’s 71.38%.Source: Related work · p. 2
  • Ensembling three CNNs improved accuracy by 2.6%, and ensembling eight improved performance by 2.5%; this work instead optimizes single networks, with auxiliary training data out of scope.Source: Related work · p. 2

Experiments · A. Dataset, Preprocessing, and Augmentation

The experiments use FER2013’s official train, validation, and test sets and apply randomized geometric and crop-based augmentation during training.

A. Dataset, Preprocessing, and Augmentation

  • FER2013 contains 35,888 images across seven emotions, and experiments follow its official training, validation, and test sets.The emotions are anger, neutral, disgust, fear, happiness, sadness, and surprise.Source: A. Dataset, Preprocessing, and… · p. 3
  • Human accuracy on FER2013 was estimated at 65–68% in a Kaggle forum discussion by the competition organizers.Source: A. Dataset, Preprocessing, and… · p. 3
  • Training augmentation randomly rescales, shifts, and rotates images, then applies ten crops with random erasures and pixel normalization.Each geometric transformation is applied with 50% probability; images are rescaled by up to ±20%, shifted by up to ±20% of image size, and rotated by up to ±10 degrees. Crops are 40×40, erasure probability is 50%, and normalization divides each pixel by 255.Source: A. Dataset, Preprocessing, and… · p. 3

B. Training and Inference

Experiments run for 300 epochs with cross-entropy loss, varying the optimizer and learning-rate scheduler while holding other parameters constant. Models use fixed momentum and weight decay, gradient scaling, validation-accuracy evaluation, and standard ten-crop testing.

B. Training and Inference

  • All experiments run for 300 epochs using cross-entropy loss, with optimizer and learning-rate scheduler varied while other parameters remain constant.Source: B. Training and Inference · p. 3
  • Models use momentum 0.9, weight decay 0.0001, gradient scaling, validation accuracy, and standard ten-crop averaging for testing.Source: B. Training and Inference · p. 3

C. VGGNet Architecture

The VGGNet variant uses four convolutional stages for feature extraction and dimension reduction, followed by three fully connected layers for emotion classification. Its convolutional blocks combine convolution, ReLU, and batch normalization, while max-pooling reduces dimensions.

C. VGGNet Architecture

  • The variant has four convolutional stages and three fully connected layers, with convolutional blocks extracting image features and the fully connected layers classifying emotion.Figure 1 depicts a face-expression image entering the network, whose convolutional blocks extract high-level features for classification.Source: C. VGGNet Architecture · p. 3, Figure 1 · p. 3
  • Each convolutional stage contains two convolutional blocks and a max-pooling layer; each block combines convolution, ReLU, and batch normalization.Batch normalization is used to speed learning, reduce internal covariance shift, and prevent gradient vanishing or explosion.Source: C. VGGNet Architecture · p. 3
  • The first two fully connected layers are followed by ReLU, while the third performs classification; convolutional stages extract features, reduce dimensions, and add non-linearity.VGGNet is described as a classical architecture for large-scale image processing and pattern recognition.Source: C. VGGNet Architecture · p. 3

D. Tuning · D.1 Optimizer · D.2 LR Schedule

The paper first optimizes architecture, then compares optimizers and learning-rate schedules while holding other parameters constant. It also tests fine-tuning trained weights, including a variation that adds validation data to training.

D. Tuning

  • Architecture tuning began with SGD and a grid search over batch size and dropout before optimizer, scheduler, and final-weight experiments.The final experiment was designed to fine-tune trained weights.Source: D. Tuning · p. 4

D.1 Optimizer

  • Seven optimizer algorithms were explored to compare how their differences affected training: SGD, Nesterov SGD, Average SGD, Adam, AMSGrad Adam, Adadelta, and Adagrad.Source: §D.1 · p. 4
  • Optimizer comparisons used either a grid-searched fixed learning rate of 0.001 or a scheduler starting at 0.01, reduced by 0.75 after five validation-accuracy plateau epochs.Weight decay, momentum, dropout, and batch size remained constant after initial optimization.Source: §D.1 · p. 4

D.2 LR Schedule

  • Scheduler experiments used the previously selected optimizer to compare RLRP, Cosine, CosineWR, OneCycleLR, and StepLR against a grid-searched constant-learning-rate baseline.All schedules started at 0.01, with parameters selected by grid search; other parameters were held constant.Source: §D.2 · p. 4
  • StepLR lowers the learning rate after set epochs, whereas RLRP reacts to performance; CosineWR periodically resets model parameters, unlike cosine-based annealing alone.Both cosine methods oscillate the learning rate between two values.Source: §D.2 · p. 4
  • Weight fine-tuning reloaded the trained parameters for 50 epochs at an initial learning rate of 0.0001 using Cosine or CosineWR to keep updates small.CosineWR was also selected for its regular warm restarts, which return weights to a good location during updates.Source: §D.2 · p. 4
  • A second fine-tuning variation added the validation set to training while leaving the test set unchanged, allowing comparison of tuning effects and added-data benefits.Source: §D.2 · p. 4

Results · A. Optimizer

SGD with Nesterov momentum achieved the best validation accuracy in both learning-rate experiments, and all optimizers except Adadelta exceeded 70%. The results also show that all SGD variants outperformed the other optimizers.

A. Optimizer

  • 73.2% and 73.5% validation accuracy were achieved by SGD with Nesterov momentum in the constant- and decaying-learning-rate experiments, respectively.It performed best in both experiments.Source: A. Optimizer · p. 5
  • All variants of SGD outperformed all other optimizers on this dataset.Source: A. Optimizer · p. 5
  • Above 70% validation accuracy was attained by every optimizer except Adadelta.Source: A. Optimizer · p. 5
  • Adam and AMSGrad performed better than Adadelta and Adagrad.Source: A. Optimizer · p. 5
  • The paper attributes Adam’s performance to combining AdaGrad and RMSProp advantages with gradient momentum.Source: A. Optimizer · p. 5
  • The optimizer comparison used two learning-rate settings: constant and decaying.Figure 2's yellow bars represent the constant-rate variation, and its orange bars represent the decaying-rate variation.Source: A. Optimizer · p. 5

B. LR Schedule

The section compares learning-rate schedulers using VGGNet and finds that Reducing Learning Rate on Plateau achieves the strongest reported performance. It also explains how its performance-based reductions compare with other scheduling strategies.

B. LR Schedule

  • Constant LR outperforms OneCycleRL and StepLR; OneCycleRL may be unsuitable because it targets fast training with larger learning rates.Source: B. LR Schedule · p. 5
  • Cosine Annealing and Cosine Annealing with Warm Restart perform comparably, while RLRP outperforms StepLR by monitoring performance before reducing LR.StepLR reduces LR systematically, whereas RLRP monitors current performance to decide when to drop it.Source: B. LR Schedule · p. 5
  • 73.59% validation accuracy and 73.06% testing accuracy: Reducing Learning Rate on Plateau performs best and surpasses prior single-network state-of-the-art performance.All scheduler runs use SGD with Nesterov momentum, the best-performing optimizer from the previous section.Source: Figure 3 · p. 5, B. LR Schedule · p. 5

C. Fine Tuning · D. Confusion Matrix

Fine-tuning with cosine annealing yields the final 73.28% accuracy, while the confusion matrix shows that classification performance varies across emotions.

C. Fine Tuning

  • 0.05% improvement is achieved by cosine annealing over the model being tuned in the extra tuning experiment.The best-performing model was tuned for 50 extra epochs at a small learning rate.Source: C. Fine Tuning · p. 6
  • 0.42% accuracy reduction results from cosine annealing with warm restarts during the tuning stage.Source: C. Fine Tuning · p. 6
  • Both scheduler models perform better after tuning on combined training and validation data than in their respective separate runs.Source: C. Fine Tuning · p. 6
  • 73.28% accuracy is achieved by the final model after tuning on combined training and validation data, with the testing data left unaltered.The combined-data tuning run follows the same extra tuning procedure; testing data is not used during training.Source: Table 1 · p. 6, C. Fine Tuning · p. 6

D. Confusion Matrix

  • The final VGGNet classifies happiness and surprise best, but makes the most mistakes between disgust and anger.The confusion matrix is for the FER2013 public test set.Source: D. Confusion Matrix · p. 6, Figure 4 · p. 6
  • Low accuracy for disgust and fear may reflect their fewer original training samples, while fear–sadness errors may reflect inter-class similarities.Source: D. Confusion Matrix · p. 6

E. Performance Comparison

The work reports a 73.28% FER2013 accuracy, exceeding the previous best-reported single-network accuracy of 72.7%. Most reported methods also perform better than estimated human performance.

E. Performance Comparison

F. Saliency map · Conclusion

The paper reports improved single-network FER2013 accuracy with a tuned VGGNet and uses saliency maps to show which image regions influence its emotion classifications. The maps highlight facial features and often suppress irrelevant regions, though some background pixels remain highlighted.

F. Saliency map

  • Figure 5 shows saliency maps for each FER2013 emotion; the CNN emphasizes most facial features and often suppresses backgrounds, hair, and hands.The map in (f) almost perfectly maps the woman’s entire face.Source: Figure 5 · p. 7, F. Saliency map · p. 7
  • A saliency map propagates loss back to pixel values to highlight pixels with the greatest impact on loss and reveal features captured for classification.This visualization is intended to help explain the importance of input-image features to the final classification decision.Source: F. Saliency map · p. 7
  • Some maps still highlight background pixels in (b), (e), and a corner of (g), and the authors expect better facial-feature identification and removal of irrelevant information to improve performance.The authors identify these as clear mistakes in the saliency maps.Source: F. Saliency map · p. 7

Conclusion

  • 73.28 % classification accuracy was achieved after extra tuning with Cosine Annealing and combining the training and validation datasets, following 73.06 % best initial testing classification accuracy.The authors report that 73.06 % surpasses previously reported single-network accuracies; their model achieves state-of-the-art single-network accuracy on FER2013.Source: Conclusion · p. 7
  • The authors thoroughly tuned VGGNet hyperparameters, explored different optimizers and learning rate schedulers, and then used Cosine Annealing for extra tuning.The extra-tuned model was trained using combined training and validation datasets.Source: Conclusion · p. 7
  • Future work will explore image-processing techniques on FER2013 and ensembles of different deep learning architectures to improve facial emotion recognition.Source: Conclusion · p. 7

Related papers

Loading 2105.03588v1…