Source-linked AI summary
Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?
Pooya Khorrami, Tom Le Paine, Thomas S. Huang
TL;DR
The paper asks how much CNNs improve established facial-expression benchmarks and what their learned features represent. It trains and analyzes a zero-bias CNN using filter visualizations and FAU labels, reporting strong benchmark performance and learned patterns aligned with facial movements.
Problem
Limited work had examined CNN gains on established facial-expression benchmarks or what CNNs learn for expression recognition.
Method
The authors train a zero-bias CNN and analyze maximally activating convolutional filters with spatial visualizations, then compare them with CK+ FAU labels.
Results
The CNN achieves state-of-the-art performance on CK+ and TFD, while visualized filters correspond strongly with FAUs and subject facial movements.
Takeaways & Limitations
CNNs trained for emotion recognition can learn high-level features that strongly correspond to FAUs.
Abstract
from arXiv · showhide
Despite being the appearance-based classifier of choice in recent years, relatively few works have examined how much convolutional neural networks (CNNs) can improve performance on accepted expression recognition benchmarks and, more importantly, examine what it is they actually learn. In this work, not only do we show that CNNs can achieve strong performance, but we also introduce an approach to decipher which portions of the face influence the CNN's predictions. First, we train a zero-bias CNN on facial expression data and achieve, to our knowledge, state-of-the-art performance on two expression recognition benchmarks: the extended Cohn-Kanade (CK+) dataset and the Toronto Face Dataset (TFD). We then qualitatively analyze the network by visualizing the spatial patterns that maximally excite different neurons in the convolutional layers and show how they resemble Facial Action Units (FAUs). Finally, we use the FAU labels provided in the CK+ dataset to verify that the FAUs observed in our filter visualizations indeed align with the subject's facial movements.
1. Introduction
Facial expression recognition needs to identify localized facial changes, but it remains unclear how much CNNs improve established benchmarks and what they learn. The paper addresses both questions by visualizing CNN filters and verifying their correspondence with Facial Action Units (FAUs).
- Expression-relevant facial changes are concentrated around the eyes, nose, and mouth, making accurate localization non-trivial.
- CNNs have produced large gains in several vision tasks, but their benefits for facial expression recognition remain comparatively underexplored.
- The paper trains CNNs on established expression datasets and visualizes individual filters through reconstructed spatial patterns.
- Many maximally activating filters correspond to facial regions associated with FAUs, linking learned CNN patterns to facial movements.
- The authors report state-of-the-art performance on both the extended Cohn-Kanade dataset and the Toronto Face Dataset.
2. Related Work
Earlier expression-recognition systems used hand-crafted or unsupervised learned features, while prior CNN work largely addressed different settings. This paper focuses on single-image recognition and analyzes whether learned patterns correspond to FAUs.
- Traditional systems commonly used Gabor wavelets, Haar features, or LBP features to make expression representations more discriminative.
- Earlier learned-representation methods often relied on unsupervised pre-training or models such as boosted deep belief networks and contractive auto-encoders.
- Some prior work connected learned features to FAU receptive fields using K-means filter banks, mutual information, and restricted Boltzmann machines.
- Previous CNN studies addressed audio-visual emotion recognition or combined appearance and geometric features rather than the paper’s single-image setting.
- The paper combines benchmark evaluation with qualitative analysis showing that learned facial patterns correspond to FAUs.
3. Our Approach
The approach uses a feed-forward CNN with three convolutional layers, pooling, a fully connected layer, and softmax classification. It removes convolutional biases and uses training regularization and augmentation to improve learning.
- 3.1. Network Architecture: The CNN contains three convolutional layers with 64, 128, and 256 filters, followed by pooling, 300 fully connected units, and a softmax classifier.
- 3.1. Network Architecture: The convolutional layers omit biases, which the authors found enabled faster training while reducing the number of learned parameters.
- 3.2. Network Training: Training starts from random initialization with stochastic gradient descent, batch size 64, momentum 0.9, weight decay 1e-5, and a constant learning rate of 0.01.
- 3.2. Network Training: Dropout and transformations including translations, flips, rotations, scaling, and pixel-intensity changes regularize the network and combat overfitting.
4. Experiments and Analysis
Experiments evaluate CNN expression recognition on TFD and CK+, then inspect higher-level filters and test whether their activations correspond to labeled Facial Action Units.
- Performance on TFD: TFD experiments compare a zero-bias CNN with dropout, data augmentation, or both, using grayscale 96x96 images and five-fold evaluation.The TFD contains 4,178 images across seven expression labels; models are selected on validation sets and evaluated on held-out test sets.
- Performance on TFD: 3.6%: combining dropout and data augmentation exceeds previous state-of-the-art performance on TFD.Data augmentation improves over the regular CNN by 9.4%, compared with 2.8% for dropout alone.
- Performance on CK+: 16.9%: combining dropout and data augmentation increases eight-class CK+ performance, while data augmentation alone gives a 16.4% boost.The eight-class model reaches state-of-the-art accuracy, while the six-class model reaches near state-of-the-art accuracy.
- Visualization of higher-level neurons: The visualization procedure selects the 10 strongest third-convolutional-layer filter responses and reconstructs their influential pixel-space regions with a deconvolutional network.Guided backpropagation further masks reconstructions using both deconvolutional and forward-pass ReLU information.
- Visualization of higher-level neurons: TFD filter visualizations align with AU12 Lip Corner Puller, AU4 Brow Lowerer, and AU15 Lip Corner Depressor, while CK+ visualizations align with AU12, AU9, and AU27.The reported CK+ correspondences are AU12 in rows 2, 6, and 9, AU9 in row 3, and AU27 in row 8.
- Finding FAU correspondences: KL-divergence comparisons show that, in most cases, the FAUs associated with the visualized regions also have the strongest activation-distribution differences.For each filter and FAU, the experiment compares activation distributions for samples containing and lacking that FAU.
- Finding FAU correspondences: Certain neurons therefore implicitly learn specific FAUs from the relatively loose supervisory signal of emotion labels.The paper connects filter sensitivity to visually salient cues, including smiling teeth for AU12 and an “O” mouth shape for FAUs 25 and 27.
5. Conclusions
The study combines qualitative filter visualization with quantitative FAU-label analysis to show that emotion-recognition CNNs learn features corresponding strongly to facial movements. A zero-bias CNN also achieves state-of-the-art recognition accuracy on CK+ and TFD.
- CNN filters reveal spatial patterns corresponding strongly to Facial Action Units (FAUs).The visualizations identify facial portions that maximally excite convolutional filters.
- FAU labels quantitatively connect filter activations with subjects’ actual facial movements.The analysis correlates numerical activations of visualized filters with ground-truth FAU labels from CK+.
- A zero-bias CNN achieves state-of-the-art recognition accuracy on the extended Cohn-Kanade and Toronto Face datasets.
- 10 selected conv3 filters show patterns associated with FAU-related facial regions.The figure displays the top 10 highest-response images for each selected filter.