Source-linked AI summary
FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition
Hui Ding, Shaohua Kevin Zhou, Rama Chellappa
TL;DR
Small expression-recognition datasets make deep-network training challenging, and fine-tuning face networks can retain redundant identity-related information. FaceNet2ExpNet models expression features using face-net supervision in a first stage, then jointly refines the network with labels. The method improves visual expression representations and outperforms state-of-the-art methods on four public datasets.
Problem
Small expression-recognition datasets make deep-network training difficult, while fine-tuned face networks may retain subject-identification information and overfit.
Method
FaceNet2ExpNet models high-level expression-network neurons using fine-tuned face-net information, regularizes convolutional features, then adds fully connected layers and jointly trains with labels.
Results
FaceNet2ExpNet improves visual feature representation and outperforms various state-of-the-art methods on CK+, Oulu-CASIA, TFD, and SFEW.
Takeaways & Limitations
The method provides a two-stage approach for exploiting rich face information while learning discriminative expression representations from limited expression data.
Abstract
from arXiv · showhide
Relatively small data sets available for expression recognition research make the training of deep networks for expression recognition very challenging. Although fine-tuning can partially alleviate the issue, the performance is still below acceptable levels as the deep features probably contain redun- dant information from the pre-trained domain. In this paper, we present FaceNet2ExpNet, a novel idea to train an expression recognition network based on static images. We first propose a new distribution function to model the high-level neurons of the expression network. Based on this, a two-stage training algorithm is carefully designed. In the pre-training stage, we train the convolutional layers of the expression net, regularized by the face net; In the refining stage, we append fully- connected layers to the pre-trained convolutional layers and train the whole network jointly. Visualization shows that the model trained with our method captures improved high-level expression semantics. Evaluations on four public expression databases, CK+, Oulu-CASIA, TFD, and SFEW demonstrate that our method achieves better results than state-of-the-art.
I. INTRODUCTION
Expression recognition often has too little labeled data for effective deep-network training, while fine-tuned face networks may retain identity information and overfit. FaceNet2ExpNet regularizes expression learning with face-domain knowledge, then refines the network using labels; experiments report improved expression representations and state-of-the-art results on four datasets.
- Small expression-recognition datasets make deep-network training difficult, and standard regularization may still leave results unsatisfactory.
- Fine-tuned face networks may retain subject-identification information and can be too large for expression recognition, weakening expression representation and worsening overfitting.
- FaceNet2ExpNet models expression-network high-level neurons using fine-tuned face-net information, producing feature-level regularization before label-supervised refinement.
- Visualization indicates that FaceNet2ExpNet captures key expression properties, unlike fine-tuned face-net visualizations dominated by faces.
- Experiments on CK+, Oulu-CASIA, TFD, and SFEW report better results than the current state-of-the-art across constrained and unconstrained datasets.
II. RELATED WORKS
Prior expression-recognition work uses sparse facial patches, deep feature learners, temporal networks, and alternative architectures or staged training procedures. These studies address spatial, temporal, architectural, and training-design aspects of the task.
- A two-stage multi-task sparse-learning framework searches for common and expression-specific facial patches, using face recognition to identify the latter.
- Earlier approaches combine restricted Boltzmann machines or deep belief networks with feature selection, patch-based learning, and jointly trained classifiers.
- Other work uses temporal geometry and appearance networks, inception architectures, or two-step training procedures for expression recognition.
A. Motivation
The method models high-level expression-network responses with a distribution whose mean is supplied by a fine-tuned face network. Maximum-likelihood estimation yields a regularization loss that keeps expression features near face-net features.
- High-level neurons are modeled through spatially averaged channel responses from a convolutional layer.
- The distribution uses X_l as the vector of channel responses, C_p as a normalization constant, and the pth norm.
- The face network supplies the distribution mean through μ = G(I), with G representing its convolutional layers.
- Maximum-likelihood estimation produces a loss that regularizes expression-network features toward the face-network representation; when p = 2 without G, it becomes the standard l2 regularizer.
B. Training Algorithm
Training proceeds in two stages: frozen face-net features supervise convolutional-layer learning, then newly initialized fully connected layers are added and the complete expression network is jointly trained.
- Stage one: Stage one trains only the expression network’s convolutional layers while the face network remains frozen and supplies last-pooling-layer supervision.
- Stage two: Stage two appends randomly initialized fully connected layers and jointly trains the whole network with cross-entropy loss.
- Network architecture: The expression network uses five convolutional layers with ReLU and max-pooling, followed by a 1 × 1 convolution and one 256-dimensional fully connected layer.
- Training rationale: The complete two-stage procedure separates face-net guidance for convolutional learning from label-based optimization of the full network.
C. Which Layer to Transfer?
The study compares neuron expression sensitivity before and after face-network fine-tuning to identify which layers are most suitable for supervising expression-network training.
- Layer selection: Neuron entropy is estimated by ranking high-response images, binning their expression labels, and measuring the resulting label-distribution entropy.Low entropy indicates a peakier expression-label distribution and therefore greater expression sensitivity.
- Layer selection: Fine-tuning produces more low-entropy neurons in deeper layers, especially the fully connected layers, while pool4 changes little.The comparison uses entropy histograms for pool4, pool5, FC6, and FC7 on CK+ and Oulu-CASIA.
- Layer selection: Low expressive score (LES) neurons are defined as neurons whose entropy falls below the minimum average entropy across the four selected layers.The LES count is then used to compare layer discriminativeness between pre-trained and fine-tuned networks.
- Layer selection: Compared with the pre-trained network, the fine-tuned network has dramatically more LES neurons, particularly from pool5 onward.The passage also reports that convolutional layers contain a larger number of these neurons.
IV. EXPERIMENTS
The experiments evaluate FaceNet2ExpNet on four widely used facial-expression databases and refer to the method as FN2EN.
- Datasets: FaceNet2ExpNet is evaluated on CK+, Oulu-CASIA, the Toronto Face Database, and Static Facial Expression in the Wild.The image counts for different expressions are reported in Table II.
A. Implementation
The implementation applies standard face preprocessing and augmentation, then trains the model in two stages with specified optimization settings.
- Optimization: Training uses a batch size of 64, momentum 0.9, and dropout 0.5.These settings accompany the stated preprocessing and augmentation pipeline.
- Optimization: The first training stage uses a 1e-7 learning rate initially, decreases it after 100 epochs, and runs for 300 epochs because the regression loss is initially large.Gradient clipping allowed a larger learning rate but did not improve results over the small-learning-rate setting.
- Optimization: The second stage randomly initializes the fully connected layer from a Gaussian distribution before continuing joint training.The supplied passage ends while describing the remaining training configuration.
B. Neuron Visualization
Neuron visualizations examine learned expression semantics and CK+ classification behavior, while dataset and benchmark passages provide the experimental context and reported accuracy gains.
- Neuron Visualization: High-response images are averaged to visualize hidden-neuron semantics on CK+ and Oulu-CASIA.The visualizations let humans assign semantic concepts to individual neurons.
- Neuron Visualization: On CK+, the model captures expression-specific properties such as anger’s frowned eyebrows and closed mouth and surprise’s wide-open mouth and eyes.The visualized neurons are interpreted as encoding recognizable facial-expression structure.
- CK+: CK+ contains 1,308 images formed from selected frames of 327 expression-labeled videos and is split into 10 subject-based folds.Each source video contributes its first frame and last three frames.
- CK+: 98.6% average accuracy is achieved for six CK+ classes versus the previous best 97.3%, while eight classes reach 96.8% versus 92.1%.The comparison includes traditional, deep-learning, image-based, and image-sequence methods.
- CK+: In the eight-class CK+ confusion matrix, disgust and fear are perfectly classified, whereas contempt is the most difficult class.The paper attributes contempt’s difficulty to fewer training images and subtle facial presentation, while visualization still captures its raised mouth corner.
D. Oulu-CAS VIS
On Oulu-CASIA, FaceNet2ExpNet improves average accuracy over the previous best method. Its confusion pattern is strongest for fear and happy, while angry is most often confused with disgust.
- The Oulu-CASIA evaluation uses 1,440 images from Strong-illumination VIS videos and subject-independent tenfold cross-validation.
- FaceNet2ExpNet gains 3.1% in average accuracy over the previous best performance achieved by PPDN.
- Fear and happy are recognized well, whereas angry is the hardest expression and is mostly confused with disgust.
E. TFD
On TFD, FaceNet2ExpNet achieves the best reported performance, exceeding a strong bootstrap-recon baseline. Fear remains the weakest class and is easily confused with surprise because their facial appearances are similar.
- TFD contains 4,178 images with seven expression labels, and results are averaged across five test folds.
- FaceNet2ExpNet performs best on TFD, significantly outperforming bootstrap-recon by 2%.
- The fine-tuned VGG face baseline reaches 86.7% average accuracy, nearly matching the 86.8% state-of-the-art result.
- Fear has the lowest recognition rate and is easily confused with surprise, whose images share similarly wide-open mouths and eyes.
F. SFEW
On unconstrained SFEW, FaceNet2ExpNet achieves the best recognition rates in both training-data settings. With additional FER2013 data, it nearly closes the gap to the runner-up while using a compact, fast network.
- SFEW contains 891 training images from film clips, with large pose variation and validation-only evaluation because test data are unavailable.
- FaceNet2ExpNet achieves the best recognition rates on SFEW both with SFEW training images alone and with additional FER2013 data.
- With more training data, the method surpasses Multiple Deep Network Learning by almost 3%.
- The network has 11M parameters versus 138M for the VGG-16 baseline and processes one image in 3ms on a single Titan X GPU.
- The paper concludes that FaceNet2ExpNet improves visual feature representation and outperforms state-of-the-art methods across four public datasets.