Source-linked AI summary
Covariance Pooling For Facial Expression Recognition
Dinesh Acharya, Zhiwu Huang, Danda Paudel, Luc Van Gool
TL;DR
Facial expression recognition requires capturing regional facial-landmark distortions, which the paper addresses using covariance as a second-order representation. It combines covariance pooling with manifold networks in CNNs for image and video recognition, reporting state-of-the-art image results and comparable-to-state-of-the-art video results.
Problem
Facial expression recognition requires representations that capture regional distortions of facial landmarks rather than only first-order facial features.
Method
The paper combines covariance pooling with manifold networks and conventional CNNs for end-to-end spatial pooling in images and temporal pooling of per-frame video features.
Results
The method achieves state-of-the-art results on SFEW 2.0 and RAF for image recognition, while video recognition results are comparable to state of the art.
Takeaways & Limitations
Second-order manifold networks provide a unified approach for pooling regional image features and temporal video features in facial expression recognition.
Takeaways & Limitations
Face detection failed on several SFEW 2.0 and AFEW samples, requiring validation scores based partly on randomly assigned success probabilities.
Abstract
from arXiv · showhide
Classifying facial expressions into different categories requires capturing regional distortions of facial landmarks. We believe that second-order statistics such as covariance is better able to capture such distortions in regional facial fea- tures. In this work, we explore the benefits of using a man- ifold network structure for covariance pooling to improve facial expression recognition. In particular, we first employ such kind of manifold networks in conjunction with tradi- tional convolutional networks for spatial pooling within in- dividual image feature maps in an end-to-end deep learning manner. By doing so, we are able to achieve a recognition accuracy of 58.14% on the validation set of Static Facial Expressions in the Wild (SFEW 2.0) and 87.0% on the vali- dation set of Real-World Affective Faces (RAF) Database. Both of these results are the best results we are aware of. Besides, we leverage covariance pooling to capture the tem- poral evolution of per-frame features for video-based facial expression recognition. Our reported results demonstrate the advantage of pooling image-set features temporally by stacking the designed manifold network of covariance pool-ing on top of convolutional network layers.
1. Introduction
The paper argues that facial expression recognition benefits from modeling regional landmark distortions with second-order statistics. It introduces end-to-end covariance pooling with manifold networks for image and video recognition.
- Facial expression recognition concerns distortions of facial landmarks rather than merely the presence or absence of specific landmarks.
- Covariance is proposed as a more suitable regional descriptor than first-order statistics for capturing facial-feature distortions.
- The method applies manifold networks after covariance pooling to learn second-order information with dimensionality reduction and nonlinear rectification in an end-to-end CNN.
- The framework performs end-to-end second-order-statistics pooling for both image- and video-based facial expression recognition.
- The paper reports state-of-the-art image-based facial expression recognition results.
2. Related Works
Prior facial-expression systems use standard CNNs for images and temporal models such as LSTMs or 3D convolutions for videos. Earlier covariance-based video recognition used kernel PLS, whereas this work substitutes SPD Riemannian networks and reports slight improvement.
- Facial Expression Recognition from Images: Image-based recognition commonly uses VGG, Inception, residual, and related CNN architectures, often with pretraining, fine-tuning, or ensemble score fusion.
- Covariance Pooling: Traditional CNN architectures are described as capturing first-order statistics, while covariance pooling captures second-order statistics for feature extraction.
- Facial Expression Recognition from Videos: Video-based methods summarize per-frame features using CNNs, recurrent networks such as LSTMs, or 3D convolutional networks to model temporal evolution.
- Covariance Pooling for Videos: Covariance matrices have previously summarized per-frame features, with kernel-based PLS used for recognition.
- Covariance Pooling for Videos: This work replaces kernel-based PLS with SPD Riemannian networks and obtains slight improvement.
3. Facial Expression Recognition and Covariance Pooling
The paper applies covariance pooling to spatial convolutional features and temporal video features, then uses SPD manifold layers to learn second-order representations while preserving their geometric structure.
- Covariance Pooling: Covariance pooling summarizes second-order information from feature sets for facial expression recognition.The approach targets regional facial distortions that first-order statistics may not capture as effectively.
- Covariance Pooling: Covariance matrices are regularized with a trace-scaled identity matrix when needed to obtain SPD inputs for the manifold network.The regularization adds a multiple of the trace to the covariance matrix’s diagonal entries, with λ as the regularization parameter.
- Spatial Pooling: For image recognition, final convolutional feature maps are flattened across spatial locations before covariance computation.If the feature-map dimensions are w, h, and d, flattening produces n = w × h feature vectors in R^d.
- Temporal Pooling: For video recognition, per-frame features are pooled temporally by computing their covariance matrix.The method treats fully connected-layer outputs as image-set features and uses temporal covariance to capture facial motion patterns.
- SPD Manifold Network: SPDNet uses BiMap and ReEig layers to reduce covariance dimensionality, preserve geometric structure, and introduce nonlinearity.BiMap plays a role analogous to fully connected layers, while ReEig rectifies eigenvalues; a final LogEig layer enables Euclidean operations after flattening.
4. Experiments
Experiments evaluate covariance-pooling architectures on in-the-wild image and video datasets, using aligned facial inputs and comparisons with standard and existing methods. Image results improve over baselines, while video results slightly surpass the baseline but remain difficult to compare directly across nonuniform pretraining settings.
- Datasets: The experiments use SFEW 2.0 and RAF for image recognition, and AFEW for video recognition, emphasizing in-the-wild rather than posed data.SFEW 2.0 has 958 training and 436 validation images; RAF has 12,271 training and 3,068 validation images; AFEW has 773 training and 383 validation videos.
- Face Detection and Alignment: Face detection and landmark-based alignment remove irrelevant image content and support comparison of local facial-feature variations.MTCNN detects faces and landmarks, followed by a three-point constrained affine transformation using the eyes and lip-corner midpoint.
- Image Architectures: The image experiments compare standard architectures with covariance-pooling models that insert SPD manifold-network layers after the final convolutional layer.Covariance pooling is evaluated before fully connected layers, with multiple architectures summarized in Table 2 and compared in Table 3.
- Image Results: Almost 3.7% improvement over baseline on SFEW 2.0 supports using SPDNet for facial expression recognition, while the single model also outperformed reported CNN ensembles.For SFEW 2.0, the method used the penultimate fully connected layer and was compared with prior ensemble results.
- Experimental Caveat: Face detection failed for several SFEW 2.0 and AFEW samples, and failed samples received a random uniform probability of 1/7 for correct recognition.This affects the reported validation scores for both image and video datasets.
- Video Results: The video method represents 128-dimensional per-frame features with a covariance matrix and applies SPDNet or kernel-based classifiers, with SPDNet outperforming the other tested methods.The reported comparisons include a baseline and C3D and CNN-RNN models, but their pretraining datasets were not uniform.
5. Conclusion
The paper applies SPDNet and covariance pooling to image- and video-based facial expression recognition. It reports state-of-the-art image results, comparable-to-state-of-the-art video results, and limited end-to-end video accuracy in brief experimentation.
- SPDNet uses covariance matrices of convolutional or image feature vectors for image- and video-based facial expression recognition.The approach targets second-order representations for both modalities.
- State-of-the-art results were obtained on image-based facial expression recognition problems on the SFEW 2.0 and RAF datasets.
- Video-based recognition achieved results comparable to state-of-the-art results when SPDNet was trained on image-based features.
- 32.5% accuracy was obtained in brief end-to-end training experiments for video-based facial expression recognition.The experiment used the architecture presented in Figure 8.
- The conclusion includes samples and tabulated comparisons of correctly and incorrectly predicted expressions and video-based recognition accuracies.The cited materials include class prediction samples and a video-recognition accuracy table.
6. Further Works
The paper proposes Gaussian pooling as a future extension of covariance pooling. Gaussian matrices can capture both first- and second-order statistics and may improve effectiveness.
- Gaussian pooling is proposed as an extension of covariance pooling for capturing second-order statistics.The paper notes that Gaussian matrices may further improve the effectiveness of second-order statistics.
- The Gaussian matrix uses the mean of samples and captures both first-order and second-order statistics.