Source-linked AI summary
A Comprehensive Analysis of Deep Learning Based Representation for Face Recognition
Mostafa Mehdipour Ghazi, Hazim Kemal Ekenel
TL;DR
Face-recognition representations had been tested extensively on wild datasets, but their robustness to specific pose, illumination, occlusion, and misalignment variations remained insufficiently assessed. The paper evaluates pretrained VGG-Face and Lightened CNN representations across these conditions and finds that preprocessing is especially beneficial when training data lacks the tested variations, while misalignment remains tolerable.
Problem
Deep face representations had not been thoroughly assessed under individual appearance variations such as pose, illumination, occlusion, and misalignment.
Method
The study uses higher-layer features from pretrained VGG-Face and Lightened CNN models and evaluates them across five face datasets covering pose, illumination, occlusion, and misalignment.
Results
Across the tested conditions, deep representations provide promising results but fall below reported state-of-the-art performance, especially when tested variations are scarce in CNN training data.
Takeaways & Limitations
Preprocessing for pose and illumination normalization, or including these variations during training, substantially improves the practical robustness of pretrained deep representations.
Takeaways & Limitations
Face recognition under mismatched conditions remains challenging, particularly when only limited task-specific data are available.
Abstract
from arXiv · showhide
Deep learning based approaches have been dominating the face recognition field due to the significant performance improvement they have provided on the challenging wild datasets. These approaches have been extensively tested on such unconstrained datasets, on the Labeled Faces in the Wild and YouTube Faces, to name a few. However, their capability to handle individual appearance variations caused by factors such as head pose, illumination, occlusion, and misalignment has not been thoroughly assessed till now. In this paper, we present a comprehensive study to evaluate the performance of deep learning based face representation under several conditions including the varying head pose angles, upper and lower face occlusion, changing illumination of different strengths, and misalignment due to erroneous facial feature localization. Two successful and publicly available deep learning models, namely VGG-Face and Lightened CNN have been utilized to extract face representations. The obtained results show that although deep learning provides a powerful representation for face recognition, it can still benefit from preprocessing, for example, for pose and illumination normalization in order to achieve better performance under various conditions. Particularly, if these variations are not included in the dataset used to train the deep learning model, the role of preprocessing becomes more crucial. Experimental results also show that deep learning based representation is robust to misalignment and can tolerate facial feature localization errors up to 10% of the interocular distance.
1. Introduction
The paper examines whether deep face representations remain effective under pose, illumination, occlusion, and misalignment variations that are underassessed in unconstrained benchmarks. It evaluates pretrained VGG-Face and Lightened CNN models and finds that preprocessing is especially valuable when training data lacks the tested variations.
- Approach: The paper studies transfer learning by using higher-layer weights from publicly available, pretrained face-recognition CNNs to extract representations.Direct use of pretrained weights is particularly suitable when the target dataset is small.
- Study scope: The study evaluates deep face representations under pose, illumination, occlusion, and misalignment conditions using pretrained VGG-Face and Lightened CNN models.The models are assessed across five face datasets representing these appearance variations.
- Research gap: Deep learning face representations had been extensively tested on wild datasets, but their ability to handle individual appearance variations had not been comprehensively assessed.The paper identifies this as the central evaluation gap.
- Main findings: Preprocessing or variation-aware training is needed because deep learning does not achieve state-of-the-art performance against pose, illumination, and occlusion by itself.The paper specifically discusses pose and illumination normalization as complementary preprocessing.
2. Related Work
Related work contrasts traditional handcrafted-feature pipelines with deep CNN face-recognition systems trained on increasingly large datasets. Representative methods such as DeepFace and FaceNet achieved strong LFW and YTF accuracies.
- Traditional methods: Traditional systems commonly extracted LBP, SIFT, or HOG features before classifying identities with SVMs or nearest neighbors.Deep architectures replaced this hand-crafted feature-and-classifier sequence with learned representations.
- Deep learning transition: Deep learning architectures achieved impressive face-recognition results as large datasets and computational resources became available.This followed earlier pipelines based on handcrafted features and conventional classifiers.
- Representative methods: DeepFace achieved 97.35% accuracy on LFW and 91.4% on YTF using a nine-layer CNN, 3D alignment, and an ensemble of CNNs.The model was trained on four million facial images from over 4,000 identities.
- Representative methods: FaceNet achieved 99.63% on LFW and 95.12% on YTF by mapping roughly aligned faces into a compact Euclidean space for similarity measurement.Its training data contained 100 to 200 million images representing around eight million identities.
3. Methods
The method extracts face representations from pretrained VGG-Face and Lightened CNN models and tests them under controlled appearance variations. It combines model features with alignment, illumination normalization, or contrast enhancement when needed.
- 3.1. VGG-Face Network: VGG-Face is a 144-million-parameter, 16-layer CNN trained on 2.6 million facial images from 2,622 identities.It accepts 224 × 224 color patches and uses ReLU activations in its convolutional layers.
- 3.2. Lightened CNN: Lightened CNN provides a lower-complexity alternative using Max-Feature-Map activation and models containing 3,961K or 3,244K parameters.Its two variants use different convolutional architectures and dimensionality-reduction layers.
- 3.3. Face Representation with CNN Models: Features are extracted from layers before the final dropout and Softmax components to evaluate robustness across appearance variations.The selected layers are FC6 for VGG-Face and FC1 for Lightened CNN.
- 3.3. Face Representation with CNN Models: The experiments use alignment, illumination normalization, and contrast enhancement when needed, with eye-center alignment and model-specific input resizing.Images are scaled to 224 × 224 or 128 × 128 pixels after cropping.
- Experimental protocol: Neutral frontal images form gallery sets, while probe images contain pose, illumination, occlusion, and misalignment variations.The AR database includes neutral, sunglasses, and scarf conditions across two sessions.
4. Experiments and Results
Experiments evaluate VGG-Face and Lightened CNN representations across occlusion, illumination, pose, and misalignment conditions using multiple face datasets. Results show strong but variation-dependent robustness, with preprocessing improving performance for unrepresented illumination and pose changes.
- Experimental Setup: The experiments use aligned and cropped facial images from AR, CMU PIE, Extended Yale B, and color FERET, with frontal or neutral training samples and varied-condition testing samples.The Extended Yale setup trains on near-frontal illumination and tests on progressively stronger illumination subsets; FERET trains on frontal faces and tests on six non-frontal poses.
- AR Face Database – Face Occlusion: Deep representations struggle with upper-face occlusion from sunglasses, and VGG-Face is more robust than the Lightened CNN models in the AR experiments.The results indicate that occlusion-specific training on large datasets may be needed for stronger performance.
- CMU PIE Database – Illumination Variations: VGG-Face remains robust to CMU PIE illumination variations, although its accuracies are slightly below state-of-the-art illumination-robust methods.The findings motivate further improvement through illumination-based preprocessing.
- Extended Yale Dataset – Illumination Changes: Preprocessing improves accuracy under strong illumination changes in Extended Yale subsets 4 and 5, where illumination normalization and contrast enhancement are applied.Deep representations are robust to the smaller variations in subsets 2 and 3, but performance degrades as illumination strength increases.
- Color FERET Database – Pose Variations: VGG-Face features handle pose variations up to 67.5 degrees, but performance drops significantly for profile images and can improve with pose normalization.The FERET experiments use frontal images for training and six non-frontal poses for testing.
- FRGC Database – Misalignment: Deep representations tolerate facial feature localization errors up to 10% of the interocular distance, demonstrating robustness to misalignment.This robustness contrasts with the known sensitivity of traditional appearance-based face recognition algorithms.
5. Summary and Discussion
The study finds that deep face representations are promising but remain vulnerable under mismatched conditions, especially when training data lack those variations. Preprocessing and model choice can improve robustness, while misalignment tolerance is a notable strength.
- The evaluation covers pose, illumination, occlusion, and misalignment using VGG-Face and Lightened CNN across five face datasets.
- Deep representations provide promising results, but performance remains below state-of-the-art methods, especially for conditions scarce in CNN training data.The authors identify mismatched conditions and limited task-specific data as continuing challenges.
- Preprocessing for pose and illumination normalization, or accounting for these variations during training, is proposed to reduce the performance weakness.
- Deep representations tolerate facial feature localization errors up to 10% of the interocular distance, demonstrating robustness to misalignment.
- VGG-Face transfers better than Lightened CNN, while its FC6 features are more pose-robust and FC7 features are more illumination-robust.The paper attributes VGG-Face's transferability to its more sophisticated architecture and contrasts Lightened CNN's faster design.