Source-linked AI summary
Strengths and Weaknesses of Deep Learning Models for Face Recognition Against Image Degradations
Klemen Grm, Vitomir Štruc, Anais Artiges, Matthieu Caron, Hazim Kemal Ekenel
TL;DR
The paper addresses the scarcity of systematic evidence about the strengths and weaknesses of deep CNN face-recognition models under specific covariates. It evaluates four models on LFW using controlled image degradations and model-related comparisons, finding that noise, blur, missing data, and brightness are most damaging while contrast, compression, color, and descriptor strategy have limited effects.
Problem
Systematic assessments of deep face-recognition robustness to specific variations remain limited, making model strengths and weaknesses difficult to understand.
Method
The study evaluates four pretrained CNNs on LFW face verification using controlled image-quality degradations, color information, and descriptor-computation comparisons.
Results
High levels of noise, blur, missing data, and brightness substantially harm all models, whereas contrast and JPEG compression have limited effects.
Takeaways & Limitations
No single architecture is most robust to every covariate, although appropriate architecture choices and training procedures can yield relative robustness to common degradations.
Abstract
from arXiv · showhide
Deep convolutional neural networks (CNNs) based approaches are the state-of-the-art in various computer vision tasks, including face recognition. Considerable research effort is currently being directed towards further improving deep CNNs by focusing on more powerful model architectures and better learning techniques. However, studies systematically exploring the strengths and weaknesses of existing deep models for face recognition are still relatively scarce in the literature. In this paper, we try to fill this gap and study the effects of different covariates on the verification performance of four recent deep CNN models using the Labeled Faces in the Wild (LFW) dataset. Specifically, we investigate the influence of covariates related to: image quality -- blur, JPEG compression, occlusion, noise, image brightness, contrast, missing pixels; and model characteristics -- CNN architecture, color information, descriptor computation; and analyze their impact on the face verification performance of AlexNet, VGG-Face, GoogLeNet, and SqueezeNet. Based on comprehensive and rigorous experimentation, we identify the strengths and weaknesses of the deep learning models, and present key areas for potential future research. Our results indicate that high levels of noise, blur, missing pixels, and brightness have a detrimental effect on the verification performance of all models, whereas the impact of contrast changes and compression artifacts is limited. It has been found that the descriptor computation strategy and color information does not have a significant influence on performance.
I. INTRODUCTION
The paper addresses the limited systematic evidence on how deep face-recognition models respond to specific image and model covariates. It evaluates four CNNs to identify detrimental factors, model differences, and future research directions.
- Unconstrained datasets combine many image variations, making it difficult to isolate sources of deep CNN face-recognition errors.
- The study examines image-quality factors including blur, compression, noise, brightness, contrast, and missing data, alongside color information and descriptor computation.
- Four state-of-the-art CNN models—AlexNet, VGG-Face, GoogLeNet, and SqueezeNet—are comparatively evaluated on the LFW face-verification benchmark.
- The analysis identifies the most detrimental covariates affecting verification and points to potential areas for improving deep CNN models.
- The paper also makes the trained models publicly available to the research community.
II. RELATED WORK
Prior work analyzes deep-model robustness, representations, and image covariates across recognition tasks. This paper extends earlier studies with a broader, more systematic face-verification evaluation on LFW.
- Model-analysis studies examine deep-model characteristics directly or alongside another contribution, reflecting the importance of understanding strengths and weaknesses for real-world applications.
- Earlier image-classification experiments found noise and blur to be the most detrimental covariates among noise, blur, contrast, and JPEG compression.
- Other related work compared traditional and deep models while examining preprocessing and color information, finding deep learning models had an edge over traditional models.
- This work extends preliminary studies to LFW face verification with more CNN models and a wider range of image-quality and model-related covariates.
- Representation studies investigated transformation properties such as equivariance, invariance, and equivalence, including rotation, mirroring, and affine transformations.
III. METHODOLOGY
The methodology introduces four CNN models, the LFW dataset, model-training procedures, and the image-quality and model-related covariates used for evaluation.
- The methodology first presents the evaluation approach and four selected CNN models, then describes the dataset, training procedure, and studied covariates.
A. Evaluation methodology
The evaluation uses pretrained CNN feature outputs as face descriptors, compares descriptor pairs with cosine similarity, and measures verification accuracy under LFW cross-validation.
- Each pretrained deep model maps an input face image x to an image descriptor y through y = f(x).
- A cosine-similarity score between two image descriptors is compared with a predefined threshold to classify matching and non-matching identities.
- One image in each verification pair is artificially degraded while the other remains unaltered, enabling direct observation of quality-related performance changes.
- Performance is reported as mean and standard deviation of verification accuracy using LFW’s 10-fold cross-validation protocol.
B. Deep CNN Models
The paper evaluates four representative deep CNN architectures for face recognition, differing in depth, connectivity, parameterization, and descriptor-generation design.
- The evaluation covers AlexNet, VGG-Face, GoogLeNet, and SqueezeNet as representative architectures for recognition problems.
- AlexNet uses five convolutional layers, three fully connected layers, and rapid downsampling through strided convolutions and max-pooling.
- VGG-Face is a 16-layer network using repeated 3 × 3 convolutions followed by max-pooling and fully connected layers.
- GoogLeNet uses Inception modules and global average pooling, producing a 2048-channel feature vector without fully connected layers.
- SqueezeNet reduces parameters and computational complexity with squeeze layers and shortcut connections, then produces a 2048-unit image descriptor.
- The models are used as black-box feature extractors, with feature-space distances supplying similarity scores for biometric verification.
C. Datasets
The study trains and evaluates the four models on separate face datasets, using VGG face for training and LFW for standardized verification testing.
- The models are trained on the VGG face dataset and evaluated on the Labeled Faces in the Wild dataset.
- The retrieved VGG face version contains approximately 1.8×10^6 images from a reported dataset of around 2.6×10^6 images and 2622 identities.
- All four CNNs are trained from scratch on the same VGG face data using an added fully connected softmax layer and Adam optimization.
- LFW contains 13233 images of 5749 subjects with predefined protocols and substantial variation in pose, lighting, expression, and background.
D. Performance Covariates
The paper studies verification covariates arising from input-image quality and deep-model characteristics, including controlled degradations, architecture, color, and descriptor computation.
- The covariates are grouped into image-quality factors and model-related factors affecting deep face-recognition performance.
- Image-quality covariates: Image-quality experiments apply controlled distortions to probe images, including blur, JPEG compression, Gaussian and salt-and-pepper noise, contrast, brightness, and missing pixels.
- Image-quality covariates: Blur is simulated with Gaussian filters whose standard deviation σ varies from 2 to 20, generating 19 degraded probe sets.
- Image-quality covariates: Brightness is varied with factors from 1.5 to 9, while contrast is evaluated at 15 factors ranging from 0.03 to 0.79.
- Image-quality covariates: Missing-pixel regions are removed around prominent facial areas, producing degradations similar in effect to partial occlusions.
- Model-related covariates: Model-related experiments vary architecture, color space, and descriptor computation, including direct facial-area descriptors and averaged representations from sampled patches.
IV. EXPERIMENTAL RESULTS AND DISCUSSION
The experimental section assesses the selected models' strengths and weaknesses through image-quality experiments followed by model-related experiments.
- The experiments are designed to assess the strengths and weaknesses of four selected deep models.
- The study first presents experiments on image-quality covariates.
- It then reports results for the model-related covariates described earlier.
A. Impact of image-quality covariates
Image degradations affect the four deep models unevenly: blur, noise, brightness, and missing data can substantially reduce verification performance, while JPEG compression and contrast changes are comparatively limited. The experiments also reveal model- and region-specific differences in robustness.
- Blur and compression: Blur significantly reduces verification performance for all models, with GoogLeNet losing accuracy faster than the other three.Performance drops quickly as the Gaussian-filter standard deviation increases.
- Blur and compression: JPEG compression leaves all models mostly unaffected until the compression quality reaches its lowest possible value.At quality 0, only DC components remain unaltered, so each MCU becomes a constant color.
- Noise: VGG-Face performs best and most robustly under Gaussian and salt-and-pepper noise, whereas GoogLeNet is affected most.AlexNet and SqueezeNet perform more or less the same under both noise types.
- Brightness and contrast: Brightness increases significantly harm all models, while contrast reduction has limited impact, with accuracy remaining above 0.9 after more than 60% contrast removal.Brightness affects the models to more or less the same extent; VGG-Face remains the highest-accuracy model in absolute terms.
- Missing data: Missing information is most detrimental around the periocular region, followed by the eyes, nose, and mouth; GoogLeNet shows the smallest performance variations.VGG-Face is especially affected by missing data around the periocular region.
- Implications: The authors conclude that image quality is crucial and recommend quality assessment, image enhancement, or data augmentation to improve robustness.These measures are presented as ways to mitigate image-quality problems in deep-learning face recognition.
B. Impact of model-related covariates
Model-related covariates produce smaller differences than image degradations, but architecture, descriptor computation, and color handling still reveal distinct robustness patterns. VGG-Face has a slight overall robustness advantage, while descriptor averaging and color removal have limited effects.
- CNN architecture: VGG-Face has a slight overall robustness advantage, SqueezeNet and AlexNet perform almost identically, and the implemented GoogLeNet is least robust.Radar-chart area represents performance, while similarity among color-coded curves represents robustness across covariates.
- Descriptor computation: The experiments compare four descriptor-computation strategies on original LFW images without degradations using box-and-whisker plots over ten verification folds.The setup evaluates how descriptor construction affects verification performance.
- Descriptor computation: Averaging generated patch representations marginally benefits SqueezeNet and VGG-Face, with more than one patch increasing verification accuracy by 1%-2%.The reported trend concerns descriptor computation on the original, undegraded LFW images.
- Color information: All models perform best when target and probe images are both in color, but removing color information generally causes only marginal accuracy drops.The models were trained exclusively on color data; AlexNet is the exception to the marginal-drop pattern.
- Color information: The difference in performance is not statistically significant, suggesting that two-thirds of the input information may be redundant in the models’ architecture.This observation concerns the comparison involving reduced color information.
V. CONCLUSION
All four deep CNN models degrade substantially on lower-quality images, but their vulnerabilities differ by covariate. Blur, noise, brightness, and missing data are especially damaging, while grayscale input and contrast changes have limited effects; no single architecture is robust to every covariate.
- All four models degrade quickly and significantly when evaluated on lower-quality images than those used for training.
- Blurring is the most consistent performance degradation, followed by noise, brightness, and missing data; contrast and JPEG compression have marginal effects.
- A larger curve area in Figure 6 indicates better performance, while closer differently colored curves indicate greater robustness to image-quality degradations.
- Figure 7 compares descriptors computed from 1, 5, 10, or 30 face patches across the 10 LFW verification folds; Figure 8 compares color-color, color-grayscale, and grayscale-grayscale inputs.
- Grayscale conversion negligibly affects most models despite their training on full-color images, indicating limited reliance on color information.
- No architecture is significantly more robust across all covariates: VGG-Face is strongest against noise but weakest under brightness changes, whereas GoogLeNet performs worst on noise and blur but slightly better under reduced contrast.