Source-linked AI summary
When Face Recognition Meets with Deep Learning: an Evaluation of Convolutional Neural Networks for Face Recognition
Guosheng Hu, Yongxin Yang, Dong Yi, Josef Kittler, William Christmas, Stan Z. Li, Timothy Hospedales
TL;DR
The paper asks why CNNs work well for face recognition and how to design architectures when prior comparisons use different training databases. It systematically evaluates CNN architectures and implementation choices using publicly available LFW data, finding that network fusion and metric learning substantially improve performance.
Problem
The paper addresses the open questions of why CNNs work well for face recognition and how to design a good architecture, amid unfair comparisons from differing training databases.
Method
The authors systematically compare CNN architectures and implementation choices, training three LFW-adapted architectures and evaluating CNN features with metric learning on a common ground.
Results
Network fusion and metric learning such as Joint Bayesian significantly improve face-recognition performance.
Takeaways & Limitations
Different networks capture information from different facial regions and scales, forming a more powerful face representation when fused.
Takeaways & Limitations
The architectures are designed for a training set much smaller than those used by state-of-the-art methods, and Joint Bayesian assumes Gaussian distributions for face-pair hypotheses.
Abstract
from arXiv · showhide
Deep learning, in particular Convolutional Neural Network (CNN), has achieved promising results in face recognition recently. However, it remains an open question: why CNNs work well and how to design a 'good' architecture. The existing works tend to focus on reporting CNN architectures that work well for face recognition rather than investigate the reason. In this work, we conduct an extensive evaluation of CNN-based face recognition systems (CNN-FRS) on a common ground to make our work easily reproducible. Specifically, we use public database LFW (Labeled Faces in the Wild) to train CNNs, unlike most existing CNNs trained on private databases. We propose three CNN architectures which are the first reported architectures trained using LFW data. This paper quantitatively compares the architectures of CNNs and evaluate the effect of different implementation choices. We identify several useful properties of CNN-FRS. For instance, the dimensionality of the learned features can be significantly reduced without adverse effect on face recognition accuracy. In addition, traditional metric learning method exploiting CNN-learned features is evaluated. Experiments show two crucial factors to good CNN-FRS performance are the fusion of multiple CNNs and metric learning. To make our work reproducible, source code and models will be made publicly available.
1. Introduction
The paper addresses why CNNs perform well for unconstrained face recognition and how to design them fairly when existing systems use different, often private, training databases. It evaluates architectures and implementation choices on a common LFW-based ground and emphasizes reproducibility.
- CNN features are more robust than hand-crafted features to pose, illumination, expression, and occlusion variations in unconstrained face recognition.
- Face-recognition CNN architectures require systematic evaluation because face alignment makes this task different from object recognition.
- Different private training databases can make published CNN architecture comparisons unfair, motivating evaluation on a common ground.
- The study trains and evaluates CNNs on publicly available LFW data to clarify component contributions and make the comparison reproducible.
- The evaluation compares CNN architectures, data augmentation, colour versus grey pixels, similarity measures, network fusion, and downstream metric learning.
- The authors will publicly release CNN architectures, trained networks, source code, and a competitive reproducible baseline.
2. Related Work
Prior face-recognition CNNs differ substantially in architecture, alignment strategy, training data, and supervision. These differences motivate systematic comparison on a common ground.
- DeepFace is an 8-layer CNN trained on four million facial images from 4,000 subjects.
- DeepFace uses one pooling layer as a balance between robustness to local transformations and preservation of texture details in aligned faces.
- DeepFace additionally uses 3D alignment to address out-of-plane rotations that 2D similarity alignment cannot handle.
- DeepID fuses features from many small CNNs processing RGB and grey facial crops, producing a 19,200-dimensional representation.
- DeepID2 adds verification supervision to identification supervision, while DeepID2+ extends supervision to all convolutional layers.
- WebFace trains a deeper 17-layer CNN on a public database containing around 10,000 subjects and 500,000 images.
- The published systems differ in architectures and implementation choices, motivating systematic evaluation on a common ground.
3. Methodology
The methodology evaluates CNN-based face recognition on LFW using architectures sized to the available training data, then applies Joint Bayesian metric learning to improve feature discrimination.
- CNNs are trained only on public LFW data, enabling reproducible comparison while motivating architectures adapted to this smaller training set.The paper introduces three architectures for the LFW setting rather than directly reusing models trained on larger private databases.
- The proposed CNN-S, CNN-M, and CNN-L architectures vary in size: CNN-S and CNN-M use three convolutional layers, whereas CNN-L uses four.CNN-M has more filters than CNN-S; all networks also use two fully connected layers.
- CNN training uses ReLU activations and a softmax output for predicting one of K mutually exclusive subject classes.The learning rate is 0.001 and the batch size is 100 for all three networks.
- Joint Bayesian metric learning is evaluated as a downstream method for making CNN-learned features more discriminative in face verification.Metric learning is independent of feature extraction, so both hand-crafted and learning-based features can be supplied to it.
- Joint Bayesian models verification as a Bayesian decision between intra-personal matched and extra-personal unmatched hypotheses using feature-pair likelihoods.The likelihoods are assumed Gaussian, with covariance parameters learned from data using expectation-maximization.
- A face feature is modeled as the sum of independent Gaussian identity and intra-personal variation variables, whose covariance structures determine the Joint Bayesian score.The identity and variation covariances are represented by Sµ and Sε.
4. Evaluation
The evaluation systematically tests CNN architectures and implementation choices for face recognition on LFW, using a common protocol and reproducible preprocessing. It finds that feature normalization, low-dimensional representations, network fusion, and metric learning affect recognition performance, while some choices offer limited gains.
- Evaluation setup: LFW evaluation uses 10-fold cross validation, with nine predefined splits for training and one 600-pair split for testing each time.The face recognition rate is measured by mean classification accuracy and standard error of the mean.
- Architectures: CNN-M achieves the best face recognition performance among the three architectures evaluated using only LFW data.Its face recognition rate of 0.7882 is used as the baseline for subsequent investigations.
- Feature distance: Cosine distance provides the best recognition performance among six tested distance measures.Cosine and correlation achieve the highest recognition rates, but cosine has the smaller standard deviation.
- Learned feature analysis: 0.7927±0.0126 with normalization versus 0.7882±0.0118 without normalization shows that feature normalization improves recognition.The normalized features are standardized before cosine-distance computation.
- Learned feature analysis: 16 PCA dimensions achieve recognition rates comparable to the original 160-dimensional feature space.The reduction can lower storage and computation requirements for large-scale applications and mobile devices.
- Network fusion and metric learning: Network fusion and Joint Bayesian metric learning significantly improve face recognition performance.Fusion concatenates features from networks trained on different facial regions and scales, while Joint Bayesian is applied after PCA reduction of the fused representation.
5. Conclusions
The paper presents a rigorous empirical evaluation of CNN-based face recognition systems on common ground. It identifies network fusion and metric learning as the two most important factors affecting performance.
- The study quantitatively evaluates CNN architectures and implementation choices for face recognition on common ground.
- Network fusion improves recognition by combining information from different facial regions and scales into a powerful representation.
- Metric learning, including Joint Bayesian, greatly improves the effectiveness of CNN-learned features for face recognition.
- Network fusion and metric learning are identified as the two most important factors affecting CNN performance and future investigation.