Source-linked AI summary
Deeply learned face representations are sparse, selective, and robust
Yi Sun, Xiaogang Wang, Xiaoou Tang
TL;DR
Face-recognition systems can perform well without clearly explaining what their neurons encode or how robust those representations are. This paper develops DeepID2+ with identification-verification supervision and architectural and training-data improvements, achieving state-of-the-art benchmark results while revealing sparse, selective, and corruption-robust activations.
Problem
The paper addresses the need to understand what high-performing deep face-recognition neurons encode and how robust their representations are.
Method
The paper develops DeepID2+, enlarging hidden representations, adding supervision to early convolutional layers, expanding training data, and using identification-verification supervision.
Results
DeepID2+ sets state-of-the-art results on LFW and YouTube Faces, while its activations are moderately sparse, identity- and attribute-selective, and robust to image corruption.
Takeaways & Limitations
These properties arise naturally from large-scale training without extra regularization and show that binary activation patterns can remain effective for recognition.
Abstract
from arXiv · showhide
This paper designs a high-performance deep convolutional network (DeepID2+) for face recognition. It is learned with the identification-verification supervisory signal. By increasing the dimension of hidden representations and adding supervision to early convolutional layers, DeepID2+ achieves new state-of-the-art on LFW and YouTube Faces benchmarks. Through empirical studies, we have discovered three properties of its deep neural activations critical for the high performance: sparsity, selectiveness and robustness. (1) It is observed that neural activations are moderately sparse. Moderate sparsity maximizes the discriminative power of the deep net as well as the distance between images. It is surprising that DeepID2+ still can achieve high recognition accuracy even after the neural responses are binarized. (2) Its neurons in higher layers are highly selective to identities and identity-related attributes. We can identify different subsets of neurons which are either constantly excited or inhibited when different identities or attributes are present. Although DeepID2+ is not taught to distinguish attributes during training, it has implicitly learned such high-level concepts. (3) It is much more robust to occlusions, although occlusion patterns are not included in the training set.
1. Introduction
The paper studies how DeepID2+ achieves high face-recognition performance and identifies sparsity, selectiveness, and robustness as important properties of its learned activations.
- DeepID2+ and performance: DeepID2+ improves DeepID2 by enlarging hidden representations and adding supervision to early convolutional layers.A single net reaches 98.70% verification accuracy on LFW, compared with 96.72% for DeepID2.
- DeepID2+ and performance: Combining 25 DeepID2+ nets achieves state-of-the-art results across LFW identification and verification and YouTube Faces verification.Reported results include 99.47% LFW verification, 95.0% closed-set identification, 80.7% open-set identification, and 93.2% YouTube Faces verification.
- Activation properties: DeepID2+ activations are moderately sparse: roughly half the top-layer neurons activate for an input, and each neuron activates on roughly half the face images.Different identities have different active subsets, while same-identity images have similar activation patterns.
- Activation properties: Binarizing top-layer responses reduces LFW verification accuracy by only 1% or less, indicating that activation patterns matter more than magnitudes for recognition.The binary representation also saves storage and computation for large-scale face search.
- Activation properties: Higher-layer neurons are selective to identities and identity-related attributes, despite DeepID2+ not being trained to distinguish attributes.Some neurons are consistently excited or inhibited by particular identities or attributes, and individual neurons can recognize them accurately.
- Activation properties: Higher-layer representations are more robust to occlusion than handcrafted LBP features or lower-layer neurons, even without artificial occlusions during training.Binary activation patterns remain stable under partial occlusion, although activation magnitudes may change.
2. Related work
The related work situates DeepID2+ within deep face recognition, attribute learning, and occlusion handling. The paper reverses the usual attribute-learning direction and argues that occlusion invariance can emerge without explicit handling components.
- Deep face recognition: Deep face recognition recently surpassed systems based on low-level features through large-scale identification and joint identification-verification supervision.Large-scale identity classification produces discriminative last-layer features, while verification supervision further improves feature learning.
- Attribute learning: Prior attribute-learning methods predict attributes for face recognition, whereas this paper predicts identities first and uses identity-related features to predict attributes.
- Occlusion handling: Earlier occlusion methods explicitly modeled corrupted pixels or designed components for occlusion handling; DeepID2+ instead learns occlusion-invariant features implicitly.The paper reports this invariance without model regularization or artificial occlusion patterns in training data.
3. DeepID2+ nets
DeepID2+ extends DeepID2 with a larger network, more training data, and earlier supervision while retaining identification and verification training signals.
- DeepID2 baseline: DeepID2 uses four convolutional layers followed by a 160-dimensional feature layer supervised by identification and verification tasks.The identification task classifies each feature vector among 8192 training identities, while verification uses paired feature distances.
- DeepID2+ improvements: DeepID2+ increases each convolutional layer to 128 feature maps and expands the final representation from 160 to 512 dimensions.
- DeepID2+ improvements: DeepID2+ trains on around 290,000 images from 12,000 identities, compared with 160,000 images from 8,000 identities for DeepID2.
- DeepID2+ improvements: DeepID2+ adds supervision to lower convolutional layers, whereas DeepID2 supervises one fully connected layer and relies on back-propagated gradients below it.
4. High-performance of DeepID2+ nets
DeepID2+ is evaluated against architectural and training-data alternatives on face verification and identification benchmarks. Its 25-net system achieves high reported accuracy on LFW and YouTube Faces.
- DeepID2+ is compared with networks lacking layer-wise supervision, using less training data, or having fewer feature maps.The comparison evaluates how these design and training changes affect performance.
- DeepID2+ nets trained on 25 DeepID2 face regions improve approximately 2% accuracy on average over DeepID2 on LFW verification.The comparison uses the same 25 face regions selected by DeepID2.
- 99.47% verification accuracy is achieved on LFW and 93.2% on YouTube Faces by combining features from 25 DeepID2+ nets.After correcting wrongly labeled test pairs, the reported accuracies increase to 99.52% on LFW and 93.8% on YouTubeFaces.
- 95.0% closed-set and 80.7% open-set identification accuracies are reported on LFW.The open-set result is the Rank-1 Detection and Identification rate at a 1% False Alarm Rate.
- Figures 5 and 6 compare receiver operating characteristic curves for face verification on LFW and YouTube Faces.The paper states that DeepID2+ outperforms previous results on both datasets.
5. Moderate sparsity of neural activations
DeepID2+ activations exhibit moderate sparsity, with activation patterns carrying substantial discriminative information. Binarizing these patterns preserves much of the face-verification accuracy.
- Approximately half of the neurons activate for each image, while each neuron activates on approximately half of the validation images.For the FC-4 layer, the validation analysis uses 46,594 images and 512 neurons.
- Moderate sparsity is associated with maximal distinguishability between different identities and maximal discrimination abilities for neurons.The paper evaluates activation counts across images and neurons to verify the observed distributions.
- Binary representations with Joint Bayesian achieve 97.67% accuracy for a single net and 99.12% when combining 25 nets on LFW verification.These results sacrifice 1% or less accuracy relative to the corresponding original representations.
- Direct Hamming-distance comparison of binary codes achieves 96.45% accuracy with one net and 97.47% with 25 nets.The results show that neuron excitation or inhibition states contain most of the discriminative information.
6. Selectiveness on identities and attributes
DeepID2+ neurons selectively encode identities and facial attributes through distinct excitation and inhibition patterns. Compared with LBP features, individual DeepID2+ neurons provide substantially stronger discrimination.
- DeepID2+ features significantly outperform LBP features for attribute classification on LFW.
- Individual DeepID2+ neurons can distinguish particular identities through high activation for some identities or inhibition for others.Excitatory neurons show higher activation on the target identity, while inhibitory neurons remain near zero or low activation.
- Neurons with overlapping activation distributions have weak identity discrimination and uncertain excitation or inhibition states.
- LBP feature distributions overlap across identities and attributes, with most per-feature accuracies near the 50% random-guess level.
- Higher-layer DeepID2+ neurons also show selective excitation or inhibition for attributes including sex, race, age, and hair.
7. Robustness of DeepID2+ features
DeepID2+ features remain robust under partial and random-block occlusions, especially in deeper layers and when multiple networks are combined. This robustness arises without training on artificial occlusions.
- DeepID2+ and Joint Bayesian models are trained on original face images without artificially added occlusions, then tested under area and random-block occlusions.The tests use 10%–70% area occlusions and random blocks from 10 × 10 to 70 × 70 pixels.
- Deeper DeepID2+ layers are more robust than shallow FC-1 layers and local LBP features, while combining 25 FC-4 features provides the strongest robustness.
- At 50 × 50 block occlusions, a single DeepID2+ FC-4 feature reaches 89.2% accuracy, while 25 combined nets reach 92.4%.
- LBP accuracies begin dropping quickly beyond 20 × 20 blocks and fall to approximately 70% at 50 × 50 blocks.
- Activation patterns remain largely unchanged under both partial and random-block occlusions until occlusion becomes extensive.
8. Conclusion
DeepID2+ achieves state-of-the-art face-recognition performance while learning representations that are sparse, selective, and robust. These properties emerge through large-scale training without explicit regularization for them.
- DeepID2+ sets new state-of-the-art results on LFW and YouTube Faces for face identification and verification.
- Its learned representations are moderately sparse, highly selective to identities and attributes, and robust to image corruption.
- These properties arise naturally through large-scale training without explicitly adding corresponding model components or regularizations.
- Binary neural activation patterns are reported as efficient and effective for face recognition.