Source-linked AI summary
Sparsifying Neural Network Connections for Face Recognition
Yi Sun, Xiaogang Wang, Xiaoou Tang
TL;DR
Deep face-recognition networks have parameter counts constrained by available training data. The paper learns sparse ConvNets by pruning connections according to neural correlations and iteratively retraining each sparser model from a denser model's weights. The sparse model raises face-verification accuracy from 98.95% to 99.30%, while baseline performance is retained with 12% of the original parameters.
Problem
Deep neural-network parameter counts are constrained by the amount of training data, motivating lower-parameter models for face recognition.
Source: §1 · p. 1Method
Results
98.95% to 99.30% face verification accuracy is reported after sparsifying the baseline, with its performance retained at 12% of the original parameters.
Source: §1 · p. 1Takeaways & Limitations
A previously learned denser model is critical for the sparser model to continue learning effective face representations.
Source: §6 · p. 8FaceNet was not chosen as a baseline because its around 200 million training samples were nearly 700 times the paper's data and could not be reproduced by the authors.
Source: §3 · p. 3
Abstract
from arXiv · showhide
This paper proposes to learn high-performance deep ConvNets with sparse neural connections, referred to as sparse ConvNets, for face recognition. The sparse ConvNets are learned in an iterative way, each time one additional layer is sparsified and the entire model is re-trained given the initial weights learned in previous iterations. One important finding is that directly training the sparse ConvNet from scratch failed to find good solutions for face recognition, while using a previously learned denser model to properly initialize a sparser model is critical to continue learning effective features for face recognition. This paper also proposes a new neural correlation-based weight selection criterion and empirically verifies its effectiveness in selecting informative connections from previously learned models in each iteration. When taking a moderately sparse structure (26%-76% of weights in the dense model), the proposed sparse ConvNet model significantly improves the face recognition performance of the previous state-of-the-art DeepID2+ models given the same training data, while it keeps the performance of the baseline model with only 12% of the original parameters.
1. Introduction
The paper introduces sparse ConvNets to reduce parameters in deep face-recognition networks, with sparsity improving verification performance while allowing a much smaller model. It prunes connections by neural correlation and learns sparser models iteratively from previously trained denser models.
1. Introduction
- The method starts from a high-performance VGG-like network and introduces sparse connections to reduce deep ConvNet parameters.The baseline is trained on approximately 300,000 face images, the same training set used by DeepID2+.Source: §1 · p. 1
- Figure 1 contrasts pruning in locally connected and convolutional layers, where convolutional connections sharing a color also share weights.The figure marks reserved connections with solid lines and pruned connections with dashed lines.Source: Figure 1 · p. 1, §1 · p. 2
- 98.95% to 99.30% face verification accuracy, a 33% error-rate reduction, is achieved by adding sparsity to the baseline model.The baseline model retains its performance with 12% of its original parameters.Source: §1 · p. 1
- Neural-correlation pruning preserves more connections between strongly correlated neurons than between weakly correlated neurons.Correlation is measured from the magnitude of neural-activation correlations.Source: §1 · p. 2
- Each iteration sparsifies one additional layer and retrains the model using the previously trained model's weights and neural correlations.Direct training of a sparse model was difficult and led to inferior performance; the authors conjecture denser models provide better initialization.Source: §1 · p. 2
2. Related work
Prior work pruned parameters using second-derivative saliency, weight magnitude, and probabilistic weight estimates, while other compression approaches used low-rank approximation or distillation. This paper investigates neural-activation correlations as a criterion for connection significance.
2. Related work
- Earlier pruning approaches used second-derivative criteria [25], small-magnitude weights [11], or Gaussian weight estimates [7].Optimal Brain Damage reduced parameters four to eight times without loss of prediction ability; some cited methods did not require fine-tuning after pruning.Source: §2 · p. 2
- Neural correlations between activations are reported as better indicators of connection significance than weight magnitudes or second derivatives.The paper studies this criterion on face recognition and also finds that a learned dense model is critical for subsequent sparse-model learning.Source: §2 · p. 2
- Weight pruning is presented as complementary to low-rank layer approximation [13] [18] [34] and knowledge distillation [15].The paper suggests pruning a distilled small model as one possible combination.Source: §2 · p. 2
3. Baseline model
The baseline is a VGG-like ConvNet modified with locally connected layers for face-region-specific features, trained with identification-verification supervision. On LFW, it reaches 98.95% accuracy using approximately 300,000 training images.
3. Baseline model
- The baseline is VGG-like, but replaces its last two convolutional layers with locally connected layers to learn different features in different face regions.A 512-dimensional fully connected layer follows the second locally connected layer; Table 1 gives the detailed structure.Source: §3 · p. 2, Table 1 · p. 3
- Joint identification-verification supervision is applied at the final fully connected layer and at selected earlier layers to shape face representations.The paper also uses ReLU activations and dropout rates of 30% and 50% in the last locally connected and fully connected layers, respectively.Source: §3 · p. 2
- 98.95% face verification accuracy on LFW is reported for the single baseline model, versus 98.70% for a single DeepID2+ model trained on the same approximately 300,000 face images.The paper attributes the improvement mainly to larger input dimensions and increased model depth.Source: §3 · p. 2
- FaceNet's reported 99.63% LFW accuracy is not used as the baseline because it required around 200 million training samples, nearly 700 times the paper's data.The authors state that reproducing FaceNet was impossible for them.Source: §3 · pp. 2–3
4. Sparse ConvNets
Sparse ConvNets are learned by progressively pruning a trained baseline and retraining each sparser model from the preceding model’s weights. Connections are selected using neural correlations, with the procedure adapted for shared convolutional weights.
4. Sparse ConvNets
- Layer-wise pruning starts from a trained baseline, sparsifies one layer at a time, and initializes each retrained model with the preceding, denser model’s weights.Pruning proceeds from fully connected to locally connected and convolutional layers, whose parameter redundancy allows substantial reductions without degrading performance.Source: §4 · p. 3
4.1. Sparsify connections
- Connections are selected by neural correlation because more-correlated neurons are considered more predictive, while a small portion of weakly correlated connections may provide complementary predictions.The complementarity of weakly correlated neurons is presented as a conjecture.Source: §4.1 · pp. 3–4
- For fully and locally connected layers, positive and negative correlations are handled separately, sampling more high-ranked connections and using absolute correlation magnitude for negative coefficients.The procedure samples from both higher- and lower-ranked groups; λ is 0.75 in the experiments.Source: §4.1 · p. 4
- For convolutional layers, correlations across positions sharing a weight are aggregated by mean magnitude before selecting the corresponding weights.The selection then samples from higher- and lower-ranked mean coefficients, with λ set to 0.75 in the experiments.Source: §4.1 · p. 4
5. Experiments
Across LFW and YouTube Faces experiments using the DeepID2+ training set, the paper tests layerwise sparsity, correlation-guided selection, and initialization from denser models. Sparse ensembles outperform DeepID2+ on reported verification and identification benchmarks, while tested sparsity retains near-baseline accuracy with substantially fewer parameters.
5.1. Sparsity improves performance
- 0.9930 face verification accuracy, versus the 0.9895 baseline, accompanies reducing parameters to 74% by sparsifying layers f, 5b, and 4b.Source: §5.1 · p. 5
- Performance remains almost unchanged at 26% of the original parameters and is still close to 99% accuracy at 12%.Source: §5.1 · p. 5
5.2. Correlation guided weight selection
- Across tested sparsity configurations, correlation-based selection outperforms random and high-correlation-only selection; a small random fraction increases prediction complementarity, but too much weakens individual predictive power.Mean absolute neural correlations on selected connections also increase after retraining.Source: §5.2 · p. 6
5.3. Why do we need a denser network?
- For the same sparse structures, random initialization performs significantly worse than initialization from previously learned models, even below the baseline; the authors conjecture dense models help find good initial solutions.Source: §5.3 · p. 7
5.4. Method comparison
- Correlation pruning performs best on layers 5b and 4b, while OBD performs better on f; near-uniform selected weight-magnitude ranks suggest magnitude alone poorly indicates connection significance.Source: Table 8 · p. 7, §5.4 · pp. 7–8
5.5. Sparse ConvNet ensemble
- Across 25 face regions, mean accuracy rises from 97.26% to 97.66%, above DeepID2+ at 96.61%; on YouTube Faces, single and ensemble accuracies are 92.7% and 93.5%, versus 91.9% and 93.2%.Source: §5.5 · p. 8
- 0.9955 ± 0.0010 LFW face verification exceeds DeepID2+’s 0.9947 on the same training data; sparse ensembles also reach 0.962 closed-set and 0.864 open-set identification, versus 0.950 and 0.807.Source: §5.5 · p. 8
6. Conclusion
The paper learns sparse deep ConvNets iteratively from denser models using neural-correlation-based weight selection. Moderate sparsity improves performance, while ensembles achieve state-of-the-art face verification and identification on LFW and YouTube Faces.
6. Conclusion
- The proposed sparse ConvNets are learned iteratively from previously learned denser models using neural-correlation-based weight selection.The denser model helps avoid bad local minima and provides initialization for the sparser model.Source: §6 · p. 8
- Neural correlations are better indicators of connection significance than weight magnitudes or second derivatives.Source: §6 · p. 8
- With 26%-76% of dense-model weights, sparse ConvNets significantly improve performance over the original dense model, with performance degrading slowly at greater sparsity.Source: §6 · p. 8
- Ensembles of the proposed sparse ConvNets achieve state-of-the-art face verification and identification performance on LFW and YouTube Faces.Source: §6 · p. 8