Source-linked AI summary
Wasserstein CNN: Learning Invariant Features for NIR-VIS Face Recognition
Ran He, Xiang Wu, Zhenan Sun, Tieniu Tan
TL;DR
NIR-VIS face recognition must match images across sensing modalities despite large appearance differences and limited paired training data. WCNN learns compact modality-invariant representations using orthogonal subspaces, Wasserstein distribution matching, and low-rank correlation constraints. On CASIA NIR-VIS 2.0, it reaches 98.4% verification at FAR=0.1%, up from 91.0%.
Problem
NIR-VIS face recognition is difficult because heterogeneous sensing creates large feature differences while paired VIS-NIR training data remain scarce.
Method
WCNN uses one network with orthogonal identity and spectrum subspaces, Wasserstein distance between NIR and VIS feature distributions, and a low-rank correlation prior.
Results
98.4% verification rate (@FAR=0.1%) improves on the previous best 91.0% on CASIA NIR-VIS 2.0.
Takeaways & Limitations
WCNN provides a compact 128-D representation and reports improved NIR-VIS recognition performance on challenging databases.
Takeaways & Limitations
The formulation assumes shared low-level NIR-VIS features and Gaussian distributions for one subject after nonlinear feature mapping.
Abstract
from arXiv · showhide
Heterogeneous face recognition (HFR) aims to match facial images acquired from different sensing modalities with mission-critical applications in forensics, security and commercial sectors. However, HFR is a much more challenging problem than traditional face recognition because of large intra-class variations of heterogeneous face images and limited training samples of cross-modality face image pairs. This paper proposes a novel approach namely Wasserstein CNN (convolutional neural networks, or WCNN for short) to learn invariant features between near-infrared and visual face images (i.e. NIR-VIS face recognition). The low-level layers of WCNN are trained with widely available face images in visual spectrum. The high-level layer is divided into three parts, i.e., NIR layer, VIS layer and NIR-VIS shared layer. The first two layers aims to learn modality-specific features and NIR-VIS shared layer is designed to learn modality-invariant feature subspace. Wasserstein distance is introduced into NIR-VIS shared layer to measure the dissimilarity between heterogeneous feature distributions. So W-CNN learning aims to achieve the minimization of Wasserstein distance between NIR distribution and VIS distribution for invariant deep feature representation of heterogeneous face images. To avoid the over-fitting problem on small-scale heterogeneous face data, a correlation prior is introduced on the fully-connected layers of WCNN network to reduce parameter space. This prior is implemented by a low-rank constraint in an end-to-end network. The joint formulation leads to an alternating minimization for deep feature representation at training stage and an efficient computation for heterogeneous data at testing stage. Extensive experiments on three challenging NIR-VIS face recognition databases demonstrate the significant superiority of Wasserstein CNN over state-of-the-art methods.
1 INTRODUCTION
NIR-VIS heterogeneous face recognition must bridge substantial sensing and data limitations: modality gaps, scarce paired training data, and overfitting risks. WCNN addresses these issues with invariant feature learning, Wasserstein distribution matching, and correlation regularization, achieving strong CASIA NIR-VIS 2.0 verification performance.
- NIR-VIS matching is needed because VIS galleries are widespread while probe images often arrive in NIR, creating a cross-modality recognition problem.
- Deep NIR-VIS recognition remains difficult because VIS and NIR sensing patterns produce different feature representations and paired cross-modality training data are scarce.Most existing HFR databases contain fewer than 10,000 samples despite feature dimensions of at least 100×100 pixels.
- WCNN maps NIR and VIS images into a compact Euclidean space using orthogonal identity and spectrum subspaces, with Wasserstein distance reducing distribution differences.The architecture combines invariant feature extraction and subspace learning in one end-to-end network.
- A correlation prior on fully connected layers alleviates overfitting on small-scale NIR-VIS datasets and improves verification performance.
- 98.4% verification rate (@FAR=0.1%) improves on the previous best 91.0% on CASIA NIR-VIS 2.0, while error rate (1-VR) is reduced by 82% with a 128-D feature.
2 RELATED WORK
Prior NIR-VIS HFR work includes image synthesis, handcrafted or learned feature representation, subspace learning, and deep learning. WCNN extends invariant deep representation with Wasserstein distance and correlation constraints, reporting improved recognition over earlier work.
- Image synthesis transforms faces between modalities so heterogeneous images can be compared in a common distance space, but synthesis is ill-posed and only partly reduces modality differences.
- Feature representation methods seek modality-invariant features using handcrafted descriptors such as LBP, HOG, DoG, and SIFT, alongside sparse representation.
- Subspace learning projects homogeneous data into a common space that minimizes inter-modality differences, with CCA and PLS as representative methods.
- Deep learning methods typically pretrain CNNs on large VIS datasets and fine-tune them on NIR faces to learn modality-invariant representations.A cited method achieved 91.03% verification at FAR=0.1% and 95.74% rank-1 accuracy on CASIA NIR-VIS 2.0.
- Compared with the authors’ earlier invariant deep representation work, WCNN introduces Wasserstein distance and correlation constraints and reduces error rate by 62% at FAR=0.1%.
3 THE PROPOSED WASSERSTEIN CNN
Wasserstein CNN learns modality-invariant NIR-VIS representations by combining orthogonal subspace learning, distribution matching, and correlation regularization in one end-to-end network.
- Modality-Invariant Feature Learning: WCNN uses a low-dimensional modality-invariant subspace and Wasserstein distance to reduce the discrepancy between NIR and VIS feature distributions.The shared representation is learned within a single CNN architecture for heterogeneous face matching.
- Network Structure: NIR and VIS channels share convolutional parameters, allowing both modalities to be mapped into a common feature-extraction process.The network assumes heterogeneous images share common low-level features and uses ΘN = ΘV = Θ.
- Optimization and Testing: Training alternates CNN back-propagation with updates to the mapping matrices, while testing extracts NIR and VIS features from the shared layer and compares them using cosine distance.The alternating procedure updates Θ, W, Pi, and Fi, then returns Θ and W.
- Modality-Invariant Feature Learning: The feature layer separates modality-invariant identity information from modality-variant light-spectrum information using orthogonal subspaces.WXi and PiXi represent shared and unique features, while the orthogonal constraint makes the subspaces unrelated.
- Correlation Prior: A low-rank correlation prior on the fully connected layers reduces the estimated parameter space and is intended to alleviate over-fitting on small-scale HFR data.The prior encourages correlated matrices through a nuclear-norm constraint, while the fully connected matrices are not used during testing.
4 EXPERIMENTS AND RESULTS
Experiments on three NIR-VIS face databases compare WCNN with traditional and deep learning methods under standard evaluation protocols. WCNN achieves the strongest overall recognition performance, with especially clear gains at low false acceptance rates and further improvement from its low-rank constraint.
- Datasets and protocols: Experiments evaluate WCNN against traditional and deep learning methods on CASIA NIR-VIS 2.0, Oulu-CASIA NIR-VIS, and BUAA-VisNir databases.CASIA is the largest and most challenging database, while BUAA images are captured simultaneously and are well aligned.
- CASIA NIR-VIS 2.0: VIS-only deep models perform poorly for NIR-VIS recognition, while specialized deep methods begin to outperform traditional methods.The sensing gap limits the benefit of large-scale VIS training when paired NIR data are scarce.
- CASIA NIR-VIS 2.0: 98.7% rank-1 accuracy and 98.4% verification rate at FAR=0.1% improve over IDR's 97.3% and 95.7%, respectively.WCNN+low-rank further reduces the error rate by 62% at FAR=0.1%.
- Oulu-CASIA NIR-VIS: WCNN methods achieve the highest rank-1 accuracy and verification rates on the Oulu-CASIA database.The methods are nearly ordered as MPL3, KPS, KCSR, KDSR, H2(LBP3), TRIVET, IDR, WCNN, and WCNN+low-rank.
- BUAA-VisNir: The low-rank constraint significantly improves WCNN ROC curves on the small-scale BUAA database, especially when FAR is low.The corresponding comparison indicates that constraining the fully connected layer can alleviate over-fitting.
5 CONCLUSION
The paper concludes that Wasserstein CNN unifies invariant feature extraction and subspace learning in one network for NIR-VIS face matching. It combines distribution alignment with a low-rank constraint and reports superior performance on three challenging databases.
- WCNN maps NIR and VIS images into a compact Euclidean space using one network.
- Two orthogonal subspaces separate modality-invariant identity information from modality-variant light spectrum information.
- Wasserstein distance measures heterogeneous feature-distribution differences and reduces the sensing gap.
- A low-rank constraint addresses over-fitting on small-scale NIR-VIS face data.
- Experiments on three challenging NIR-VIS databases show significant outperformance over state-of-the-art methods.