Shared Representation Learning for Heterogeneous Face Recognition

• The paper focuses on Sketch-Photo and NIR-VIS recognition, motivated by law-enforcement matching and face recognition in dark environments.Source: §1 · p. 1
• Cross-modal face relationships are highly nonlinear because modality differences interact with pose, identity, and other variations, while limited samples constrain mapping-based methods.Source: §1 · p. 2
• The method extracts local Gabor features, learns shared representations with RBMs, then applies PCA and cosine similarity for matching.Source: §1 · p. 2
• The framework reports perfect results on CUFS and state-of-the-art performance on CASIA HFB and NIR-VIS 2.0.Source: §1 · p. 2

• Synthesis-based methods transformed sketches or VIS images toward another modality, but recognition from synthesized images remained moderate despite good visual quality,,.Source: §2 · pp. 2–3
• Classification-oriented methods reduce modality differences with image-processing or coding features, then map data into discriminative spaces using linear, nonlinear, or tree-based methods [7][18].Source: §2 · p. 3
• Multimodal deep-learning work includes denoising autoencoders, RBM-CCA, and DBMs, with DBMs distributing multimodal modeling across layers,,.Source: §2 · p. 3
• The paper plugs a multimodal RBM into a traditional face-recognition pipeline for heterogeneous face recognition.Source: §2 · p. 3

3 Background

• The background frames RBMs as generative models for learning data distributions, including real-valued face-image data.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1 Restricted Boltzmann Machines

• RBMs model training-data distributions with stochastic visible and hidden units connected through an energy function.For real-valued image data, the paper uses a Gaussian RBM; normalized inputs allow a simplified energy function.Source: §3.1 · pp. 3–4

3.2 Multi-modal RBM

• The multimodal model uses two Gaussian RBMs to model face data from two modalities jointly.Its hidden layer represents shared properties, and the paper infers a common-space representation from either modality.Source: §3.2 · pp. 4–5, Figure 1 · p. 5

4.1 Framework

• Gabor features at facial points feed local RBMs, whose shared representations are concatenated, processed by PCA, and compared using cosine similarity.The framework also uses a sum rule to fuse similarities from the two face halves.Source: §4.1 · p. 5, Figure 2 · p. 6, §4.4 · p. 8

4.2 Level 1 Representations

• Facial points are warped onto each input face using RBF warping, while landmarks provide the alignment information.The figure distinguishes standard landmarks and facial points from their warped positions on the input image.Source: §4.2 · p. 6, Figure 3 · p. 7
• At 176 facial points, each point receives 40 Gabor features from five resolutions and eight orientations, grouped into left- and right-half vectors.The features support robust, discriminative representations while the facial points address small pose variations.Source: §4.2 · pp. 6–7

4.3 Level 2 Representations

• The method uses 176 local 40-80-40 RBMs to learn shared representations, modeling each facial point’s cross-modal relationship separately.The paper motivates local RBMs by the greater learnability of local relationships and the location-dependent variation in cross-modal relationships.Source: §4.3 · p. 7

4.4 Cross-modal Matching

• PCA removes leading components from the concatenated 80 × 176 representations to further address holistic heterogeneity before cosine matching.The number of removed components is tuned on training or development data, and similarities from the two face halves are fused by sum rule.Source: §4.1 · p. 6, §4.4 · p. 8
• LDA and metric learning on shared representations performed worse than PCA in the experiments.The authors suggest limited data may explain this result and expect supervised methods may outperform PCA with larger databases.Source: §4.4 · p. 8

5 Experiments

• Across CUFS, CASIA HFB, and CASIA NIR-VIS 2.0, the proposed method outperforms current state-of-the-art methods.The authors report these results for both Sketch-Photo and NIR-VIS recognition.Source: §5 · p. 8

5.2 NIR to VIS

• CUFS uses 306 training and 300 testing subjects across 10 splits; HFB uses 102 training and 100 testing subjects, with View 1 for tuning and View 2 for reporting.For both NIR–VIS experiments, VIS images are gallery and NIR images are probe; NIR-VIS 2.0 uses its standard evaluation protocol.Source: §5.1 · p. 8, §5.2 · p. 9

5.1 Sketch to Photo

• 100% recognition rate is achieved on CUFS without RBMs, after removing the first 20 principal components.Using Gabor features alone yields Rank1=99.50%, comparable to state-of-the-art methods; CUFS is described as relatively easy.Source: §5.1 · p. 8, Table 1 · p. 9

CASIA HFB

• Without using the whole gallery, the method reports higher Rank1 and VR than and, but NN+SR+Cognitec and P-RS are reported to outperform it in some comparisons.Z-score normalization raises VR from 92.25% to 96.33%; P-RS uses a larger training set and is described as slower.Source: CASIA HFB · pp. 9–10
• VR@FAR=0.1% on CASIA HFB rises from 71.70% to 92.25% after introducing RBMs, and the optimal number of removed principal components falls from 20 to 11.With RBMs, performance curves become smoother and reach their optimum more quickly as components are removed.Source: CASIA HFB · pp. 9, 11, Table 2 · p. 10, Figure 4 · p. 11
• Local RBMs provide the best trade-off between complexity and generalization among the global, convolutional, and local architectures.The global RBM performs well on training data, while the convolutional RBM performs moderately on both training and testing data; further tuning of locality and weight sharing is left for future work.Source: CASIA HFB · p. 10, Table 3 · p. 11
• The four reported HFB failure cases involve NIR probes with visible pose, eyeglass-reflectance, or expression variation.The authors note that these factors are harder to solve when coupled with spectrum variation.Source: CASIA HFB · p. 11, Figure 5 · p. 12

CASIA NIR-VIS 2.0

• On CASIA NIR-VIS 2.0, Rank1 and VR drop 10–20% versus HFB, while removing the first PCs and adding RBM bring improvements of about 40% and 10%, respectively.The experiment tunes parameters on View 1 and reports results on View 2; the database is described as more challenging and practical than the preceding databases.Source: §5.2 · p. 9, CASIA NIR-VIS 2.0 · p. 12, Table 4 · p. 12

6 Conclusion

• Local RBMs learn modality-free shared representations from Gabor features at a low level, which PCA processes at a higher level.The framework combines RBMs with modules from traditional face recognition.Source: §6 · p. 12
• The framework performed perfectly on CUFS and significantly outperformed state-of-the-art methods on CASIA HFB and NIR-VIS 2.0.The authors also report that experimental results illustrated the success of local RBMs in learning shared representations.Source: §6 · p. 13
• Future work will explore stacked multi-modal RBMs for high-level representations and ways to fine-tune the model with identity information.Source: §6 · p. 13