Source-linked AI summary
Face Recognition Using Deep Multi-Pose Representations
Wael AbdAlmageed, Yue Wua, Stephen Rawlsa, Shai Harel, Tal Hassner, Iacopo Masi, Jongmoo Choi, Jatuporn Toy Leksut, Jungyeon Kim, Prem Natarajan, Ram Nevatia, Gerard Medioni
TL;DR
Face recognition remains challenging under unconstrained pose and other variations. The paper addresses this with a multi-pose representation using 3D-rendered poses and pose-specific CNN features, and reports improved performance over state-of-the-art methods on IJB-A and CS2 tasks.
Problem
Face recognition remains difficult in uncontrolled conditions with extreme pose, illumination, expression, and age variation.
Method
The method generates aligned and rendered face poses, extracts pose-specific CNN features, and combines them into a multi-pose representation.
Results
The multi-pose representation significantly improves IJB-A recognition and outperforms state-of-the-art methods without target-domain tuning or metric learning.
Takeaways & Limitations
Increasing the number of pose representations significantly improves recognition performance across the evaluated CNN architectures.
Abstract
from arXiv · showhide
We introduce our method and system for face recognition using multiple pose-aware deep learning models. In our representation, a face image is processed by several pose-specific deep convolutional neural network (CNN) models to generate multiple pose-specific features. 3D rendering is used to generate multiple face poses from the input image. Sensitivity of the recognition system to pose variations is reduced since we use an ensemble of pose-specific CNN features. The paper presents extensive experimental results on the effect of landmark detection, CNN layer selection and pose model selection on the performance of the recognition pipeline. Our novel representation achieves better results than the state-of-the-art on IARPA's CS2 and NIST's IJB-A in both verification and identification (i.e. search) tasks.
1. Introduction
The paper addresses difficult face recognition under extreme pose and other uncontrolled variations by introducing a multi-pose representation that explicitly uses pose-specific CNNs. It reports that this approach surpasses prior methods on IJB-A without domain adaptation or metric learning.
- Face recognition includes verification, which asks whether two people are the same, and identification, which searches a database for a person.
- Despite near-100% rates on LFW, recognition remains unsolved in uncontrolled environments with extreme pose, illumination, expression, and age variation.
- Prior methods considered pose variations implicitly when representing faces or learning similarity measures.
- The proposed pipeline represents each face with aligned and rendered poses using different pose-specific CNNs, then compares only matching poses.
- The multi-pose approach surpasses state-of-the-art algorithms on IJB-A without additional domain adaptation or metric learning.
2. Facial Datasets Employed
The study trains pose-specific models on CASIA-WebFace and evaluates them on the challenging Janus datasets. IJB-A and CS2 contain substantial pose variation but use different identification and verification protocols.
- CASIA-WebFace is used for training, while IJB-A and IARPA Janus CS2 are used for evaluation.
- After cleanup, CASIA-WebFace provides approximately 400,000 images from 10,500 subjects for training.
- Janus datasets contain full-pose variation, manual annotations, and templates that collect images and videos of the same subject.
- IJB-A supports open-set identification and verification, whereas CS2 focuses on closed-set identification.
- Figure 2 compares face-pose distributions in the LFW and JANUS datasets.
3.1. From Images to Representations
The representation pipeline converts an input face image into a fixed-dimensional feature through landmark detection, pose correction, alignment, and feature extraction. It combines geometric preprocessing with learned or handcrafted feature representations.
- The pipeline transforms a 2D color face image into a fixed-dimensional feature vector through a representation function rep(X).
- Facial Landmark Detection: Landmark detection estimates predefined facial key points, whose coordinates are represented along the x and y axes.
- Facial Landmark Detection: Detected landmarks estimate roll, guide pose correction by image rotation, and are detected again after correction to address errors.
- Face Alignment: Face alignment maps images into a common coordinate system by minimizing distances between detected landmarks and reference landmarks.
- Face Alignment: The pipeline uses non-reflective similarity transformation for in-plane alignment and perspective transformation for out-of-plane alignment.
- Face Alignment: A 3D generic face model enables rendering images with arbitrary yaw, pitch, and roll after fitting the model to the input image.
- Feature Extraction: Feature extraction produces representations intended to discriminate between different subjects.
- Feature Extraction: The pipeline considers both handcrafted features such as LBP and learned DNN features extracted from aligned images.
3.2. Face Representation in Practice
The practical representation pipeline emphasizes alignment and feature extraction, using pose-aware preprocessing and CNNs fine-tuned on pose-specific training partitions. The system compares alternative alignment, architecture, and training configurations before using selected CNN layers as feature extractors.
- Face alignment: 3D alignment fits a generic face shape, renders arbitrary yaw-pitch-roll views, and aligns images to a common configuration.
- Alignment and feature extraction are identified as the two most important components of the face recognition pipeline.
- Representation pipelines: The study compares face representation pipelines across alignment references, CNN architectures, and preprocessing configurations.
- CNN training: CASIA-WebFace is preprocessed into approximately 400,000 samples from 10,500 subjects for CNN training and validation.
- Pose-wise fine tuning: Pose-specific CNNs are learned by fine-tuning baseline models on near-frontal, near-profile, or rendered-pose data.
- Pose-wise fine tuning: After fine tuning, classification layers are discarded and outputs from one or more CNN layers are concatenated as features.
3.3. Multi-Modal Representation for Recognition
The multi-modal representation combines several feature-extraction pipelines for each face and performs recognition by comparing corresponding representations, then fusing their similarity scores. For IJB-A templates, an additional fusion step aggregates scores across image pairs.
- Representation construction: A multi-pose representation contains k pose-specific representations produced by applying the conceptual pipeline with different feature extractors.
- Similarity fusion: Recognition first compares features within the same representation pipeline and then fuses scores across different pipelines.
- Similarity fusion: The method uses direct pairwise feature comparison rather than constructing new classifiers on the target dataset.
- Similarity fusion: Cosine similarity measures feature correspondence, while softmax weights fuse scores with bandwidth parameter β = 10.
- Template matching: For IJB-A templates, softmax fusion is applied once more across all pairwise scores between images in the two templates.
3.4. Time Complexity
The pipeline has substantial computational costs for CNN training and testing, with PCA adaptation identified as the most time-consuming testing step.
- Approximately one week is required to fully train or fine-tune a VGG19-AF-like CNN model on one NVIDIA Tesla K40 GPU.
- Roughly one hour is required to test a single IJB-A/CS2 data split with one CNN model using the proposed pipeline.
- PCA adaptation costs about 20 minutes per IJB-A/CS data split and is the most time-consuming testing step.
4. Experimental Evaluation
The evaluation examines feature-layer selection, landmark detection, feature type, and the number of pose representations across CS2 and IJB-A protocols. Multi-pose representations and deep features improve recognition, while the VGG19-quintuple pipeline outperforms compared state-of-the-art methods on IJB-A.
- Evaluation Protocols: IJB-A evaluates open- and closed-set search using TAR at various FARs, while compare measures verification accuracy between two templates.The experiments report averaged scores across ten data splits for CS2 and IJB-A.
- CNN Layer Selection: An optimal feature can combine layers: fc7 is best for VGG19-AF, whereas ALEX-AF uses (pool5, fc7, fc8, prob).VGG* representations use (fc8) in subsequent experiments.
- Feature Comparison: Deep learning features outperform HDBLP features by a large margin across the compared CS2 pipelines.The comparison includes ALEX-AF, VGG16-AF, and VGG19-AF, with grid-search time for HDBLP described as almost comparable.
- Landmark Detection: Facial landmarks substantially improve face recognition performance in the CS2 baseline evaluation.The evaluated detectors include DLIB, FPS3K, TDCNN, CLNF, and CLNFs.
- Pose Representations: Recognition performance significantly improves as pose representations increase from single to quadruple and quintuple, for both ALEX and VGG19 architectures.The evaluated tuples include frontal, profile, and synthesized yaw representations.
- Comparison with State of the Art: The VGG19-quintuple pipeline outperforms other state-of-the-art methods on IJB-A without target-domain-specific tuning.The method is described as data-agnostic and used out of the box on CS2 and IJB-A.
5. Conclusion
The paper introduces a multi-pose representation that collects face representations learned from specific poses. On IJB-A, it improves recognition over single best CNN representations and state-of-the-art methods relying on supervised learning.
- Conclusion: The proposed representation collects face representations learned from specific face poses.It is evaluated on the IJB-A benchmark against single CNN representations and supervised-learning-based methods.
- Conclusion: On IJB-A, the multi-pose representation significantly improves face recognition performance over single best CNN representations and methods using gallery fine-tuning or metric learning.