Source-linked AI summary
Beyond Frontal Faces: Improving Person Recognition Using Multiple Cues
Ning Zhang, Manohar Paluri, Yaniv Taigman, Rob Fergus, Lubomir Bourdev
TL;DR
Person recognition in photo albums remains difficult because many images lack suitable frontal faces and existing large-scale datasets do not capture unconstrained appearances. The paper introduces the PIPA dataset and PIPER, which combines poselet-based part recognizers with global and face cues. PIPER significantly outperforms a strong CNN-plus-frontal-face baseline, while the dataset and method support single-example learning and image retrieval.
Problem
Only about half of person images contain high-resolution frontal faces, while large-scale unconstrained recognition data and ways to exploit non-frontal cues are lacking.
Method
PIPER combines poselet-based part-level identity classifiers with face and global recognition components to account for pose variation.
Results
PIPER significantly outperforms a strong baseline combining a fine-tuned CNN with a state-of-the-art frontal face recognizer.
Takeaways & Limitations
PIPA provides a large-scale benchmark for person coidentification in photo albums, and PIPER also performs effectively with a single training example and on image retrieval.
Takeaways & Limitations
The problem setting does not assume frontal faces or straight pedestrian poses, and the ablation removes the global model by setting its prior to a uniform distribution.
Abstract
from arXiv · showhide
We explore the task of recognizing peoples' identities in photo albums in an unconstrained setting. To facilitate this, we introduce the new People In Photo Albums (PIPA) dataset, consisting of over 60000 instances of 2000 individuals collected from public Flickr photo albums. With only about half of the person images containing a frontal face, the recognition task is very challenging due to the large variations in pose, clothing, camera viewpoint, image resolution and illumination. We propose the Pose Invariant PErson Recognition (PIPER) method, which accumulates the cues of poselet-level person recognizers trained by deep convolutional networks to discount for the pose variations, combined with a face recognizer and a global recognizer. Experiments on three different settings confirm that in our unconstrained setup PIPER significantly improves on the performance of DeepFace, which is one of the best face recognizers as measured on the LFW dataset.
1. Introduction
Recognizing people beyond frontal faces requires combining subtle cues across pose, clothing, body parts, and context. The paper addresses the lack of suitable unconstrained data and methods with PIPER, which combines poselet-level recognizers to handle pose variation.
- Only 52% of people in the dataset have high-resolution frontal faces suitable for recognition.
- Unconstrained recognition is difficult because pose, viewpoint, clothing, resolution, and visibility vary substantially.
- The paper identifies two gaps: no high-quality large-scale unconstrained recognition dataset and no clear way to leverage cues beyond frontal faces.
- PIPER uses part-level person recognizers based on poselets to account for pose variations.Poselets detect recurring pose patterns, including body-part configurations and non-frontal views.
- Combining predictions from multiple poselets accumulates weak signals from different body parts into pose-independent recognition.
- The method achieves an approximately 40% decrease in relative error.
2. Related Work
Related work spans unconstrained face recognition, contextual photo-album identification, social-relation modeling, character naming, person re-identification, and deep convolutional networks. These lines of work address complementary aspects of recognition across appearance, context, identity matching, and learned features.
- Conventional face recognition systems commonly assume frontal faces and explicit 3D alignment or facial keypoint localization.
- Photo-album identification methods incorporate contextual cues such as clothing appearance and photo metadata.
- Related work models social relationships, people’s positions, poses, and identities in photo collections.
- Character-naming systems combine face recognition with clothing features and use aligned subtitles, scripts, or semi-supervised learning.
- Person re-identification research focuses mainly on metric learning, mid-level feature learning, and deep similarity models across cameras.
- Deep convolutional networks learn discriminative features directly from raw image input and have advanced image classification, detection, and face recognition.
3. People In Photo Albums Dataset
PIPA is designed for person recognition in unconstrained photo albums, where identities may appear in varied poses, resolutions, and occlusion conditions. The dataset is collected and annotated from Flickr albums, merged across albums, normalized by identity frequency, and split by uploader.
- PIPA addresses the absence of an existing large-scale dataset for person recognition; the closest cited alternative has only 931 instances of 32 identities.
- The task identifies people in the wild without assuming frontal faces, straight pedestrian poses, detection, or visible heads.
- The dataset was collected from public Flickr photo albums uploaded by 111 users, with shared identity labels across photos.
- Collection Method: Collection begins by filtering downloaded albums to retain people albums and remove landscapes, flowers, and albums with little person co-occurrence.
- Collection Method: Annotators tag individuals appearing at least twice, assign identity colors, and mark head boxes even when heads are partially or fully occluded.
- Collection Method: Identities are merged across albums from the same uploader when the same people recur.
- Collection Method: Individuals with fewer than 10 instances are discarded, while each remaining identity is capped at 99 instances to reduce frequency bias.
- Collection Method: Training, validation, and test sets are separated by uploader so their photos, identities, and instances are disjoint.
4. Pose Invariant Person Recognition (PIPER)
PIPER combines a global body classifier, poselet-specific classifiers, and a face-based classifier, weighting their identity predictions while correcting poselet sparsity. The method trains poselet models from matched detections and uses validation predictions to learn the mixing weights.
- Model components: PIPER combines a global full-body CNN, 107 poselet CNNs, and a DeepFace-feature SVM into 109 part classifiers.The global classifier uses the full body, while each poselet classifier targets a specific pose pattern.
- Model combination: The final identity prediction is a weighted linear combination of normalized probabilities from the component classifiers.The predicted identity is the label with the highest combined score.
- Training pipeline: The training pipeline fine-tunes CNNs on ImageNet-pretrained networks using poselet patches and full-body patches, then trains identity SVMs on FC7 features.The FC8 layer is discarded and FC7 activations are used as generic features for subsequent SVM training.
- Part activations: Poselet detections are matched to ground-truth people with globally optimal bipartite matching before extracting patches for part-based classifiers.Matching prefers detections with higher scores and greater overlap with ground-truth bounds.
- Sparsity correction: Poselet predictions are sparse because poselets activate only for matching poses and each poselet classifier covers only a subset of identities.The sparsity pattern is correlated with pose rather than identity, so it must be corrected for accurate recognition.
- Sparsity correction: PIPER fills missing poselet predictions with the global model and learns mixing weights from cross-validated part predictions on the validation set.Validation data are split into two halves for reciprocal prediction generation, followed by linear SVM training of the weights and retraining on the full validation set.
5. Experiments
Experiments evaluate PIPER on supervised recognition, one-shot identification, and unsupervised retrieval using held-out test data. Across these settings, combining global, face, and poselet-based cues improves recognition, including when frontal faces are absent and when training data is scarce.
- Experimental settings: PIPER is evaluated in three settings: person recognition, one-shot person identification, and unsupervised identity retrieval.Networks are trained on the training split, hyperparameters are tuned on validation data, and results are evaluated on the test split.
- Overall performance: Sparsity filling is essential: using sparse part predictions instead of filled predictions creates a performance gap of more than 7%.The global model supplies the sparsity-filling mechanism used in the combined prediction.
- Overall performance: More than 3% gain over the fine-tuned CNN combined with DeepFace is reported when PIPER uses its full component set.The compared components include a fine-tuned global CNN, DeepFace, and 108 poselet-based recognizers.
- Ablation study: The global model, DeepFace, and poselet recognizers are complementary, and combining all three achieves the best performance.The ablation study retrains mixture weights and retunes hyperparameters for each component combination.
- Face and non-face instances: PIPER improves recognition both without frontal faces and when frontal faces are present by combining contextual cues and multiple classifiers.The test set is divided into subsets where frontal faces are visible in 52% of instances and absent in 48%.
- Unsupervised identity retrieval: 64% of test examples have a same-identity nearest neighbor, compared with 50% for the fine-tuned Krizhevsky CNN baseline.The retrieval representation is a 366-dimensional feature vector combined from part predictions and evaluated with Euclidean nearest neighbors.
6. Conclusion
The paper presents PIPER as a pose- and viewpoint-independent approach to person recognition and introduces the large-scale PIPA dataset. PIPER also performs well with one training example and in image retrieval, with possible applications to generic instance coidentification.
- Contributions: PIPER is designed for viewpoint- and pose-independent person recognition and significantly outperforms a strong CNN-plus-frontal-face baseline.The baseline combines a state-of-the-art CNN fine-tuned on the dataset with a state-of-the-art frontal face recognizer.
- Contributions: PIPER learns effectively from a single training example and performs well at image retrieval.The conclusion identifies both capabilities as outcomes of the method.
- Broader applicability: The PIPER algorithm is presented as applicable to generic instance coidentification, including finding instances of the same car or dog.This extends the stated application beyond person recognition in photo albums.
- Dataset contribution: The paper introduces PIPA as a large-scale dataset for person coidentification in photo albums.The authors position the dataset as the first large-scale dataset of its kind.