Source-linked AI summary
PANDA: Pose Aligned Networks for Deep Attribute Modeling
Ning Zhang, Manohar Paluri, Marc'Aurelio Ranzato, Trevor Darrell, Lubomir Bourdev
TL;DR
Human attribute classification is challenging because subtle cues are affected by pose, viewpoint, appearance, and occlusion, while available labeled data may be insufficient for learning pose normalization directly. PANDA combines poselet-based part alignment with CNNs to learn pose-normalized deep representations, and the paper reports improved or state-of-the-art performance across challenging datasets, including 99.54% AP on LFW. The method has dataset and coverage constraints, including insufficient labels for some attributes and poselet parts that may not cover the whole image.
Problem
Human attribute cues can be subtle and localized, while pose, viewpoint, appearance, and occlusion complicate recognition and available labeled data may be insufficient for direct pose normalization.
Method
PANDA combines poselet-based semantic part patches with convolutional neural networks to construct pose-normalized deep representations for attribute classification.
Results
PANDA achieves state-of-the-art results on the Berkeley Attributes of People dataset and LFW, including 99.54% AP on LFW.
Takeaways & Limitations
Poselets allow CNNs to focus on pose-normalized appearance differences without requiring datasets of millions of images.
Takeaways & Limitations
Some attributes lack enough labeled data for CNN training, and poselet-detected parts may not cover the whole image region.
Abstract
from arXiv · showhide
We propose a method for inferring human attributes (such as gender, hair style, clothes style, expression, action) from images of people under large variation of viewpoint, pose, appearance, articulation and occlusion. Convolutional Neural Nets (CNN) have been shown to perform very well on large scale object recognition problems. In the context of attribute classification, however, the signal is often subtle and it may cover only a small part of the image, while the image is dominated by the effects of pose and viewpoint. Discounting for pose variation would require training on very large labeled datasets which are not presently available. Part-based models, such as poselets and DPM have been shown to perform well for this problem but they are limited by shallow low-level features. We propose a new method which combines part-based models and deep learning by training pose-normalized CNNs. We show substantial improvement vs. state-of-the-art methods on challenging attribute classification tasks in unconstrained settings. Experiments confirm that our method outperforms both the best part-based methods on this problem and conventional CNNs trained on the full bounding box of the person.
1. Introduction
Human attribute recognition is difficult because subtle attribute signals are obscured by pose, viewpoint, occlusion, and image quality. PANDA combines semantically aligned part modeling with CNNs to learn pose-normalized representations and reports state-of-the-art results across four datasets.
- Subtle attribute cues, such as glasses, may occupy little of a person image and vary with head pose, frame design, and hair occlusion.
- CNNs can underperform pose- or part-normalized methods because available training data may be insufficient for learning pose normalization directly.
- PANDA combines part-based representations with CNNs by decomposing images into pose-specific parts, simplifying learning on relatively small datasets.
- The final attributes are predicted by a linear SVM using concatenated top-level activations from CNNs trained for individual poselets.
- PANDA augments CNNs with semantically aligned part-patch inputs and combines pose-specific features into a pose-normalized deep representation.
- The method achieves state-of-the-art experimental results on four datasets spanning web attributes, Berkeley Attributes, LFW, and large-scale gender recognition.
2. Related work
Prior work includes attribute transfer, discovery, face-focused recognition, and pose-aware part models. This paper instead integrates poselets into deep learning to obtain pose-normalized representations and evaluates them against conventional and whole-person CNN baselines.
- Earlier attribute research supported knowledge transfer, relative-attribute ranking, textual descriptions, and automatic discovery from unlabeled text and images.
- Face-attribute methods addressed verification and search but mainly used frontal, high-quality face images, unlike the more varied settings evaluated here.
- Bourdev et al.’s closely related system predicts each attribute value for each poselet before combining scores through additional classifier layers.
- CNNs became highly accurate for generic recognition and pedestrian detection with abundant labeled data, but typically generalize poorly on smaller datasets.
- This work incorporates poselets into deep learning and compares conventional baselines with CNNs evaluated on the whole person region.
3. Pose Aligned Networks for Deep Attribute modeling (PANDA)
PANDA trains CNNs on poselet-localized patches, extracts attribute-specific part features, and combines them into a pose-normalized representation for linear SVM classification. The design uses pose decomposition to make deep learning feasible with fewer effective training instances, but detected parts may not cover the whole image.
- PANDA uses poselets to localize semantically corresponding parts and incorporates the normalized patches into deep CNNs for attribute classification.
- Each poselet activation is treated as a separate training example, reducing the number of distinct images required for training.
- Poselet patches are resized to 64x64 pixels, augmented by jittering and horizontal flips, and used to train a CNN for each poselet.
- Training uses joint backpropagation and stochastic gradient descent with summed attribute log-losses, gradually adding lower-confidence patches after high-score patches.
- Part representations are combined because individual body parts may carry insufficient information for some attributes, while poselet detections may not cover the whole image.
- The CNN branches into attribute-specific fully connected layers so different attributes can develop customized features while sharing lower layers.
4. Datasets
The evaluation uses challenging person-attribute datasets with substantial pose, viewpoint, and occlusion variation, supplemented by additional Facebook data for training. Labels can be uncertain or unavailable when relevant body parts are not visible.
- The Berkeley Human Attributes Dataset contains 4,013 training and 4,022 test images collected from PASCAL and H3D, with wide pose, viewpoint, and occlusion variation.
- The Berkeley training split was too small for deep-net training, so the authors collected 24,963 additional Facebook examples divided into training, validation, and test sets.
- The added dataset was selected to avoid overlap with Berkeley images and to preserve similar variation in viewpoint, pose, and occlusion.
- The experiments use selected Berkeley categories plus additional categories to examine transfer-learning ability.
- Figure 4 separates each attribute’s ground-truth labels into positive, negative, and uncertain counts.
- Some attributes cannot be labeled when the relevant body part is invisible, such as labeling whether someone wears a hat without seeing the head.
5. Results
PANDA is evaluated against prior methods, alternative components, and viewpoint conditions across attribute datasets. Results show broad gains from pose-normalized deep representations, while data availability and ambiguous visual cases remain important boundaries.
- 5.1. Results on the Berkeley Attributes of People Dataset: On the Berkeley Attributes of People dataset, PANDA outperforms prior methods across most attributes, except t-shirt, where performs better.The authors associate the t-shirt exception with the absence of a corresponding Attribute25K training attribute.
- 5.1. Results on the Berkeley Attributes of People Dataset: Attributes25K lacks shorts, jeans, and long pants labels, and its training data is insufficient to train the convolutional nets directly.The authors report that transfer learning remains effective despite this limitation.
- 5.2. Results on the Attributes25K Dataset: DL-DPM outperforms conventional part-based methods and DeCAF but does not match PANDA, whose poselet patches are less noisy and better aligned.DPM patches must activate under weak or nonexistent patterns and satisfy global location constraints.
- 5.3. Component Evaluation: PANDA combines DeCAF and DLPoselets, whose complementary global and part-level features further boost performance.The component evaluation compares deep learning and part-based representations within a shared person-level classification setup.
- 5.4. Robustness to viewpoint variations: PANDA works across a wide range of viewpoints, although performance is best for front-facing people, who comprise approximately 60% of the training distribution.The evaluation distinguishes frontal, profile, and back-facing examples by eye visibility.
- 5.5. Results on the LFW Dataset: PANDA achieves 99.54% AP for gender recognition on LFW, while manual inspection found roughly 1 in 200 examples with incorrect labels or person-detection matches.The dataset contains tightly cropped frontal faces, so this evaluation tests performance where viewpoint and pose variation are reduced.
6. Conclusion
PANDA improves human attribute classification by combining poselets with deep convolutional features, without requiring datasets of millions of images. Its pose-normalized representation achieves state-of-the-art results on the Berkeley Attributes of People dataset and LFW.
- PANDA improves performance over previously published methods for human attribute classification.
- The method uses poselets to factor out pose and viewpoint variation before learning appearance differences with convolutional networks.
- PANDA concatenates deep features from each poselet with a deep representation of the whole input image.
- The approach leverages convolutional neural networks without requiring datasets of millions of images.
- PANDA achieves state-of-the-art results on the Berkeley Attributes of People dataset and LFW, despite training its CNNs on a different dataset.