Source-linked AI summary
Deep Learning Face Attributes in the Wild
Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang
TL;DR
Predicting face attributes in the wild is difficult under pose, lighting, occlusion, and other complex variations. The paper proposes a two-CNN framework with different pre-training strategies, achieving state-of-the-art results on CelebFaces and LFW while revealing properties of learned face representations.
Problem
Predicting face attributes in the wild remains challenging because pose, lighting, occlusion, and other variations complicate face localization and alignment.
Method
A cascaded LNet–ANet framework uses weakly supervised face localization and identity-pretrained attribute recognition, jointly fine-tuned with image-level attribute tags.
Results
8% and 13% improvements over existing methods are reported on CelebFaces and LFW, respectively, with state-of-the-art attribute classification results.
Takeaways & Limitations
Different pre-training strategies improve face localization and attribute recognition, while learned representations capture face locations and semantic concepts useful for explaining attributes.
Takeaways & Limitations
The evaluation assumes that each image contains a dominant face whose attributes were labeled by users.
Abstract
from arXiv · showhide
Predicting face attributes in the wild is challenging due to complex face variations. We propose a novel deep learning framework for attribute prediction in the wild. It cascades two CNNs, LNet and ANet, which are fine-tuned jointly with attribute tags, but pre-trained differently. LNet is pre-trained by massive general object categories for face localization, while ANet is pre-trained by massive face identities for attribute prediction. This framework not only outperforms the state-of-the-art with a large margin, but also reveals valuable facts on learning face representation. (1) It shows how the performances of face localization (LNet) and attribute prediction (ANet) can be improved by different pre-training strategies. (2) It reveals that although the filters of LNet are fine-tuned only with image-level attribute tags, their response maps over entire images have strong indication of face locations. This fact enables training LNet for face localization with only image-level annotations, but without face bounding boxes or landmarks, which are required by all attribute recognition works. (3) It also demonstrates that the high-level hidden neurons of ANet automatically discover semantic concepts after pre-training with massive face identities, and such concepts are significantly enriched after fine-tuning with attribute tags. Each attribute can be well explained with a sparse linear combination of these concepts.
1. Introduction
The paper proposes a two-CNN framework for face attribute prediction in the wild that combines weakly supervised face localization with identity-pretrained face representation learning. It also introduces efficient multi-patch inference and reports improved attribute classification on CelebFaces and LFW.
- Framework: The framework cascades LNet for whole-face localization and ANet for high-level face representation and attribute prediction.LNet and ANet are jointly fine-tuned with attribute tags but pre-trained differently.
- Weakly supervised localization: LNet uses only image-level attribute tags for training, avoiding the face bounding boxes and landmark positions required by conventional detectors.It is pre-trained with massive general object categories for face localization.
- Face representation learning: ANet is pre-trained on massive face identities and then fine-tuned with attributes, enabling attribute recognition from the entire face region despite unconstrained variations.The paper reports that attribute-prediction performance drops without this pre-training stage.
- Efficient inference: A fast feed-forward scheme replaces redundant patch-by-patch evaluation, supporting arbitrary image sizes with a single feed-forward pass while aggregating multiple patches.The scheme operates within rough face regions localized by LNet.
- Results: 8 and 13 percent improvements over existing methods are reported on the CelebFaces and LFW datasets, respectively.The paper describes these as state-of-the-art attribute classification results on both challenging datasets.
2. Our Approach
The approach cascades LNet for coarse-to-fine face localization with ANet for attribute recognition. LNet uses weak image-level supervision and object pre-training, while ANet uses face-identity pre-training, multi-view patches, locally shared filters, and SVM aggregation.
- Framework Overview: LNet localizes the entire face region coarsely to finely, providing the detected face input that ANet uses for attribute recognition.The cascade includes LNeto and LNets, followed by ANet feature extraction and attribute prediction.
- LNet Training: LNet is weakly supervised using image-level attribute annotations rather than face or landmark bounding boxes.It is pre-trained on 1,000 ImageNet object categories to handle background clutter, then fine-tuned with attribute tags for complex face variations.
- Attribute Recognition: ANet extracts overlapping-patch feature vectors, applies SVMs to predict attribute values, and averages the predictions to reduce sensitivity to face misalignment.An interweaved feed-forward operation reduces redundant computation for CNN layers with locally shared filters.
- LNet Training: Fine-tuning with attribute tags produces cleaner, smoother LNet response maps, whereas direct training from scratch without pre-training cannot achieve the same localization behavior.C5 responses are used to distinguish face from background through thresholding and candidate-window scoring.
- Face-Region Refinement: When a window contains multiple faces, the method selects the face region with the highest response density using density-peak identification.This pruning step is accelerated because the averaged C5 response map is sparse.
- ANet Training: ANet is pre-trained on massive face identities using combined softmax and similarity losses, then fine-tuned on enlarged localized face regions with multiple patches.Its convolutional filters are globally shared in C1-C2 and locally shared in C3-C4.
3. Experiments
Experiments evaluate the framework on large-scale CelebA and LFWA datasets, showing strong face localization and attribute prediction performance. Analyses further demonstrate the effects of pre-training, semantic concept discovery, attribute grouping, generalization, and computational efficiency.
- Large-scale Data Collection: CelebA contains 200,000 images from 10,000 identities, while LFWA contains 13,233 images from 5,749 identities, with each image annotated for 40 attributes.The datasets contain over eight million and 500,000 attribute labels, respectively.
- LNet Performance Comparison: At FPPI = 0.01, LNet and Face++ achieve true positive rates of 93% and 85%, respectively; at FPPI = 0.1, LNet exceeds three competing detectors by 11, 9, and 22 percent.The compared detectors are DPM, ACF Multi-view, SURF Cascade, and Face++.
- LNet Performance Comparison: Pre-training improves LNet localization by 74 percent at overlap ratio 0.5, while richer attribute information facilitates localization and LNet generalizes to the 3,876-image MobileFaces test set.More accurate localization contributes to subsequent attribute prediction performance.
- Fine-tuning Expands Semantic Concepts: Only 10% of ANet’s best-performing neurons achieve 90% of an attribute’s original performance, whereas HOG+PCA requires more than 95% of its features.Fine-tuning expands identity-derived concepts into diverse attribute concepts represented by sparse activations.
- Performance Comparison: On CelebA, FaceTracer, PANDA-w, PANDA-l, and LNets+ANet achieve 81%, 79%, 85%, and 87% accuracy; on LFWA, they achieve 74%, 71%, 81%, and 84%.LNets+ANet outperforms PANDA-w by nearly 10 percent and remains superior when PANDA-l uses groundtruth boxes and landmarks.
- Generalization and Efficiency: LNets+ANet gains 8%, 10%, and 3% on LFWA+ over FaceTracer, PANDA-w, and PANDA-l, while processing 300 × 300 images in 35 ms for localization and 14 ms for feature extraction.Without ANet pre-training, average accuracies drop 4 and 5 percent on CelebA and LFWA; LNets+ANet also performs well with small training sets.
4. Conclusion
The paper proposes a deep learning framework for face attribute prediction in the wild, combining carefully designed pre-training with efficient locally shared filters. The method handles background clutter and face variation while processing arbitrary-size images in real time without normalization.
- 4. Conclusion: The framework targets robust face attribute prediction in the wild through carefully designed pre-training strategies.These strategies improve robustness to background clutters and face variations.
- 4. Conclusion: A fast feed-forward algorithm for locally shared filters reduces redundant computation.This efficiency enables real-time evaluation of images with arbitrary sizes.
- 4. Conclusion: The method accepts images of arbitrary sizes without normalization.Arbitrary-size input is supported while retaining real-time evaluation capability.