Source-linked AI summary
UMDFaces: An Annotated Face Dataset for Training Deep Networks
Ankan Bansal, Anirudh Nanduri, Carlos Castillo, Rajeev Ranjan, Rama Chellappa
TL;DR
The paper addresses the academic shortage of large, public, and diverse face datasets and benchmarks. It introduces UMDFaces with broad face annotations, a tougher face-verification protocol, and a semi-autonomous human-and-network annotation process. The dataset is intended to support improved face-analysis research, while its scope excludes general face-detection training and some annotations are not human-generated.
Problem
Academic face-recognition research lacks large, high-quality, publicly available datasets and benchmarks with sufficient variation in pose, occlusion, and image quality.
Method
The paper constructs and releases UMDFaces using human-verified bounding boxes plus neural-network-generated pose, keypoint, and gender annotations, and introduces a tougher face-verification test protocol.
Results
UMDFaces contains 367,888 face annotations from 8,277 subjects and supports comparisons of verification networks trained on similarly sized datasets.
Takeaways & Limitations
UMDFaces provides a public, wide face dataset with pose variation and annotations for recognition, verification, keypoint localization, and head-pose estimation research.
Takeaways & Limitations
The dataset is not suitable for training face-detection models because it annotates only one subject per image and has less scale variation than detection-focused datasets.
Abstract
from arXiv · showhide
Recent progress in face detection (including keypoint detection), and recognition is mainly being driven by (i) deeper convolutional neural network architectures, and (ii) larger datasets. However, most of the large datasets are maintained by private companies and are not publicly available. The academic computer vision community needs larger and more varied datasets to make further progress. In this paper we introduce a new face dataset, called UMDFaces, which has 367,888 annotated faces of 8,277 subjects. We also introduce a new face recognition evaluation protocol which will help advance the state-of-the-art in this area. We discuss how a large dataset can be collected and annotated using human annotators and deep networks. We provide human curated bounding boxes for faces. We also provide estimated pose (roll, pitch and yaw), locations of twenty-one key-points and gender information generated by a pre-trained neural network. In addition, the quality of keypoint annotations has been verified by humans for about 115,000 images. Finally, we compare the quality of the dataset with other publicly available face datasets at similar scales.
1. Introduction
UMDFaces addresses the academic community’s need for a large, public, and varied face dataset by providing extensive annotations across thousands of subjects. The paper also describes its collection and annotation process and positions the dataset for improved face-analysis models.
- Motivation: Large corporate face datasets remain unavailable publicly, disadvantaging academic research on facial recognition.The cited corporate datasets include hundreds of millions of images and millions of identities, but were not released.
- Motivation: Public face datasets still need greater diversity in head pose, occlusion, and image quality.The paper also calls for comparisons between deep data, with more images per subject, and wide data, with more subjects.
- Dataset contribution: UMDFaces is intended to facilitate training models for face recognition, head pose estimation, and keypoint localization.The dataset is described as wide and usable separately or alongside CASIA WebFace.
- Dataset contribution: 367,888 face annotations cover 8,277 subjects in the publicly released UMDFaces dataset.The dataset includes human-verified face bounding boxes, estimated pose, twenty-one keypoints, and gender information.
- Annotation procedure: Annotations combine human-curated bounding boxes with neural-network-generated pose, keypoint, and gender labels.Keypoint locations were additionally verified by humans for 115,000 images.
2. Data Collection
UMDFaces combines web image collection, human verification, and neural-network annotation to create a cleaned, richly annotated face dataset. A verification-based cleaning pass removes remaining noisy identities before release and evaluation.
- Data collection and annotation: 367,888 images across 8,277 subjects were collected, with human-verified face bounding boxes and neural annotations for pose, keypoints, and gender.The released annotations include roll, pitch, yaw, twenty-one keypoint locations with visibility, and gender.
- Data collection and annotation: Web crawling across Yahoo, Yandex, Google, and Bing produced subject-specific image URLs after duplicate removal.The remaining images were downloaded for face detection and later annotation.
- Human cleaning: A low detection threshold retained face-box proposals for high recall, while AMT annotators removed boxes that were not faces or did not belong to the target subject.Batches included known incorrect “salt” images to measure annotator reliability.
- Human cleaning: Annotator votes were combined into a score from 0 to 1 using highly weighted votes; unreliable cases were sent for reannotation.Scores closer to 0 indicated consensus that the image showed the target individual, while scores closer to 1 indicated the opposite.
- Human cleaning: Faces with scores below 0.3 were retained, and the authors report that this mechanism labeled the collected faces economically and accurately.The threshold reflects a quality–quantity trade-off.
- Other annotations: The all-in-one CNN generated landmarks, gender, and roll, pitch, and yaw from each cleaned face box in one forward pass.Humans reviewed keypoint annotations for 115,000 images; both annotators marked 28,084 as incorrect.
- Final cleaning of the dataset: A face-verification model iteratively removed noisy boxes using cosine distances among each subject’s face features.If more than five images were removed for a subject, that subject identity was discarded; the process removed 12,789 images and 156 identities.
- Evaluation protocol: The dataset was split into non-overlapping train and test sets, and the authors report that a network trained on UMDFaces achieved the best verification performance among three evaluated networks.The released division and testing protocol are intended as a tougher evaluation metric than some existing metrics.
3. Related Works
Related datasets provide large-scale face data, but the paper identifies continuing gaps in public availability, diversity, and evaluation coverage. UMDFaces is positioned as a wide dataset with verified boxes and additional pose and keypoint annotations, while remaining unsuitable for general face-detection training.
- Motivation: Publicly available face datasets remain scarce relative to the large private datasets used by corporations, motivating continued dataset development.The paper frames Table 1 as an overview of widely used public face datasets.
- Scope and limitations: UMDFaces supports face recognition, keypoint localization, and head-pose estimation, but its annotations are not designed for general face-detection training.Only faces of one subject are boxed in each image, and scale variation is lower than in detection-focused datasets.
- Comparison with related datasets: UMDFaces is about 15 times larger than AFLW and adds human-verified face boxes plus fine-grained pose and keypoint annotations.The pose and keypoint labels were generated by the all-in-one CNN rather than human annotators, with their accuracy analyzed separately.
- Open evaluation questions: The paper highlights diversity in head pose, occlusion, and image quality, alongside comparisons between deep and wide datasets.This motivates evaluating datasets by both subject count and images per subject.
- Existing datasets: Existing datasets differ in scale and structure, including CelebFaces+, CASIA WebFace, MegaFace, and video-oriented YouTube Faces.These datasets span recognition, identification, verification, and video-based settings.
4. Dataset Statistics
UMDFaces is evaluated through dataset statistics covering annotation-generation context, head-pose diversity, and the distribution of face annotations across subjects. Its yaw distribution is broader than CASIA WebFace and its per-subject annotation counts are relatively uniform.
- Evaluation setup: The dataset statistics evaluate annotations generated by the all-in-one CNN and compare the proposed dataset with similar datasets.The annotations include keypoints and pose information, with the dataset’s statistics assessed against other datasets.
- Head-pose distribution: The yaw distribution in the proposed dataset is more spread out than CASIA WebFace and nearly matches VGG Face.The figure compares yaw-angle histograms across four datasets, including AFLW, whose distribution is much wider.
- Head-pose distribution: UMDFaces has a much wider yaw distribution than CASIA WebFace and an almost identical distribution to VGG Face.VGG Face is described as deeper, with more images per subject, than the proposed dataset.
- Annotations per subject: The number of face annotations per subject is relatively uniform around 50 images per subject.This distribution is not skewed toward a few subjects containing most annotations, unlike CASIA WebFace.
5. Experiments
Experiments evaluate UMDFaces through face verification and keypoint localization, using comparisons with CASIA WebFace, VGGFace, IJB-A, AFLW, and AFW. UMDFaces-trained models achieve stronger verification and competitive or superior landmark-localization performance, while the protocol emphasizes difficult poses and low false acceptance rates.
- 5.1. Face Verification: The verification experiment compares an AlexNet-based network trained on UMDFaces with networks trained on CASIA WebFace and VGGFace.All models are evaluated on a disjoint UMDFaces test set using aligned faces.
- 5.1.1 Test Protocol: 50,000 positive and 50,000 negative pairs in each pose track enable evaluation at very low false acceptance rates.The three tracks correspond to yaw differences below 5 degrees, between 5 and 20 degrees, and at least 20 degrees.
- 5.1.1 Test Protocol: UMDFaces-trained networks outperform the CASIA WebFace and VGGFace models, particularly at low false acceptance rates such as FPR = 10^-4.This pattern is reported on the UMDFaces test set and again on the IJB-A protocol.
- 5.1.1 Test Protocol: The authors characterize the UMDFaces protocol as tougher than LFW and IJB-A and release it with the dataset for future evaluation.The protocol is intended to compare robustness across more difficult face selections.
- 5.2. Keypoint Detection: A simple CNN trained on UMDFaces performs comparably or better than more complex systems on keypoint localization.Evaluation uses the AFLW test dataset and the AFW dataset.
- 5.2. Keypoint Detection: On AFW, the UMDFaces model outperforms recently published methods at a normalized mean error of 5%, while on AFLW it performs comparably or better.The model uses the VGG-Face architecture with its final layer changed to predict keypoints.
- 5.2. Keypoint Detection: The keypoint experiment provides baseline landmark-localization results and supports the reported quality of the dataset.The authors describe the network as bare-bones and release its weights publicly.
6. Discussion
UMDFaces is released as a public resource for face recognition and verification, combining diverse annotations with a tougher verification protocol. The paper also demonstrates strong verification and keypoint-localization results and reports human verification of part of the keypoint annotations.
- 6. Discussion: The released dataset contains identity, bounding-box, twenty-one-keypoint, 3D-pose, and gender information, with greater pose variation than CASIA WebFace.The authors also report human verification of keypoint annotations for part of the data.
- 6. Discussion: The released verification protocol is tougher than commonly used protocols and is intended to support evaluation of face-verification methods.The paper compares networks trained on similarly sized datasets and reports that UMDFaces-trained networks perform best.
- 6. Discussion: A simple CNN trained on UMDFaces performs comparably or better than complex recent keypoint-localization algorithms.The paper presents this comparison as evidence of the automatically generated keypoint locations' quality.