Source-linked AI summary
MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, Jianfeng Gao
TL;DR
The paper addresses web-scale face identification with disambiguation, asking who appears in an image and which entity they represent. It defines a one-million-celebrity benchmark linked to knowledge-base entity keys, provides datasets and evaluation protocols, and reports 44.2% coverage at 95% precision on a hard measurement set.
Problem
Web-scale face recognition lacked sufficient methods for identifying a person and disambiguating among people with the same name, rather than only retrieving visually similar images.
Method
The paper constructs a one-million-celebrity benchmark linking face images to knowledge-base entity keys, with manually labeled measurements and evaluation based on maximum coverage at precision P(t) ≥ 0.95.
Results
44.2% of images in the hard measurement set were recognized at 95% precision without extra model-structure fine-tuning.
Takeaways & Limitations
The benchmark, measurement set, and large training dataset provide a baseline and support further research on recognition coverage, noisy data, clustering, outlier removal, and face-based property estimation.
Takeaways & Limitations
The training data contains unmanually removed noise because labeling a dataset of this scale was impractical.
Abstract
from arXiv · showhide
In this paper, we design a benchmark task and provide the associated datasets for recognizing face images and link them to corresponding entity keys in a knowledge base. More specifically, we propose a benchmark task to recognize one million celebrities from their face images, by using all the possibly collected face images of this individual on the web as training data. The rich information provided by the knowledge base helps to conduct disambiguation and improve the recognition accuracy, and contributes to various real-world applications, such as image captioning and news video analysis. Associated with this task, we design and provide concrete measurement set, evaluation protocol, as well as training data. We also present in details our experiment setup and report promising baseline results. Our benchmark task could lead to one of the largest classification problems in computer vision. To the best of our knowledge, our training dataset, which contains 10M images in version 1, is the largest publicly available one in the world.
1 Introduction
The paper defines a one-million-celebrity face-recognition benchmark that links images to unique knowledge-base entities, addressing web-scale identity disambiguation. It also provides measurement data, a large training dataset, and a baseline evaluated at high precision.
- Task definition: The benchmark recognizes one million celebrities from face images and links each identity to a unique knowledge-base entity key.Entity linking distinguishes people with the same or similar names and connects recognition to associated properties.
- Motivation: The task addresses limited web-scale disambiguation, where conventional face identification emphasizes image similarity rather than determining which person is depicted.The paper frames disambiguation as distinguishing entities such as different people sharing a name.
- Challenges: One million classes create both low inter-class variance and high intra-class variance because celebrities may look alike while individual appearances vary across age, makeup, or surgery.Popular celebrities may also have millions of available images, increasing within-class variation.
- Datasets and evaluation: The authors provide a manually labeled measurement set, an evaluation protocol, and training data with cropped and aligned face regions.The measurement set emphasizes popular celebrities while retaining about 25% tail celebrities for coverage.
- Datasets and evaluation: The version-one training dataset contains about 10M images for 100K top celebrities and is presented as the largest publicly available face-recognition dataset.The data is intended to facilitate research on the proposed benchmark and is available with face-region preprocessing.
- Baseline: 44.2% of measurement-set images are recognized at 95% precision without extra model-structure fine-tuning, providing a baseline for the benchmark.The authors describe the baseline as promising and encourage further work on data, algorithms, and generalization.
2 Related works
Prior face-recognition benchmarks largely target verification or gallery-based identification, while existing public datasets differ in scale and purpose. The paper positions its dataset as larger and more focused on broad celebrity coverage and disambiguated identities.
- Face verification: Face verification determines whether two face images belong to the same person, with LFW providing 3,000 matched and 3,000 mismatched pairs.LFW supports reporting verification accuracy under different settings.
- Face identification: Face identification typically retrieves similar faces from a gallery for each query image, especially when only a few gallery images represent each individual.This differs from the paper’s entity-linking objective, which asks who appears in the image and which same-named entity is intended.
- Existing datasets: Public datasets such as LFW, YFD, CelebFaces+, and CASIA-WebFace support face verification or identification training with substantially smaller reported collections.The cited datasets include 13,000 LFW images, 3,425 YFD videos, and 202,599 CelebFaces+ images for 10,177 celebrities.
- Dataset distinction: The proposed training dataset is considerably larger and prioritizes coverage of popular celebrities plus image disambiguation rather than only learning generalizable face features.Existing datasets may remove ambiguous-name celebrities to improve collection precision, whereas this benchmark explicitly addresses disambiguation.
3 Benchmark construction
The benchmark defines a one-million-celebrity recognition task linked to unique Freebase entity keys and builds diverse, web-scaled data and evaluation sets. Its measurement design balances popular celebrities with tail coverage through adjusted sampling, rigorous labeling, and complementary image subsets.
- 3.1 One million celebrity list: The task selects one million real people with public attention from Freebase, representing each entity with a unique machine identifier and rich associated properties.Entities with unknown or unclear facial appearance are excluded, and candidates are ranked by web occurrence frequency.
- 3.1 One million celebrity list: The celebrity list spans more than 2000 professions, more than 200 countries or regions, major races, and a large age range, with more females than males.This diversity emerges from the dataset’s scale rather than manual balancing.
- 3.1 One million celebrity list: Property statistics are limited to celebrities for whom the corresponding information is available in Freebase.Celebrities missing a property are excluded from that property’s statistic.
- 3.2 Celebrity selection for measurement: The measurement set samples mainly popular celebrities while retaining about 25% from the bottom 90% of the list to evaluate coverage of less frequently mentioned people.Sampling is adjusted because direct weighting by web frequency would leave too few tail celebrities; without adjustment, fewer than 10% would come from the bottom 90%.
- 3.3 Labeling for measurement: The measurement set construction uses web scraping, multiple query variations, manual labeling with at least two judges, and consensus review of about 30K images.Scraping collects about 30 candidates per celebrity, while only two labeled images per celebrity are used for evaluation.
- 3.3 Labeling for measurement: Two evaluation subsets use one image per celebrity: a random labeled image measures coverage, while the image most different from training data measures generalization.The published development set contains 500 celebrities, with the remaining 1000 held for grand challenges.
4 Celebrity recognition
The paper defines an evaluation protocol for large-scale celebrity recognition, supplies training data, and reports a baseline while leaving data coverage and noise handling as open challenges.
- 4.1 Evaluation Protocol: The measurement set combines labeled celebrity images with hidden distractors to test recognition across as many people as possible.The protocol hides which images are labeled and which are distractors.
- 4.1 Evaluation Protocol: The evaluation uses precision and coverage, with maximum coverage at precision P(t) ≥ 0.95 as the primary metric.Models output an entity key and confidence, and may reject predictions below a threshold.
- 4.2 Training dataset: The training dataset selects the top 100K celebrities and retrieves approximately 100 images per celebrity from popular search engines.The dataset is intended to facilitate the one-million-celebrity benchmark without covering the entire celebrity list.
- 4.2 Training dataset: The dataset covers about 75% of measurement-set celebrities, so recognition recall based only on the provided training data cannot exceed 75%.The authors encourage outside data and entity-key labeling to surpass this bound.
- 4.2 Training dataset: Training images may contain noise, including a synthesized Steve Jobs–Ashton Kutcher image, because manual cleaning at this scale is impractical.The authors leave noisy-label removal and learning with noisy data as open research problems.
5 Discussion and Future work
The paper defines a million-celebrity face-recognition benchmark linked to knowledge-base entity keys and provides evaluation measures and a large training dataset. It also highlights applications beyond recognition, including clustering, outlier removal, and property estimation from face images.
- The benchmark recognizes one million celebrities from face images and links each face to a corresponding knowledge-base entity key.
- The paper provides concrete measurement methods and a large training dataset to facilitate evaluation and further research.
- The datasets could support unsupervised or semi-supervised clustering and methods for locating and removing outliers in large real-world data.
- Knowledge-base properties associated with the images could support estimators for predicting attributes such as gender from faces in the wild.