PaperlayerView exact revision on arXiv

Source-linked AI summary

VGGFace2: A dataset for recognising faces across pose and age

Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, Andrew Zisserman

arXiv:1710.08092v2cs.CV

TL;DR

Existing face datasets were not specifically designed to capture broad pose and age variation. This paper introduces and evaluates VGGFace2, whose models achieve state-of-the-art performance across IJB-A, IJB-B, and IJB-C benchmarks.

  • Problem

    Existing face datasets were not specifically designed to explore pose and age variation alongside other facial variations.

  • Method

    The paper constructs VGGFace2 with pose- and age-diverse images, applies automated and manual filtering, and trains ResNet-50 and SENet models for evaluation.

  • Results

    VGGFace2-trained models achieve state-of-the-art performance on IJB-A, IJB-B, and IJB-C, including a ~4.4% improvement over MS1M at FAR=0.001 on IJB-A verification.

  • Takeaways & Limitations

    The publicly released dataset and models provide a resource for face recognition across pose and age variation.

Abstract

from arXiv · show

In this paper, we introduce a new large-scale face dataset named VGGFace2. The dataset contains 3.31 million images of 9131 subjects, with an average of 362.6 images for each subject. Images are downloaded from Google Image Search and have large variations in pose, age, illumination, ethnicity and profession (e.g. actors, athletes, politicians). The dataset was collected with three goals in mind: (i) to have both a large number of identities and also a large number of images for each identity; (ii) to cover a large range of pose, age and ethnicity; and (iii) to minimize the label noise. We describe how the dataset was collected, in particular the automated and manual filtering stages to ensure a high accuracy for the images of each identity. To assess face recognition performance using the new dataset, we train ResNet-50 (with and without Squeeze-and-Excitation blocks) Convolutional Neural Networks on VGGFace2, on MS- Celeb-1M, and on their union, and show that training on VGGFace2 leads to improved recognition performance over pose and age. Finally, using the models trained on these datasets, we demonstrate state-of-the-art performance on all the IARPA Janus face recognition benchmarks, e.g. IJB-A, IJB-B and IJB-C, exceeding the previous state-of-the-art by a large margin. Datasets and models are publicly available.

I. INTRODUCTION · II. DATASET REVIEW

The paper introduces VGGFace2 to address limitations in existing face datasets by combining broad identity coverage with substantial pose and age variation while reducing label noise. It reviews prior datasets, highlighting trade-offs between dataset breadth, per-identity variation, and annotation quality.

  • I. INTRODUCTION: Earlier datasets generally emphasised either intra-class depth, with many images per subject, or inter-class breadth, with many subjects and limited images per subject.The introduction notes that these datasets were not specifically designed to explore pose-related variation.
  • I. INTRODUCTION: VGGFace2 contains over nine thousand identities, more than 3M images, and 80–800 images per identity for public release.The dataset is designed to provide both many identities and substantial image depth per identity.
  • I. INTRODUCTION: Its generation pipeline promotes pose and age diversity while using multiple automatic and manual filtering stages to minimise label noise.The paper also provides template annotations for explicitly evaluating pose and age recognition.
  • I. INTRODUCTION: Training deep CNNs on VGGFace2 substantially exceeds state-of-the-art performance on the IJB benchmark datasets.Experiments include Squeeze-and-Excitation networks and pre-training on the breadth-oriented MS-Celeb-1M dataset.
  • II. DATASET REVIEW: LFW, CelebFaces+, CASIA-WebFace, and VGGFace provide progressively larger public face datasets, with VGGFace containing 2.6 million images across 2,622 people.The curated VGGFace version contains 800,000 images and approximately 305 images per identity after human removal of label noise.
  • II. DATASET REVIEW: MegaFace contains 4.7 million training images from 672,057 identities, but averages only 7 images per identity, restricting per-identity variation.Its challenge uses FaceScrub subsets to study pose and age variation in face recognition.
  • II. DATASET REVIEW: MS-Celeb-1M has the largest number of training images but averages 81 images per person and contains label noise because images were retrieved without manual filtering.The paper uses MS-Celeb-1M for pre-training.
  • II. DATASET REVIEW: YTF and UMDFaces-Videos target unconstrained video face recognition, containing 3,425 videos from 1,595 identities and 22,075 videos from 3,107 identities, respectively.The identities in UMDFaces-Videos are a subset of those in UMDFaces.

III. AN OVERVIEW OF THE VGGFACE2 · A. Dataset Statistics

VGGFace2 is a large-scale face dataset designed to capture substantial variation across identities, ethnicity, profession, pose, age, lighting, and background. It contains separate training and evaluation splits, with pose and apparent-age statistics estimated using pretrained classifiers.

  • A. Dataset Statistics: 3.31 million images cover 9131 celebrities across varied ethnicities and professions, including politicians and athletes.The dataset includes more Chinese and Indian faces than VGGFace, although celebrity demographics limit ethnic balance.
  • A. Dataset Statistics: Images were downloaded from Google Image Search and exhibit large variations in pose, age, lighting, and background.
  • A. Dataset Statistics: Pose statistics are estimated for yaw, pitch, and roll using pretrained pose classifiers.Pose statistics and example images are presented in Figure 1.
  • A. Dataset Statistics: The dataset’s ethnic coverage is broad but remains limited by the distribution of celebrities and public figures.
  • A. Dataset Statistics: Apparent-age information is estimated using pretrained age classifiers.Age statistics and example images are presented in Figure 1.
  • A. Dataset Statistics: The dataset uses two splits: 8631 classes for training and 500 classes for evaluation.

B. Pose and Age Annotations · IV. DATASET COLLECTION · A. Stage 1: Obtaining and selecting a name list

VGGFace2 provides pose- and age-based evaluation annotations, while its collection pipeline obtains candidate identities, downloads and filters images, and cleans the dataset automatically and manually. Candidate names begin with 500,000 public figures and are reduced to 9,244 through identity-consistency screening.

  • B. Pose and Age Annotations: The dataset supports face matching across different poses and across different ages.
  • B. Pose and Age Annotations: Pose templates contain five same-subject faces with frontal, three-quarter, or profile views.For 300 evaluation subjects, two templates per pose view are provided, totaling 1.8K templates and 9K images.
  • B. Pose and Age Annotations: Age templates contain five same-subject faces labeled young below 34 or mature at 34 and above.For 100 evaluation subjects, two templates per age period are provided, totaling 400 templates and 2K images.
  • IV. DATASET COLLECTION: The collection process covers candidate-identity selection, image collection, and automatic and manual dataset cleanup.The paper summarizes this process in Table II.
  • A. Stage 1: Obtaining and selecting a name list: The first collection stage seeks subjects with many sufficiently distinct images, focusing on celebrities and public figures such as actors, politicians, and athletes.An initial list of 500k public figures is obtained from the Freebase knowledge graph.
  • A. Stage 1: Obtaining and selecting a name list: For each of the 500K names, 100 Google Image Search images are downloaded for human annotators to assess identity consistency.Candidates are retained when approximately 90% or more of the images belong to a single identity.
  • A. Stage 1: Obtaining and selecting a name list: This screening removes names lacking sufficient images or returning mixed identities, reducing the candidate list to 9244 names.The method uses an annotator team to remove unsuitable identities from the candidate list.

B. Stage 2: Obtaining images for each identity · C. Stage 3: Face detection · D. Stage 4: Automatic filtering by classification

The dataset pipeline retrieves 1,400 images per identity, detects and normalizes faces with landmark predictions, and automatically filters likely outliers using identity classifiers.

  • C. Stage 3: Face detection: Faces are detected with the model from, using its recommended hyper-parameters to balance precision and recall.The detector is configured to favor a good trade-off between precision and recall.
  • C. Stage 3: Face detection: Detected face bounding boxes are extended by 0.3 to include the whole head.The expansion factor is applied after face detection.
  • C. Stage 3: Face detection: The same model predicts five facial landmarks for each detected face.Landmark prediction accompanies the expanded face bounding boxes.
  • D. Stage 4: Automatic filtering by classification: Automatic filtering removes identity-specific outlier faces by discarding images whose classifier scores fall below a threshold.The stage learns classifiers to identify faces and remove possible erroneous faces.
  • D. Stage 4: Automatic filtering by classification: 1-vs-rest classifiers are trained across 9,244 subjects using the top 100 retrieved images per identity as positives and the top 100 images from other identities as negatives.This training setup uses retrieved images to discriminate each subject from all other subjects.

E. Stage 5: Near duplicate removal · F. Stage 6: Final automatic and manual filtering

Stage 5 removes exact and near-duplicate images by VLAD-descriptor clustering, while Stage 6 combines automated and manual filtering to eliminate outliers, overlapping identities, and residual label noise. The final procedure targets purity greater than 96% and achieves very low label noise without manually checking every image.

  • E. Stage 5: Near duplicate removal: Exact and near-duplicate images are removed by clustering VLAD descriptors and retaining one image per cluster.Duplicates arise from images found at different internet locations or differing only slightly in colour balance or JPEG artifacts.
  • F. Stage 6: Final automatic and manual filtering: Stage 6 addresses outlier images and classes containing multiple people or overlapping identities using automated algorithms and manual filtering.These are the two residual error types identified after earlier processing stages.
  • F. Stage 6: Final automatic and manual filtering: 20 subjects confused with others are identified by training ResNet-50 on half of each class and measuring top-1 error on the remaining half.The resulting confusion matrix is used to detect overlapping subjects such as ‘Will I Am’ and ‘William James Adams’.
  • F. Stage 6: Final automatic and manual filtering: The outlier-removal process aims for purity greater than 96%, including cases where couples or band members consistently appear together.High classifier scores can still correspond to noisy images when mixed examples cause the stage-4 classifiers to fail.
  • F. Stage 6: Final automatic and manual filtering: Images are divided by classifier score into H ([1, 0.95]), I ((0.95, 0.8]) and L ((0.8, 0.5]) sets for subject-level cleaning.Human annotators clean these sets according to whether the high-score set contains multiple people or is clean.
  • F. Stage 6: Final automatic and manual filtering: If H contains several people, all three sets are manually cleaned; if H is clean, only L is manually cleaned before model-based cleaning of I.A new model is trained on the cleaned H and L sets, then predicts which images in I should be removed.
  • F. Stage 6: Final automatic and manual filtering: The combined procedure achieves very low label noise without requiring manual checking of every image.This result follows from retraining on cleaned sets and using model prediction for the intermediate-score set.

G. Pose and age annotations · V. EXPERIMENTS · A. Experimental setup

The paper derives pose and apparent-age annotations with dedicated ResNet-50 classifiers, then evaluates VGGFace2 through controlled baseline experiments against public IJB benchmarks. The setup compares multiple training regimes and specifies the architectures, descriptor extraction, template similarity, and optimization procedures.

  • G. Pose and age annotations: A 5-way ResNet-50 trained on CASIA-WebFace predicts head roll, pitch, and yaw for every dataset image.The trained model is applied across the full dataset to obtain pose annotations.
  • G. Pose and age annotations: An 8-way ResNet-50 trained on the IMDB-WIKI-500k+ dataset predicts apparent age for faces.The resulting model supplies age estimates for the dataset images.
  • V. EXPERIMENTS: The experiments assess VGGFace2 on its test set and the public IJB benchmarks, using training subjects disjoint from benchmark subjects.MS-Celeb-1M and the two benchmarks are also de-overlapped before network training.
  • A. Experimental setup: ResNet-50 and SE-ResNet-50 serve as backbone architectures for comparisons across training datasets.SE blocks adaptively recalibrate channelwise responses by modelling channel relationships and are intended to improve representational power.
  • A. Experimental setup: Four training settings compare scratch training on VGGFace, MS-Celeb-1M, or VGGFace2 with MS-Celeb-1M pre-training followed by VGGFace2 fine-tuning.The settings are denoted VF, MS1M, VF2, and VF2 ft, respectively.
  • A. Experimental setup: Template similarity uses aggregated face descriptors, with averaging and L2 normalisation for mixed still-image and video-frame templates and cosine similarity between templates.For identification in section V-B, SVM classifiers are used after averaging face descriptors.
  • A. Experimental setup: Face descriptors are 2048-dimensional, L2-normalised outputs extracted adjacent to the classifier layer after resizing and centre-cropping the extended face bounding box.The shorter image side is resized to 256 pixels and the centre 224 × 224 crop is used as network input.
  • A. Experimental setup: Networks use soft-max classification training with random 224×224 crops, channel-mean subtraction, 20% monochrome augmentation, and balanced mini-batches of size 256.Stochastic gradient descent starts at learning rate 0.1 for scratch-trained models, with two tenfold reductions when errors plateau.

B. Experiments on the new dataset

Experiments assess identification and template similarity across pose and age using ResNet-50 models trained on three datasets. VGGFace2 achieves strong identification and the highest cross-pose and cross-age similarity, supporting its coverage and low label noise.

  • Face identification: The evaluation measures top-1 identification error on 500 subjects, using 50 randomly selected test images per subject and 1-vs-rest SVM classifiers.ResNet-50 models are trained from scratch on three datasets and evaluated on the VGGFace2 test set.
  • Face identification: VGGFace2-trained models outperform MS1M despite MS1M having tenfold more subjects and threefold more images, indicating high dataset quality and little label noise.The very low top-1 error is presented as evidence that VGGFace2 contains little label noise.
  • Probing across pose: Similarity is higher for matching identical pose views and drops for cross-pose matches, showing that front-to-three-quarter and front-to-profile recognition are harder.The analysis averages similarity matrices over 300 subjects annotated with front, three-quarter, and profile templates.
  • Probing across age: Similarity drops substantially between young and mature faces, making cross-age matching harder than within-age matching; young-to-young matching is also harder than mature-to-mature matching.The age analysis averages 2 × 2 similarity matrices over 100 subjects.
  • Discussion: VGGFace2-trained models consistently achieve the highest pose- and age-similarity scores, while MS1M performs worst, highlighting the value of greater intra-class variation.MS1M’s emphasis on inter-class diversity is described as harmful to matching across pose and age.
  • Discussion: VGGFace2 model scores are higher for front-to-profile pairs differing mainly in pose and lower when expression, resolution, pose, or occlusion introduce additional variation.This pattern appears in both front-to-profile and young-to-mature template examples.

C. Experiments on IJB-A

Experiments on IJB-A evaluate models trained on different datasets and architectures using unconstrained face verification and identification protocols. VGGFace2-trained models outperform prior training sets across reported metrics, while SENet further improves performance and fine-tuning offers only a small additional gain.

  • Evaluation protocols: IJB-A evaluates 1:1 face verification and 1:N face identification across ten splits using cosine similarity features.The dataset contains 5712 images and 2085 videos from 500 subjects, with substantial expression and image-quality variation.
  • The effect of training set: ∼4.4% improvement over MS1M on FAR=0.001, ∼3.7% on FPIR=0.01, and ∼1.5% on Rank-1 are achieved by VGGFace2 on IJB-A.VGGFace2 also outperforms VGGFace despite similar scale and surpasses MS1M despite MS1M having 10 times more subjects.
  • The effect of training set: VGGFace2-trained models surpass previously reported results on all IJB-A verification and identification metrics.The results are reported for the 1:1 verification and 1:N identification protocols.
  • The effect of training set: 0.908 vs. 0.895 is the reported difference between MS1M-pretrained, VGGFace2-fine-tuned generalisation and the comparison model.Fine-tuning with VGGFace2 after MS1M pretraining further improves generalisation, but the difference is described as small.
  • The effect of architectures: SENet trained on VGGFace2 consistently outperforms ResNet-50 on both verification and identification.SENet trained from scratch achieves comparable results to ResNet-50 first pretrained on MS1M, indicating that VGGFace2 diversity can be further exploited.

D. Experiments on IJB-B

On IJB-B, the authors evaluate face-recognition models using standard mixed-media verification and identification protocols. VGGFace2 training, dataset integration, and SENet architectures improve performance over MS1M, ResNet-50, and prior benchmark results.

  • Evaluation protocol: IJB-B contains 1,845 subjects, 21.8K still images, and 55K frames from 7,011 videos, evaluated with standard 1:1 verification and 1:N identification protocols.The protocols match Mixed Media probes against two galleries.
  • Training-set comparison: Models trained on VGGFace2 significantly surpass models trained on MS1M, while integrating both datasets further improves performance.The comparison is reported in Table VII and Figure 10.
  • Architecture comparison: SENet outperforms ResNet-50 in both verification and identification when trained from scratch or fine-tuned.The superiority is observed under both training settings.
  • Benchmark comparison: The evaluated models considerably improve over other reported benchmark results across all measures.The comparison is shown in Table VII.

E. Experiments on IJB-C

The models are evaluated on IJB-C using standard verification and identification protocols. Compared with results reported in, they show considerable improvement across all measures.

  • Dataset and protocols: IJB-C contains 3531 subjects, 31.3K still images, and 117.5K frames from 11,779 videos.The dataset extends IJB-B.
  • Dataset and protocols: The evaluation uses the standard 1:1 verification protocol.
  • Dataset and protocols: The evaluation also uses the standard 1:N identification protocol.
  • Results: Compared with results reported in, the models achieve considerable improvement for all measures.Results are reported in Table VIII and Figure 11.

VI. CONCLUSION

The paper proposes a high-quality VGGFace2 dataset collection pipeline spanning pose and age, and shows that ResNet-50 and SENet models trained on it achieve state-of-the-art performance on IJB-A, IJB-B, and IJBC.

  • VI. CONCLUSION: The proposed pipeline collects the high-quality VGGFace2 dataset across a wide range of pose and age.The dataset and trained models are publicly available.
  • VI. CONCLUSION: ResNet-50 and SENet models trained on VGGFace2 achieve state-of-the-art performance on the IJB-A, IJB-B, and IJBC benchmarks.
Loading 1710.08092v2…