PaperlayerView exact revision on arXiv

Source-linked AI summary

How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)

Adrian Bulat, Georgios Tzimiropoulos

arXiv:1703.07332v3cs.CVcs.LG

TL;DR

The paper asks how close deep neural networks are to saturating existing 2D and 3D face-alignment datasets. It builds strong alignment networks, creates the unified LS3D-W dataset from 2D-guided 3D annotations, and evaluates performance across datasets and conditions. Both 2D and 3D networks achieve remarkable accuracy probably close to saturation on the datasets used.

  • Problem

    It remains unclear what has been achieved so far for face alignment, especially how close neural networks are to saturating existing 2D and 3D datasets.

  • Method

    The paper combines strong landmark-localization components for 2D alignment, uses a 2D-guided CNN to convert annotations and build LS3D-W, then trains and evaluates a 3D alignment network.

  • Results

    Both 2D and 3D face-alignment networks achieve remarkable accuracy probably close to saturating the datasets used.

  • Takeaways & Limitations

    The results indicate that these networks are resilient to pose, resolution, initialization, and network-parameter count across the evaluated datasets.

  • Takeaways & Limitations

    The evaluation does not explore some very unfamiliar poses, and discrepancies in 3D landmarks are mainly attributed to inaccurate semi-automatic annotations for difficult poses.

Abstract

from arXiv · show

This paper investigates how far a very deep neural network is from attaining close to saturating performance on existing 2D and 3D face alignment datasets. To this end, we make the following 5 contributions: (a) we construct, for the first time, a very strong baseline by combining a state-of-the-art architecture for landmark localization with a state-of-the-art residual block, train it on a very large yet synthetically expanded 2D facial landmark dataset and finally evaluate it on all other 2D facial landmark datasets. (b) We create a guided by 2D landmarks network which converts 2D landmark annotations to 3D and unifies all existing datasets, leading to the creation of LS3D-W, the largest and most challenging 3D facial landmark dataset to date ~230,000 images. (c) Following that, we train a neural network for 3D face alignment and evaluate it on the newly introduced LS3D-W. (d) We further look into the effect of all "traditional" factors affecting face alignment performance like large pose, initialization and resolution, and introduce a "new" one, namely the size of the network. (e) We show that both 2D and 3D face alignment networks achieve performance of remarkable accuracy which is probably close to saturating the datasets used. Training and testing code as well as the dataset can be downloaded from https://www.adrianbulat.com/face-alignment/

1. Introduction

The paper addresses the unclear progress of face alignment by building powerful 2D and 3D neural-network baselines, creating a large unified 3D dataset, and evaluating performance across existing datasets. It concludes that both networks achieve remarkable accuracy probably close to saturating the datasets used.

  • The paper targets the unresolved question of how close current deep networks are to saturating performance on 2D and 3D face alignment datasets.
  • The authors construct a strong 2D baseline by combining a state-of-the-art landmark-localization architecture with a state-of-the-art residual block and training it on a synthetically expanded dataset.They evaluate the model across all other 2D datasets, totaling approximately 230,000 images.
  • They introduce a guided-by-2D-landmarks CNN that converts 2D annotations to 3D and unifies existing datasets into LS3D-W, containing approximately 230,000 images.
  • They train a 3D face-alignment network on the newly introduced LS3D-W dataset and investigate traditional factors such as pose, initialization, and resolution.
  • Both 2D and 3D face-alignment networks achieve remarkable accuracy that is probably close to saturating the datasets used.

2. Closely related work

Earlier face-alignment methods largely solved controlled-pose 2D datasets, but evidence remained limited for large-pose and 3D alignment. Existing datasets had correspondence, scale, annotation-quality, and coverage limitations that motivated broader cross-database evaluation.

  • Cascaded regression methods were considered to have largely solved 2D face alignment for controlled-pose datasets such as LFPW, Helen, and 300-W.
  • The paper uses the frontal LFPW error curve as a comparison point for judging whether another dataset is close to saturation.
  • CNN face-alignment methods had mainly been demonstrated on near-frontal 300-W faces, leaving large-pose and 3D performance less conclusive.
  • Existing large-pose and 3D studies relied on datasets ranging from 3,900 training and 1,200 testing images to 16,2000 training and 4,900 testing images, with limited pose coverage in one case.
  • The paper unifies AFLW-2000, the 300-W test set, 300-VW, and Menpo to provide 3D annotations across datasets totaling about 230,000 images.
  • 2D annotations can lose correspondence at moderately large poses, while available 3D test coverage was limited largely to AFLW2000-3D.
  • AFLW2000-3D includes yaw from −90o to 90o but has less accurate annotations for larger poses or occluded faces.
  • The evaluation normalizes point-to-point Euclidean landmark error by the square root of the ground-truth bounding-box area, using 2D landmarks for both 2D and 3D comparisons.

4. Method

The method uses Face Alignment Networks for 2D and 3D landmark localization and a guided 2D-to-3D variant to construct LS3D-W. FAN combines stacked HourGlass networks with hierarchical, parallel, multi-scale residual blocks, while 2D-to-3D FAN uses RGB images and 2D landmark guidance.

  • FAN is the network used for 2D and 3D face alignment, while 2D-to-3D FAN constructs LS3D-W, containing more than 230,000 3D landmark annotations.
  • The experiments call the network Face Alignment Network (FAN), described as the first such powerful network trained and evaluated for large-scale 2D/3D face-alignment experiments.
  • FAN uses a stack of four HourGlass networks and replaces their bottleneck blocks with hierarchical, parallel, multi-scale blocks.
  • The 2D-to-3D-FAN takes an RGB image and 2D landmarks as input and outputs the corresponding 2D projections of 3D landmarks.
  • 2D-to-3D FAN augments RGB input with 68 channels containing Gaussian maps centered on the 2D landmarks, then converts 2D facial landmarks to 3D.
  • The 2D-to-3D FAN is trained on 300-W-LP, which provides paired 2D and 3D annotations for the same images.
  • The experiments independently train 2D-FAN, 3D-FAN, and 2D-to-3D-FAN with distinct training procedures and augmentation settings.

5. 2D face alignment

2D-FAN is evaluated across more than 220,000 images and achieves near-saturating performance on the reported 2D datasets, despite synthetic training data and annotation mismatch.

  • 2D-FAN was evaluated on 300-W test, 300-VW, and Menpo, covering more than 220,000 images.The model was trained on 300-W-LP-2D and fine-tuned on the original 300-W training set.
  • 2D-FAN achieves essentially the same performance across datasets except 300-VW Category C, outperforming MDM and ICCR while matching MDM-on-LFPW.MDM-on-LFPW is used as an unconventional near-saturation reference.
  • 0.25% of 7,200 Menpo and 300-W test images are failures under the NME > 7% criterion.The count is 18 failure cases.
  • 300-VW Category C contains 101 failure frames, representing 0.38% of the dataset, and annotation quality is identified as the main reason for the performance drop.In many highest-error cases, the predictions appear more accurate than the ground-truth landmarks.
  • The authors conclude that 2D-FAN achieves near-saturating performance on the evaluated 2D datasets.This conclusion holds despite primarily synthetic training data and mismatched training and testing landmark annotations.

6. Large Scale 3D Faces in-the-Wild dataset

The paper converts existing 2D annotations into 3D using a guided network and unifies the resulting annotations into LS3D-W, while identifying inaccuracies in prior AFLW2000-3D labels.

  • A guided 2D-to-3D FAN converts 2D annotations into 3D landmarks for the 300-W test set, 300-VW, and Menpo.The conversion is guided by predictions from 2D-FAN.
  • On AFLW2000-3D, the pipeline compares its generated 3D landmarks against the original annotations while retaining 2D bounding boxes for normalization.The pipeline applies 2D-FAN first and then converts the resulting landmarks to 3D.
  • 19 AFLW2000-3D cases, representing 0.9% of the data, are removed as failures before inspecting the highest-error examples.The reported discrepancy is mainly attributed to the semi-automatic annotation pipeline used for the original labels, especially in difficult poses.
  • The paper reports that some AFLW2000-3D annotations, particularly for large poses or occluded faces, are inaccurate.This annotation limitation is also visible in the comparison between original labels and generated landmarks.
  • ~230,000 images are annotated with 3D landmarks, creating LS3D-W, described as the largest 3D face alignment dataset to date.AFLW2000-3D is included alongside the other datasets.

7. 3D face alignment

3D-FAN is evaluated on approximately 230,000 images from LS3D-W and produces consistent accuracy across the datasets, substantially outperforming 3DDFA.

  • 3D-FAN is evaluated on approximately 230,000 images from the 3D landmarks in LS3D-W and re-annotated AFLW2000-3D.The evaluation includes 300-W test, 300-VW, Menpo, and AFLW2000-3D.
  • 3D-FAN produces essentially the same accuracy on 300-W test, 300-VW, Menpo, and AFLW2000-3D, substantially outperforming 3DDFA.3DDFA was trained on the same dataset as 3D-FAN.
  • 3D-FAN slightly improves on 2D-FAN, especially for errors below 2%.The authors relate this increase to matching landmark markup between training and testing datasets.

8. Ablation studies

The ablations evaluate 3D-FAN under balanced pose, reduced face resolution, noisy initialization, and varying network size. Performance degrades only slightly at extreme poses and higher initialization noise, remains high down to 30-pixel faces, and is nearly unchanged for 12–24M-parameter networks.

  • 7,200 images form LS3D-W Balanced, with equal representation across yaw ranges [0°–30°], [30°–60°], and [60°–90°].
  • Performance across pose: Only slight AUC degradation occurs for very large poses ([60°–90°]).AUC is calculated at a 7% threshold.
  • Performance across resolution: Significant performance drop appears only when face size reaches 30 pixels, so resolution is not a major issue for 3D-FAN.The network was not retrained specifically for low resolutions.
  • Performance across noisy initializations: Only small performance decrease occurs at 30% initialization noise, above the 20% noise level used during training.
  • Performance across different network sizes: Between 12–24M parameters, performance remains almost the same.The largest tested 3D-FAN uses 24M parameters, with size varied by reducing hourglass networks and internal channels.

9. Conclusions

The paper trains a state-of-the-art landmark-localization network for 2D and 3D face alignment and evaluates it on hundreds of thousands of images. The results indicate near-saturating performance with resilience to pose, resolution, initialization, and network-parameter count.

  • A state-of-the-art landmark-localization network is trained for 2D and 3D face alignment and evaluated on hundreds of thousands of images.
  • The networks nearly saturate the evaluated datasets and remain resilient to pose, resolution, initialization, and network-parameter count.The conclusion notes that some very unfamiliar poses were not explored.

A1. Additional numeric results

Additional numeric results report 2D-FAN performance on the 300-VW training set using NME over all 68 facial landmarks.

  • Figure 10 reports NME using all 68 points on the 300-VW training set.

A2. Additional visual results

Additional visual results show randomly selected 2D-FAN and 3D-FAN predictions on LS3D-W Balanced under challenging visual conditions.

  • Figures 13 and 14 show randomly selected 2D-FAN and 3D-FAN samples on LS3D-W Balanced.The examples include extreme poses, expressions, and lighting conditions, which the method copes with very well.

A3. Full 3D face alignment

The paper extends 2D-to-3D-FAN into a full 3D face-alignment network that predicts facial-landmark depth alongside x and y coordinates. It combines RGB input with 2D landmark heatmaps to produce 3D landmarks and demonstrates fitting results on LS3D-W and AFLW2000-3D.

  • Full-2D-to-3D-FAN adds prediction of the z coordinate, representing each facial landmark’s depth.
  • The network takes an RGB image and 2D landmarks as input and outputs corresponding 3D landmarks.
  • The depth subnetwork receives RGB images and stacked heatmaps generated by 2D-to-3D-FAN.The heatmaps indicate where depth should be predicted and provide additional pose-related information.
  • A ResNet-152 subnetwork is adapted for 3+N input channels and an N × 1 landmark-depth output.It is trained with L2 loss for 50 epochs using the same learning rates as the other networks.
  • Figure 12 compares AFLW2000-3D depth errors between original annotations and annotations generated by 3D-FAN-Full.
  • Reported visualizations include 2D-FAN and 3D-FAN fitting examples on LS3D-W, plus full 3D fitting examples on AFLW2000-3D.
Loading 1703.07332v3…