PaperlayerView exact revision on arXiv

Source-linked AI summary

Face Alignment Across Large Poses: A 3D Solution

Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, Stan Z. Li

arXiv:1511.07212v1cs.CV

TL;DR

Face alignment methods designed for medium poses struggle with invisible landmarks, dramatic profile-view appearance changes, and difficult large-pose annotation. The paper introduces 3DDFA, which fits a dense 3D face model with cascaded CNNs and uses face profiling to synthesize training samples. Experiments report state-of-the-art performance across AFLW, AFLW2000-3D, and 300W, while the method’s cost design has optimization limitations.

  • Problem

    Large-pose face alignment up to ±90° remains difficult because landmark models assume visibility, appearance varies sharply near profile views, and occluded landmarks are hard to label.

  • Method

    3DDFA fits a dense 3D morphable face model with cascaded CNN regression, PNCC features, WPDC, and synthesized profile-view training samples.

  • Results

    3DDFA achieves state-of-the-art performance on AFLW, AFLW2000-3D, and 300W, while face profiling substantially improves large-pose alignment for multiple methods.

  • Takeaways & Limitations

    Dense 3D modelling and profile-view synthesis provide a supported approach for addressing self-occlusion, nonlinear fitting, and limited large-pose training data.

  • Takeaways & Limitations

    The vertex-distance cost has pathological curvature, making gradient-descent optimization slow and favoring expensive second-order methods.

Abstract

from arXiv · show

Face alignment, which fits a face model to an image and extracts the semantic meanings of facial pixels, has been an important topic in CV community. However, most algorithms are designed for faces in small to medium poses (below 45 degree), lacking the ability to align faces in large poses up to 90 degree. The challenges are three-fold: Firstly, the commonly used landmark-based face model assumes that all the landmarks are visible and is therefore not suitable for profile views. Secondly, the face appearance varies more dramatically across large poses, ranging from frontal view to profile view. Thirdly, labelling landmarks in large poses is extremely challenging since the invisible landmarks have to be guessed. In this paper, we propose a solution to the three problems in an new alignment framework, called 3D Dense Face Alignment (3DDFA), in which a dense 3D face model is fitted to the image via convolutional neutral network (CNN). We also propose a method to synthesize large-scale training samples in profile views to solve the third problem of data labelling. Experiments on the challenging AFLW database show that our approach achieves significant improvements over state-of-the-art methods.

1. Introduction

Face alignment works well for medium poses but remains difficult near ±90° because landmarks become invisible, appearance changes sharply, and large-pose landmark annotation is impractical. 3DDFA addresses these challenges by fitting a dense 3D face model with cascaded CNN regression and synthesizing profile-view training samples.

  • 1. Introduction: Traditional face alignment locates fiducial points for face normalization and supports tasks including face recognition, expression recognition, and inverse rendering.Existing methods include analysis-by-synthesis and regression-based approaches.
  • 1. Introduction: Large-pose alignment is challenging because invisible landmarks lack image evidence, facial appearance varies dramatically near profile views, and manual annotation requires guessing occluded landmarks.These limitations affect modelling, fitting, and data labelling respectively.
  • 1. Introduction: 3DDFA fits a dense 3D face model rather than a sparse landmark model, incorporating 3D information to address self-occlusion and appearance variation in large poses.The method is designed for alignment across poses up to ±90°.
  • 1. Introduction: A cascaded CNN regression method fits the 3D model using Projected Normalized Coordinate Code and Weighted Parameter Distance Cost.The paper presents this as the first CNN-based attempt to solve 3D face alignment.
  • 1. Introduction: Face profiling synthesizes 60k+ large-pose training samples from paired 2D images and 3D face models, improving both prior and proposed alignment algorithms.The database, profiling code, and 3DDFA code are released by the authors.

2. Related Works

Prior face-alignment methods include generative, regression, CNN, multi-view, and 3DMM-based approaches. Large-pose methods remain limited by sparse landmarks, view-specific computation, self-occlusion, or slow optimization.

  • 2. Related Works: 2D face alignment includes AAM, CLM, regression, and CNN methods, but many CNN approaches detect only sparse landmarks with limited descriptive power.Examples include CNNs that regress five landmark points or landmark response maps.
  • 2. Related Works: Multi-view large-pose methods use different landmark configurations or shape models for each view, but testing every view increases computation cost.TSPM and CDM are examples of this framework.
  • 2. Related Works: 3D face alignment fits a 3D morphable model from a 2D image and can cover arbitrary poses, but conventional fitting may require one minute per image.Regression-based 3DMM fitting improves efficiency but can still rely on self-occluded landmark features.
  • 2. Related Works: 3DMM-based regression remains non-pose-invariant when landmark features are self-occluded, while 2D and 3D alignment results can be mutually transformed through landmark coordinates and reconstruction.The latter connection distinguishes 3D face reconstruction from 3D face alignment.

3. 3D Dense Face Alignment (3DDFA)

3DDFA fits a 3D morphable face model to a single image using cascaded CNN regression, with PNCC providing pose-dependent coordinate features and WPDC weighting parameter importance.

  • 3.1. 3D Morphable Model: The model constructs and projects a 3D face through weak-perspective projection, producing 2D vertex locations from pose, shape, expression, scale, and translation parameters.The 3DMM combines a mean face with identity and expression bases before projection.
  • 3. 3D Dense Face Alignment (3DDFA): 3DDFA estimates a 3D morphable face model from one image using a unified cascaded CNN that predicts successive parameter updates.The parameter vector includes scale, pose, translation, identity shape, and expression components.
  • 3.3. Projected Normalized Coordinate Code: PNCC renders the current projected 3D face with normalized 3D coordinates as colors, then stacks this feature with the input image for CNN regression.Z-Buffer rendering automatically ignores invisible regions while encoding visible vertex locations and feeding back the current estimate.
  • 3.4. Cost Functions: PDC treats parameter dimensions equally despite unequal influence on the fitted face, while VDC better reflects fitting error but suffers pathological curvature and slow zig-zagging optimization.The paper motivates WPDC as a simpler alternative that avoids VDC’s derivative and optimization difficulties.
  • 3.4.3 Weighted Parameter Distance Cost (WPDC): WPDC weights parameter errors by their induced alignment effects, prioritizing parameters whose perturbations produce larger fitted-face errors.The weighting uses the alignment error caused by deteriorating one parameter while holding the others at ground truth.

4. Face Profiling

Face profiling estimates depth and rotates reconstructed faces in 3D to synthesize realistic large-pose training images without filling invisible regions.

  • 4.1. 3D Image Meshing: 3D meshing fits a face model on the face region and estimates external-region depth using anchors, producing a depth image for rotation.The face-region fit uses MFF with ground-truth landmarks as constraints; external depth follows Zhu et al.'s meshing method.
  • 4.2. 3D Image Rotation: Depth images are rotated in 3D space to generate appearances at larger yaw angles, while self-occluded regions expand rather than require artificial filling.This avoids the artifacts associated with filling invisible regions in face frontalization.

5. Implementation Details

The implementation combats cascade overfitting by regenerating initializations from validation samples with similar face posture and refines sparse landmarks after dense alignment.

  • 5.1. Initialization Regeneration: Initialization regeneration addresses CNN overfitting by perturbing training samples according to similar face postures and their characteristic fitting errors.Profile errors may reflect yaw perturbations, while open-mouth errors may reflect expression parameters.
  • 5.1. Initialization Regeneration: For each training sample, a validation subset of similar-posture samples is constructed and used to regenerate the initial parameter at each cascade iteration.The validation samples are selected separately from training data.
  • 5.2. Landmark Refinement: The dense model fits all face vertices, while HOG-based linear regression can refine sparse landmark locations after 3DDFA.The dense alignment can also be combined with other 2D face-alignment methods.

6. Experiments

Experiments evaluate 3DDFA across medium-, large-, and 3D-face alignment using 300W, AFLW, and AFLW2000-3D, with profiling, initialization regeneration, and WPDC analyzed separately.

  • 6.1. Datasets: The experiments use 300W, AFLW, and AFLW2000-3D, with 300W-LP providing 122,450 flipped and profiled training samples.AFLW contains 21,080 faces spanning yaw from −90° to 90°; AFLW2000-3D enables evaluation using visible and invisible landmarks.
  • 6.2. Performance Analysis: Initialization regeneration keeps testing error descending across cascade iterations, whereas ordinary cascades converge rapidly after two iterations.The comparison uses 300W-LP training and testing splits without identity overlap.
  • 6.2. Performance Analysis: WPDC performs best among the tested costs because it explicitly models parameter priorities and adaptively optimizes their weights.PDC converges to an unsatisfactory result, while VDC is limited by pathological curvature.
  • 6.3.1 Large Pose Face Alignment in AFLW: All methods benefit substantially from face profiling, while 3DDFA achieves state-of-the-art performance among 2D methods beyond medium poses and the lowest pose-variation standard deviation.In [60°, 90°], profiling improves RCPR by 44.06%, ESR by 40.36%, and SDM by 42.10%; SDM refinement further improves 3DDFA.
  • 6.3.2 3D Face Alignment: Evaluating all landmarks reveals reduced [60°, 90°] performance and standard deviation compared with visible-landmark evaluation, supporting all-landmark evaluation for 3D alignment.AFLW2000-3D uses all 68 MultiPIE landmarks and balanced pose subsets for evaluation.
  • 6.3.3 Medium Pose Face Alignment: 3DDFA remains competitive on the common set and achieves state-of-the-art performance on the challenging set in medium-pose alignment.The experiments compare performance across large-pose AFLW, 3D AFLW2000-3D, and medium-pose 300W tasks.

7. Conclusions

3DDFA addresses large-pose face alignment by fitting a dense 3D morphable model with cascaded CNN and synthesizing profile-view training samples, achieving state-of-the-art performance on AFLW, AFLW2000-3D, and 300W.

  • 3DDFA fits a dense 3D morphable model with cascaded CNN to address self-occlusion and nonlinear fitting in large poses.
  • A face profiling algorithm synthesizes profile-view appearances to provide abundant training samples.
  • Experiments show state-of-the-art performance on AFLW, AFLW2000-3D, and 300W.
Loading 1511.07212v1…