PaperlayerView exact revision on arXiv

Source-linked AI summary

Face Alignment in Full Pose Range: A 3D Total Solution

Xiangyu Zhu, Xiaoming Liu, Zhen Lei, Stan Z. Li

arXiv:1804.01005v1cs.CV

TL;DR

Existing face alignment methods are largely designed for poses below 45° and struggle with invisible landmarks, severe appearance variation, and limited large-pose labels. 3DDFA fits a dense 3DMM with cascaded CNNs and synthesizes profile-view training data, achieving state-of-the-art performance on AFLW, AFLW2000-3D, and 300W.

  • Problem

    Face alignment methods designed for medium poses struggle with self-occluded landmarks, frontal-to-profile appearance changes, and scarce labelled data at yaw angles up to ±90°.

  • Method

    3DDFA fits a dense 3DMM using cascaded CNNs with PNCC, PAF, and OWPDC, while 3D profiling synthesizes large-pose training samples.

  • Results

    The experiments report state-of-the-art performance on AFLW, AFLW2000-3D, and 300W.

  • Takeaways & Limitations

    3DMM fitting provides pose-free alignment by handling self-occluded landmarks through 3D visibility and supports alignment across the full pose range.

  • Takeaways & Limitations

    The Vertex Distance Cost has pathological curvature, so gradient descent can converge slowly and second-order optimization is expensive and difficult to implement on GPUs.

Abstract

from arXiv · show

Face alignment, which fits a face model to an image and extracts the semantic meanings of facial pixels, has been an important topic in the computer vision community. However, most algorithms are designed for faces in small to medium poses (yaw angle is smaller than 45 degrees), which lack the ability to align faces in large poses up to 90 degrees. The challenges are three-fold. Firstly, the commonly used landmark face model assumes that all the landmarks are visible and is therefore not suitable for large poses. Secondly, the face appearance varies more drastically across large poses, from the frontal view to the profile view. Thirdly, labelling landmarks in large poses is extremely challenging since the invisible landmarks have to be guessed. In this paper, we propose to tackle these three challenges in an new alignment framework termed 3D Dense Face Alignment (3DDFA), in which a dense 3D Morphable Model (3DMM) is fitted to the image via Cascaded Convolutional Neural Networks. We also utilize 3D information to synthesize face images in profile views to provide abundant samples for training. Experiments on the challenging AFLW database show that the proposed approach achieves significant improvements over the state-of-the-art methods.

1 INTRODUCTION

Large-pose face alignment challenges conventional 2D landmark models because landmarks become invisible, appearance changes sharply across views, and suitable labelled data is scarce. 3DDFA addresses these issues by fitting a dense 3DMM with cascaded CNNs and synthesizing profile-view training samples.

  • Existing alignment methods generally assume yaw below 45° and fully visible landmarks, limiting their applicability to poses up to 90°.
  • Modelling: Self-occlusion in large poses can remove landmarks’ detectable appearance and semantic meaning, causing 2D shape models to fail.
  • Fitting: Appearance changes from frontal to profile views are difficult for unified regression models, while testing multiple view-specific models increases computational cost.
  • Training Data: Large-pose landmark annotation is tedious because invisible landmarks must be guessed, leaving public datasets mostly restricted to medium poses or visible landmarks.
  • 3DDFA fits a dense 3DMM using cascaded CNNs, with PNCC, PAF, and OWPDC supporting pose variation and parameter-priority modelling.
  • 3D face profiling synthesizes 60k+ large-pose training samples that simulate profile appearances and boost alignment performance.

2 RELATED WORKS

Related work progresses from sparse 2D landmark and appearance models toward regression-based and 3DMM-based alignment. 3DMMs address self-occlusion, but regression-based 3D fitting still faces pose-invariant features and unequal parameter priorities.

  • Sparse 2D subspace models represent facial shape with fiducial points but struggle with nonlinear out-of-plane rotations across broad pose ranges.
  • Model Fitting: Template fitting depends on whether input image patterns fall within the appearance model’s described variations, limiting robustness in unconstrained environments.
  • Regression Methods: Cascaded regression updates shape parameters iteratively using feedback features, progressively reducing alignment error through cascaded weak regressors.
  • Large Pose Face Alignment: Large-pose 2D methods use multiple views or visibility estimation, but view testing is costly and occlusion estimation remains challenging.
  • 3D Face Alignment: 3DMMs inherently provide model-point visibility, while regression-based 3D fitting still faces self-occluded features and unequal parameter priorities.
  • Large Pose Face Alignment: The paper combines 3DMM, cascaded regression, and CNNs to address large-pose face alignment.

3 3D DENSE FACE ALIGNMENT (3DDFA)

The 3DDFA framework combines cascaded CNN regression with pose-dependent image features and a training cost function. Its four components define the parameter objective, image features, network, and training cost.

  • Cascaded CNN realizes 3DDFA by applying CNNs as regressors within the cascaded regression formulation.
  • The framework contains regression objective p, image features Fea, CNN structure Net, and the training cost function.

3.1 3D Morphable Model

3DDFA represents faces with a 3D Morphable Model whose vertices are constructed, projected, and fitted using pose parameters. Quaternions replace Euler angles to avoid rotation ambiguity near profile views.

  • 3D Morphable Model: 3DMM represents a face using mean shape, identity components, and expression components learned from 3D scans.The model is projected with scale orthographic projection to obtain 2D vertex positions.
  • 3D Morphable Model: The model parameter vector contains scale, rotation, 2D translation, identity coefficients, and expression coefficients.The original parameterization uses p = [f, R, t2d, αid, αexp]T.
  • Rotation Formulation: Euler angles become ambiguous at 90° yaw, where different angle settings can produce the same rotation matrix and profile appearance.This ambiguity is illustrated by two rotations with Euler angles [20°, 90°, 0°] and [0°, 90°, 20°].
  • Rotation Formulation: 3DDFA uses a four-dimensional quaternion instead of Euler angles to eliminate this rotation ambiguity during fitting.The quaternion is represented as [q0, q1, q2, q3].
  • Rotation Formulation: The final fitting parameter vector incorporates quaternion rotation, 2D translation, identity coefficients, and expression coefficients, with scale merged into the quaternion.The implementation divides the quaternion by √f and does not constrain it to be unit.

3.2 Feature Design

3DDFA combines image-view and model-view features to represent both the original image and pose-aligned facial information. PNCC encodes projected 3D coordinates, while PAC produces pose-adaptive features from semantically consistent anchors.

  • Feature Design: Face-alignment input features are divided into image-view features that retain image context and model-view features that align pixels with the current face model.The paper combines both categories through PNCC and PAF.
  • Pose Adaptive Convolution: PAC samples 64 × 64 semantically consistent anchors on a cylindrical face representation using constant azimuth and height intervals.The anchors are intended to support convolution at consistent facial locations across poses.
  • Pose Adaptive Convolution: PAC projects the anchors, crops d × d patches, concatenates them by cylindrical coordinates, and applies d × d convolutions with stride d.The resulting 64 × 64 response maps are pose-adaptive features.
  • Pose Adaptive Convolution: Responses at self-occluded vertices are divided by two, producing Pose Adaptive Features without eliminating occluded features.Vertices whose normals point toward minus z are treated as self-occluded.
  • Projected Normalized Coordinate Code: NCC normalizes each 3D mean-face coordinate between 0 and 1, creating a smooth three-channel vertex index.The x, y, and z channels can be displayed as RGB texture channels.
  • Projected Normalized Coordinate Code: PNCC renders the projected 3D face with NCC colors using a Z-Buffer and stacks the result with the input image before CNN processing.PNCC is therefore an image-view feature that preserves the original image input.

3.3 Network Structure

3DDFA uses a cascaded, unified two-stream CNN to iteratively update 3DMM parameters. One stream processes the image with PNCC, while the other processes image features with pose adaptive convolution and anchors.

  • Cascaded Network: At cascade iteration k, the network constructs PNCC and PAF from the current parameter pk and predicts an update ∆pk.The updated parameter becomes the input to the next network in the cascade.
  • Two-Stream Fusion: The outputs of the two streams are merged with an additional fully connected layer to predict the parameter update.Each subsequent cascade network has the same structure but different weights.
  • PNCC Stream: The PNCC stream stacks a 200 × 200 × 3 color image with a 200 × 200 × 3 PNCC input.This stream contains five convolutional layers, four pooling layers, and one fully connected layer.
  • PAF Stream: The PAF stream processes the 200 × 200 × 3 color image and 64 × 64 feature anchors with pose adaptive convolution.It then uses pooling, convolutional, and fully connected layers.

3.4 Cost Function

The cost function must account for unequal parameter effects and their changing priorities during 3DMM fitting. The paper progresses from parameter distance to vertex distance, WPDC, and OWPDC to improve optimization behavior.

  • Parameter Distance Cost (PDC): PDC treats all parameter dimensions equally, although deviations in pose can cause larger alignment errors than deviations in shape parameters.This mismatch between parameter error and alignment error can produce sub-optimal fitting results.
  • Vertex Distance Cost (VDC): VDC minimizes distances between the current and ground-truth 3D face vertices, explicitly reflecting parameter semantics in fitting error.Its objective compares V(p0 + ∆p) with V(pg).
  • Vertex Distance Cost (VDC): VDC is non-convex and has pathological curvature, so gradient descent converges slowly through zig-zagging while second-order methods are expensive and difficult to implement on GPUs.Pose-parameter directions have much higher curvature than PCA-coefficient directions.
  • Weighted Parameter Distance Cost (WPDC): WPDC weights parameter errors by their estimated importance, prioritizing parameters such as rotation and translation before less important parameters.The weights shrink as high-impact parameters approach the ground truth, allowing subsequent optimization of lower-priority parameters.
  • Weighted Parameter Distance Cost (WPDC): WPDC models importance but not priority: in the open-mouth near-profile example, pose should be estimated before expression, yet WPDC assigns both high weights.Restricting the CNN to pose parameters produces a better fitting result in Fig. 6.
  • Optimized Weighted Parameter Distance Cost (OWPDC): OWPDC models between-parameter priority by jointly optimizing the weights, using a constrained optimization whose computation is reduced to a p-dimensional quadratic program.The optimization uses the weight vector, CNN update, current predicted parameter, and element-wise bounds described in the text.

4 FACE PROFILING

Face profiling synthesizes large-pose training images by fitting a 3D face and rotating the triangulated image while preserving background structure. Rotations are increased in 5° steps up to 90° without filling newly occluded regions.

  • 4.1 3D Image Meshing: 3D meshing fits a 3DMM on the face and simulates external-region depth using anchors, then triangulates the whole image.The resulting representation includes both the fitted face and surrounding image geometry.
  • 4.2 3D Image Rotation: Anchor adjustment preserves source anchor offsets after rotation while keeping face-contour anchors fixed.The adjusted anchors are obtained from a least-squares solution of the edge constraints.
  • 4.2 3D Image Rotation: 5° rotations expand each image's yaw through 90°, generating large-pose samples without filling the increasingly self-occluded region.The process also augments the dataset to a large scale.
  • 4.2 3D Image Rotation: Face profiling changes the original yaw by 20°, 30°, and 40° to produce alternate 2D and 3D views.The profiling sequence is illustrated relative to the original yaw angle.

5 IMPLEMENTATION

The implementation combats cascade overfitting and initialization sensitivity through regenerated training starts and data augmentation. Testing initializes a frontal mean 3D face and scales it to the detected face box.

  • Training Strategy: Each cascade iteration regenerates initial parameters using nearest validation samples with similar face posture.The selected validation sample is not used for training.
  • Initialization: Training data are augmented tenfold with in-plane rotations up to 30° and perturbed bounding boxes.Bounding-box perturbations are sampled from differences between ground-truth and automatically detected rectangles.
  • Initialization: Testing initializes a frontal 3D mean face, centers it in the bounding box, and scales it to enclose the face region.The initial identity and expression parameters are zero, and the quaternion is [1, 0, 0, 0].

6 EXPERIMENTS

Experiments evaluate 3DDFA across large-pose, 3D, and medium-pose alignment using synthesized and real-world datasets. Ablations show complementary input features, the value of convolvability and convergence, and improved optimization from OWPDC.

  • 6.1 Databases and Protocol: The experiments use 300W-LP, AFLW, and AFLW2000-3D, with NME as the alignment-accuracy measure.300W-LP supplies synthesized large-pose training data, while AFLW and AFLW2000-3D evaluate large-pose and 3D alignment.
  • 6.2 Performance with Different Input Features: Combining PNCC and PAF performs better than either feature alone, suggesting complementary coverage of facial context and facial features.PNCC improves relative to PAF individually as pose becomes larger, while PAF assists feature fitting through implicit frontalization.
  • 6.3 Convolvable Property: Violating PNCC's convolvable requirement reduces performance because nonsmooth vertex-index inputs make learning more difficult.PIndex preserves pixel semantics but is difficult for local convolution to interpret.
  • 6.4 Convergence Property: PAF outperforms TM by over 6%, supporting the importance of the convergence property.TM cannot distinguish whether the projected model fully occludes the image face, whereas PAF retains discriminative contour context.
  • 6.5 Performance with Different Cost: OWPDC models parameter importance and priority, producing faster convergence and the best performance among the evaluated cost functions.PDC and VDC are less effective, while WPDC improves by modeling parameter importance alone.

6.5 Error Reduction in Cascade

Across alignment tasks, initialization regeneration extends error reduction through the cascade, while 3DDFA remains robust as pose increases and when invisible landmarks are evaluated. Its reported advantage is strongest for full 3D alignment, though failures remain under difficult visual conditions.

  • 6.5 Error Reduction in Cascade: Initialization regeneration keeps testing error descending across iterations, leading the authors to select three iterations for 3DDFA.Without regeneration, training and testing errors converge quickly after two iterations.
  • 6.6.1 Large Pose Face Alignment on AFLW: On AFLW, all methods improve by over 40% in the [60°, 90°] yaw range after face profiling.This compares models trained on original versus profiled data for large-pose alignment.
  • 6.6.1 Large Pose Face Alignment on AFLW: As yaw increases, most 2D methods degrade while 3DDFA maintains performance and achieves state-of-the-art results beyond medium poses.The minimum standard deviation also indicates robustness to pose variations.
  • 6.6.2 3D Face Alignment in AFLW2000-3D: 30.33% is 3DDFA's improvement over the best 2D method DCN on AFLW2000-3D, compared with 26.49% on AFLW.AFLW2000-3D evaluates all 68 landmarks, including invisible ones, and reports larger standard deviations than AFLW.
  • 6.6.3 Medium Pose Face Alignment on 300W: 3DDFA remains competitive on medium-pose 300W and shows the best robustness to detected-box initialization among the compared methods.The initialization result is based on the performance drop when replacing ground-truth boxes with detected boxes.
  • 6.6.4 Failure Cases: Typical 3DDFA failures involve shadow or occlusion, extreme pose or expression, extreme illumination, and limited 3DMM nose-shape variation.These cases identify visual and model-capacity boundaries of the approach.

7 CONCLUSIONS

The paper presents a pose-free face alignment approach that fits a dense 3D Morphable Model using cascaded CNN regression, achieving state-of-the-art performance on three benchmarks.

  • The method fits a dense 3D Morphable Model for pose-free face alignment.
  • PNCC and PAF input features support cascaded CNN regression of model parameters.
  • The OWPDC cost function accounts for parameter priorities during fitting.
  • Face profiling synthesizes profile-view appearances to provide abundant training samples.
  • Experiments show state-of-the-art performance on AFLW, AFLW2000-3D, and 300W.
Loading 1804.01005v1…