PaperlayerView exact revision on arXiv

Source-linked AI summary

Faster Than Real-time Facial Alignment: A 3D Spatial Transformer Network Approach in Unconstrained Poses

Chandrasekhar Bhagavatula, Chenchen Zhu, Khoa Luu, Marios Savvides

arXiv:1707.05653v2cs.CV

TL;DR

Off-angle facial occlusion commonly affects landmark localization, and handling wide pose changes requires 3D facial structural information. The approach uses camera projection and parametric nonlinear warping of a mean shape in an end-to-end framework where alignment and 3D modeling are codependent. The method reports more accurate, consistent landmarks than other 2D and 3D methods, state-of-the-art performance over other 3D approaches, and faster-than-real-time processing.

  • Problem

    Off-angle facial occlusion commonly affects landmark localization, and handling wide pose changes requires 3D facial structural information.

    Source: §1 · p. 1

  • Method

    The approach uses camera projection and parametric nonlinear warping of a mean shape in an end-to-end framework where alignment and 3D modeling are codependent.

    Source: §1.1 · p. 2, §3.1 · p. 3

  • Results

    The method reports more accurate, consistent landmarks than other 2D and 3D methods, state-of-the-art performance over other 3D approaches, and faster-than-real-time processing.

    Source: §1.1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The approach targets consistent alignment, 3D model generation, and frontalization across changing poses in images from the wild.

    Source: §1.1 · p. 2

Abstract

from arXiv · show

Facial alignment involves finding a set of landmark points on an image with a known semantic meaning. However, this semantic meaning of landmark points is often lost in 2D approaches where landmarks are either moved to visible boundaries or ignored as the pose of the face changes. In order to extract consistent alignment points across large poses, the 3D structure of the face must be considered in the alignment step. However, extracting a 3D structure from a single 2D image usually requires alignment in the first place. We present our novel approach to simultaneously extract the 3D shape of the face and the semantically consistent 2D alignment through a 3D Spatial Transformer Network (3DSTN) to model both the camera projection matrix and the warping parameters of a 3D model. By utilizing a generic 3D model and a Thin Plate Spline (TPS) warping function, we are able to generate subject specific 3D shapes without the need for a large 3D shape basis. In addition, our proposed network can be trained in an end-to-end framework on entirely synthetic data from the 300W-LP dataset. Unlike other 3D methods, our approach only requires one pass through the network resulting in a faster than real-time alignment. Evaluations of our model on the Annotated Facial Landmarks in the Wild (AFLW) and AFLW2000-3D datasets show our method achieves state-of-the-art performance over other 3D approaches to alignment.

1. Introduction

Facial alignment in the wild must contend with occlusion and pose variation, while 3D modeling approaches can be computationally complex. The paper couples alignment and 3D modeling to preserve landmark semantics across poses and process images faster than real time.

1. Introduction

  • Off-angle faces commonly occlude landmarks, and pose variation makes 3D structural information important for robust face analysis.The introduction also notes computation time and complexity among drawbacks of existing 3D modeling schemes.Source: §1 · p. 1

1.1. Our Contributions in this Work

  • A mean 3D shape with nonlinear warping is proposed to model unseen faces despite the limited size of many 3D-face datasets.The method uses alignment on the input image to warp the mean shape.Source: §1.1 · p. 2
  • An end-to-end framework couples alignment and 3D modeling, keeping landmarks semantically consistent across changing poses.The paper links this consistency to more consistent 3D model generation and frontalization on images in the wild.Source: §1.1 · p. 2
  • A single network pass enables faster-than-real-time processing, with state-of-the-art performance over other 2D and 3D alignment approaches.Source: §1.1 · p. 2

2. Related Work

The related-work review surveys face-alignment methods and CNN-based 3D modeling. It contrasts 2D visibility problems and limited 3D morphable-model coverage with the paper’s use of TPS to generate new 3D shapes.

2. Related Work

  • The review focuses on recent CNN approaches to face alignment, especially methods using 3D for robust alignment.Source: §2 · p. 2

2.1. Face Alignment Methods

  • Alignment work includes feature-space regression, region-specific cascaded regressors, group-sparse 3D pose models, CNN-based 3D fitting, and geometric constraints.Other approaches discussed include nonlinear statistical models and view-based methods, which can be impractical for real-time use or problematic between viewpoint modes.Source: §2.1 · p. 2

2.2. CNNs for 3D Object Modeling

  • CNN depth estimation can produce relatively flat face estimates, while face-specific 3D modeling has also been used for frontalization but requires input landmarks.Source: §2.2 · pp. 2–3
  • 2D landmarking can shift points toward visible boundaries, while 3DMM methods depend on a limited shape basis that may not fit outlier faces or unseen expressions.The paper proposes TPS warping of mean shapes to fit the input and generate new 3D shapes.Source: §2.2 · p. 3

3. 3D Spatial Transformer Networks

The 3D Spatial Transformer estimates camera projection and 3D shape-warp parameters from image features, linking a 3D face model to its 2D projection. Its differentiable modules support projection, landmark refinement, and rebuilding a 3D model after 2D landmark regression.

3. 3D Spatial Transformer Networks

  • The network estimates camera projection and 3D-warp parameters from image features, enabling a 3D model to be projected and textured while identifying visible vertices.The camera parameters also provide face pose, allowing occluded vertices to be excluded from texturing.Source: §3 · p. 3

3.1. Camera Projection Transformers

  • The camera projection module maps homogeneous 3D world points to 2D camera coordinates using an 11-parameter projection matrix, with gradients propagated through the scale-fixed matrix.The matrix relates pairs of 2D and 3D points, whose gradients are accumulated to update the estimated parameters.Source: §3.1 · pp. 3–4

3.2. 3D Thin Plate Spline Transformers

  • A 3D TPS warps a generic face model through smooth, parameterized coordinate changes defined by control points and destination points.The module uses fixed generic-model vertices and control points, with separate functions for changes in x, y, and z.Source: §3.2 · pp. 4–5

3.3. Warped Camera Projection Transformers

  • The warped camera projection module propagates gradients through the projection to the warped 3D coordinates as well as the camera parameters.It computes coordinate gradients for each 3D point independently for backpropagation.Source: §3.3 · p. 5

3.4. 2D Landmark Regression

  • Landmark refinement samples bilinearly interpolated features at projected initial points and uses a separate fully connected offset regressor for each landmark.Independent regressors allow landmarks with the same initial location or feature vector to move differently.Source: §3.4 · pp. 5–6

3.5. 3D Model Regression From 2D Landmarks

  • After 2D regression breaks the 3D-to-2D mapping, the method backprojects each landmark through the estimated camera, selects a nearby 3D point, and TPS-warps the model to restore correspondence.The resulting model projects to the regressed landmarks and can be retextured; the authors also describe expression neutralization as a qualitative future-work avenue.Source: §3.5 · p. 6

4. Experiments

Experiments compare alignment on AFLW and AFLW2000-3D, with VGG-16 leading most pose ranges. Timing tests report per-image throughput for both network architectures and 3DDFA.

4.1. Datasets

  • 300W-LP contains 122,450 synthetically generated face views and is useful for training 3D approaches; AFLW has approximately 25,000 faces with 21 landmarks, while AFLW2000-3D contains AFLW’s first 2,000 images with 68 points.AFLW2000-3D includes non-visible landmarks, whose true locations are unknown because the dataset is synthetic.Source: §4.1 · pp. 6–7

4.2. Implementation Details

  • Differentiable TPS, camera-projection, and bilinear-sampler modules enable end-to-end training; AlexNet’s first layer and VGG-16’s first four layers are frozen.Landmark regression uses Scaling, Reduction, and Bias layers, with only the new layers updated during training.Source: §4.2 · p. 7

4.3. Training on 300W-LP

  • Training used AFW, HELEN, and LFPW from 300W-LP, held out IBUG for validation, and normalized faces to 250 x 250 pixels using CMS-RCNN bounding boxes.AlexNet trained for 100,000 iterations with batch size 50; VGG-16 trained for 200,000 iterations with batch size 25.Source: §4.3 · p. 7

4.4. Ablation Experiments

  • VGG-16 outperformed AlexNet in all three AFLW pose ranges before landmark regression, and the regression stage greatly improved accuracy.The ablations used models trained on 300W-LP and tested on detected AFLW images.Source: §4.4 · p. 7, Table 1 · p. 7

4.5. Comparison Experiments

  • On AFLW, VGG-16 achieved better accuracy in all pose ranges; on AFLW2000-3D it ranked first in [0°, 30°] and (60°, 90°], and second by a small amount in (30°, 60°].Table 2 reports NME (%) for both datasets; the authors also report smaller error standard deviation on AFLW and more consistent landmarks on AFLW2000-3D.Source: §4.5 · pp. 7–8, Table 2 · p. 8

4.6. Running Speed

  • AlexNet processed 1200 faces at approximately 170 faces per second and VGG-16 at approximately 52 faces per second, versus 3DDFA’s 75.72 ms per image.The models were evaluated one image at a time on a CPU and GPU.Source: §4.6 · p. 8

5. Conclusions

TPS warping lets a 3D Spatial Transformer Network generate a face model and accurate 2D landmarks across large pose variation. The approach addresses limits of 3D morphable models and reports more accurate, consistent landmarks than other 2D and 3D methods.

  • TPS warping generates a 3D face model and accurate 2D landmarks across large pose variation.Source: §5 · p. 8
  • Unlike a 3D morphable model limited by available training shapes, TPS can model unseen faces through regression of 2D landmarks.Source: §5 · p. 8
  • The approach produces more accurate and consistent landmarks than other 2D and 3D methods.Source: §5 · p. 8

Related papers

Loading 1707.05653v2…