PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Fast, Accurate and Stable 3D Dense Face Alignment

Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, Stan Z. Li

arXiv:2009.09960v2cs.CV

TL;DR

Existing 3D dense face alignment methods emphasize accuracy, leaving speed and video stability insufficiently addressed for practical applications. 3DDFA-V2 uses lightweight 3DMM regression with meta-joint optimization and 3D-aided short-video synthesis. It reports a balance of speed, accuracy, and stability, including over 50fps on one CPU core and the best overall performance across four datasets.

  • Problem

    Existing 3D dense face alignment methods mainly concentrate on accuracy, while video stability is often ignored and public 3D alignment video databases are unavailable.

  • Method

    3DDFA-V2 combines lightweight 3DMM-parameter regression, meta-joint optimization with WPDC and VDC, landmark-regression regularization, and 3D-aided short-video synthesis.

  • Results

    Over 50fps on a single CPU core and over 130fps on multiple CPU cores are reported, with overall performance described as best across four datasets.

  • Takeaways & Limitations

    The framework supports real-time 3D dense face alignment in practical use while pursuing fast, accurate, and stable reconstruction simultaneously.

Abstract

from arXiv · show

Existing methods of 3D dense face alignment mainly concentrate on accuracy, thus limiting the scope of their practical applications. In this paper, we propose a novel regression framework named 3DDFA-V2 which makes a balance among speed, accuracy and stability. Firstly, on the basis of a lightweight backbone, we propose a meta-joint optimization strategy to dynamically regress a small set of 3DMM parameters, which greatly enhances speed and accuracy simultaneously. To further improve the stability on videos, we present a virtual synthesis method to transform one still image to a short-video which incorporates in-plane and out-of-plane face moving. On the premise of high accuracy and stability, 3DDFA-V2 runs at over 50fps on a single CPU core and outperforms other state-of-the-art heavy models simultaneously. Experiments on several challenging datasets validate the efficiency of our method. Pre-trained models and code are available at https://github.com/cleardusk/3DDFA_V2.

1 Introduction

3D dense face alignment must balance accuracy with practical speed and temporal stability. 3DDFA-V2 addresses these demands by regressing 3DMM parameters with meta-joint optimization and synthesizing short videos for stable predictions.

  • Motivation: Dense-vertex methods achieve state-of-the-art performance but rely on heavy networks that are slow and memory-consuming during inference.Pruning PRNet by 77.5% channels increased error from 3.62% to 5.24%.
  • Motivation: 3DMM parameters are low-dimensional and suitable for lightweight regression, but their unequal influence makes dynamic parameter weighting during training necessary.Cascaded regression progressively updates parameters but increases computation linearly with the number of stages.
  • Framework: 3DDFA-V2 combines a fast backbone, WPDC and VDC losses, meta-joint optimization, and landmark regularization to target CPU real-time speed and state-of-the-art performance.Meta-joint optimization looks ahead on meta-train batches and selects the better loss using a meta-test batch.
  • Video stability: 3D aided short-video synthesis transforms one still image into adjacent frames with in-plane and out-of-plane motion, enabling training for temporal consistency.The method supplies temporal information without requiring a public 3D face-alignment video database.
  • Overview: 7.2ms is the reported multi-core runtime, while single-core inference exceeds 50fps and multi-core inference exceeds 130fps.The framework is described as fast, accurate, and stable across extensive experiments on four datasets.

2 Methodology

3DDFA-V2 combines lightweight 3DMM regression, meta-joint loss optimization, landmark-regression regularization, and synthetic short-video training to improve speed, accuracy, and video stability.

  • 3DMM regression: The framework uses a lightweight backbone to regress a compact 3DMM parameter vector instead of directly predicting dense vertices.The regression target uses a similarity transformation matrix with the first 40 shape and 10 expression dimensions.
  • Meta-joint optimization: fWPDC reconstructs dense vertices once rather than 62 times, reducing a 128-sample batch cost from 41.7ms to 3.6ms while preserving outputs.This fast implementation decomposes weight calculation into similarity-transformation and shape-expression components.
  • Meta-joint optimization: Meta-joint optimization looks ahead with fWPDC or VDC and selects the lower-error update, converging faster and reaching lower error than vanilla-joint optimization.VDC is difficult to optimize from scratch, whereas fWPDC may undertrain the network in the late stage.
  • 3D aided short-video-synthesis: 3D aided short-video synthesis transforms still images into smoothly varying adjacent frames using in-plane similarity changes and progressively increased yaw and pitch.The strategy addresses video jittering when public 3D dense face-alignment video databases are unavailable.

3 Experiments

Experiments evaluate datasets, accuracy, stability, ablations, and computational efficiency. 3DDFA-V2 improves alignment results while maintaining substantially lower inference cost and video stability.

  • Datasets and Evaluation Protocols: Five datasets and established NME protocols evaluate sparse, dense, video-based alignment, accuracy, stability, and runtime.Experiments use AFLW, AFLW2000-3D, Florence, Menpo-3D, and 300W-LP-related data.
  • Ablation Study: Meta-joint optimization lowers mean NME from 4.04% to 3.73% on AFLW2000-3D and from 5.10% to 4.64% on AFLW versus fWPDC.It also outperforms vanilla-joint optimization, with k = 100 performing best in the ablation.
  • Ablation Study: Landmark-regression regularization improves mean NME by 0.15% and 0.26% over fWPDC, and by 0.14% on both datasets over meta-joint.On AFLW2000-3D, it reaches 3.59% versus 3.71% for prior landmark constraints.
  • Ablation Study: Short-video synthesis improves still-frame NME from 1.86% to 1.71% and stability from 0.52% to 0.48%.Randomly applying in-plane and out-of-plane rotations performs worse, at 1.76% / 0.50%.
  • Evaluations of Accuracy and Stability: 3DDFA-V2 achieves 3.51% versus PRNet’s 3.62% on AFLW2000-3D and 4.43% versus 3DDFA-TPAMI’s 4.55% on AFLW.It uses 3.27M parameters and takes 6.2ms on CPU for sparse alignment.
  • Evaluations of Accuracy and Stability: 3DDFA-V2 reconstructs 45K dense vertices in 1ms on CPU and surpasses PRNet in both video accuracy and stability at lower computation cost.It runs in 7.2ms on multiple CPU cores and 19.2ms on a single core, exceeding 50fps.
  • Ablation Study: The selector uses fWPDC early and VDC late for both k = 100 and k = 200.This observed progression explains the behavior of the meta-joint optimization.

4 Conclusion

3DDFA-V2 jointly targets fast, accurate, and stable 3D dense face alignment through optimization, regularization, and short-video synthesis. Experiments support the effectiveness and efficiency of these methods.

  • The framework combines fast fWPDC, meta-joint optimization, landmark-regression regularization, and 3D aided short-video synthesis.
  • The meta-joint optimization combines fWPDC and VDC to alleviate parameter-regression optimization difficulties.
  • Landmark-regression regularization promotes performance to the state of the art.
  • 3D aided short-video synthesis improves stability on videos by supporting consecutive-frame adjustment.
  • The reported results demonstrate the effectiveness and efficiency of the proposed methods.

A. Checkerboard Artifacts

The figure compares dense-mesh outputs from PRNet, VRN, and 3DDFA-V2, including full overlays and enlarged local details. The enlarged details show smoother output from 3DDFA-V2 and artifacts in the other methods.

  • The upper row overlays each dense mesh with the original image, while the bottom row enlarges local details.
  • PRNet’s output mesh is jagged and contains checkerboard artifacts in enlarged local details.
  • VRN also exhibits slight checkerboard artifacts in the enlarged local details.
  • 3DDFA-V2 produces the smoothest result among the displayed outputs.

B. Impact of Dimension Reduction

The heatmap evaluates NME across combinations of shape and expression parameter dimensions. Reducing the dimensions to [40, 10] produces an approximately 0.4% NME increase, described as acceptable.

  • The heatmap contains 29×199 NME (%) configurations, with shape dimension on the x-axis and expression dimension on the y-axis.
  • Setting the shape and expression dimensions to [40, 10] yields an NME increase of about 0.4%.
  • The approximately 0.4% NME increase at [40, 10] is considered acceptable.

C. Implementation Details

Training uses PyTorch with 120×120 normalized face crops and SGD optimization. The 3DDFA-V2 configuration uses k = 100 for meta-joint optimization and synthesizes eight perturbed frames per still image.

  • Faces are cropped and resized to 120×120, then normalized by subtracting 127.5 and dividing by 128.
  • Training uses SGD with batch size B = 128, weight decay 0.0005, and momentum 0.9.
  • For 3DDFA-V2 (M+R+S), the meta-joint optimization uses k = 100.
  • Short-video synthesis creates n = 8 frames per still image with scale perturbations ∆s ∈[0.95, 1.05].

D. Generalization and Scaling-up Ability

The proposed methods generalize across backbones and support scaling through different channel multipliers and input sizes, with performance and speed evaluated across several datasets.

  • D. Generalization and Scaling-up Ability: Table 5 and Fig. 10 compare performance and speed across architectures and scaling-up options.The evaluations cover AFLW2000-3D, AFLW and Menpo-3D, using different channel numbers and backbones.
  • D. Generalization and Scaling-up Ability: ResNet-22 improves the proposed methods’ performance, demonstrating generalization across architectures.
  • D. Generalization and Scaling-up Ability: Table 5 ignores the 1ms CPU reconstruction time for 3D dense vertices when reporting performance and speed.

E. Qualitative Results

Qualitative comparisons are presented on AFLW2000-3D and AFLW, with Fig. 10 depicting the computation-complexity versus NME trade-off and lower NME indicating better performance.

  • E. Qualitative Results: Qualitative results compare the proposed method with VRN and PRNet on AFLW2000-3D and AFLW.
  • E. Qualitative Results: Fig. 10 plots computation complexity in MACs against NME (%) on AFLW2000-3D and AFLW.Lower NME (%) is better.

F. Acceleration with ONNX Runtime5

The ONNX runtime further accelerates 3DDFA-V2 inference, with single-image timings reported for one and four CPU cores.

  • F. Acceleration with ONNX Runtime5: 4.4ms inference is achieved on a single CPU core with ONNX runtime and a single image as input.The measurement uses an i5-8259U CPU @ 2.30GHz.
  • F. Acceleration with ONNX Runtime5: 1.35ms inference is achieved on four CPU cores with ONNX runtime and a single image as input.The measurement uses an i5-8259U CPU @ 2.30GHz.
  • F. Acceleration with ONNX Runtime5: Qualitative results are shown for the MobileNet (M+R+S) model, which runs at over 50fps on a single CPU core.
Loading 2009.09960v2…