PaperlayerView exact revision on arXiv

Source-linked AI summary

KEPLER: Keypoint and Pose Estimation of Unconstrained Faces by Learning Efficient H-CNN Regressors

Amit Kumar, Azadeh Alavi, Rama Chellappa

arXiv:1702.05085v1cs.CV

TL;DR

Unconstrained faces may have variable landmark sets, challenging cascaded alignment methods that assume fixed points. KEPLER iteratively estimates landmarks while jointly predicting visibility and pose from 2D images. It reports stronger alignment than other state-of-the-art methods on AFLW and all but CCL on AFW.

  • Problem

    Occlusion and pose variation mean some unconstrained faces do not have a fixed set of 68 or 49 landmarks, leaving open whether cascaded shape regression can handle arbitrary faces without prior knowledge.

    Source: §I · p. 1

  • Method

    KEPLER iteratively predicts landmark increments and jointly estimates keypoint visibility and 3D pose from 2D color images, using a Channeled Inception network.

    Source: §I · p. 1, §III · p. 3

  • Results

    KEPLER outperforms all state-of-the-art methods on AFLW and all except CCL on AFW; its fifth iteration reduces AFLW error from 0.0369 to 0.0299 and AFW error from 0.0757 to 0.0301.

    Source: B. Results · pp. 6–7

  • Takeaways & Limitations

    The method provides 3D facial pose alongside keypoint localization, which the paper notes can be used for pose-dependent verification and 3D model generation.

    Source: §V · p. 7

    KEPLER is trained on AFLW's larger, generally square bounding boxes with the nose near the center, which the authors identify as a limitation for real-world deployment.

    Source: E. Iteration 5: Local Error Correction · p. 5

Abstract

from arXiv · show

Keypoint detection is one of the most important pre-processing steps in tasks such as face modeling, recognition and verification. In this paper, we present an iterative method for Keypoint Estimation and Pose prediction of unconstrained faces by Learning Efficient H-CNN Regressors (KEPLER) for addressing the face alignment problem. Recent state of the art methods have shown improvements in face keypoint detection by employing Convolution Neural Networks (CNNs). Although a simple feed forward neural network can learn the mapping between input and output spaces, it cannot learn the inherent structural dependencies. We present a novel architecture called H-CNN (Heatmap-CNN) which captures structured global and local features and thus favors accurate keypoint detecion. HCNN is jointly trained on the visibility, fiducials and 3D-pose of the face. As the iterations proceed, the error decreases making the gradients small and thus requiring efficient training of DCNNs to mitigate this. KEPLER performs global corrections in pose and fiducials for the first four iterations followed by local corrections in the subsequent stage. As a by-product, KEPLER also provides 3D pose (pitch, yaw and roll) of the face accurately. In this paper, we show that without using any 3D information, KEPLER outperforms state of the art methods for alignment on challenging datasets such as AFW and AFLW.

I. INTRODUCTION

The paper asks whether cascaded shape regression can handle arbitrary faces without prior knowledge of a fixed landmark count. It proposes an iterative CNN-based method that estimates landmarks while also predicting visibility and 3D pose.

I. INTRODUCTION

  • Keypoints move at subpixel level after the fourth iteration on AFLW, while the last correction stage mitigates bounding-box inconsistency across datasets in AFW.Source: §I · p. 1
  • Faces with occlusion or pose variation may not have a fixed set of 68 or 49 points, motivating alignment for arbitrary faces without prior knowledge.Source: §I · p. 1
  • The proposed Channeled Inception network pools and concatenates intermediate GoogLeNet features for use at every KEPLER stage.Source: §I · p. 1
  • KEPLER adapts cascaded regression into an iterative scheme that predicts increments jointly for all points while maintaining a shape constraint, inspired by fixed-point consolidation.Source: §I · pp. 1–2
  • As alignment error decreases across iterations, gradients become smaller, so the method uses different training policies for efficient H-CNN training.Source: §I · p. 2
  • The paper evaluates on AFLW and AFW, including a new, more challenging AFLW protocol that prior unconstrained-alignment evaluations usually omit.Source: §I · p. 2

II. RELATED WORK

Prior face-alignment work includes part-based models and regression-based methods, alongside approaches that fit 3D face models. KEPLER positions itself as regression-based work for unconstrained settings, jointly learning keypoints, visibility, and pose from 2D images.

II. RELATED WORK

  • Part-based methods model face parts or tree-structured shapes, while regression methods map image appearance to outputs; many earlier regressors were evaluated in lab settings or with visible keypoints.Source: §II · p. 2
  • Occlusion-robust regression, domain-dependent descent maps, and head-pose or domain-selective regressors were proposed to address challenges in cascaded alignment,,,.Source: §II · p. 2
  • 3D-model approaches estimate projection and shape parameters or fit dense face models, though optimizing base-shape coefficients and projection can be indirect for alignment.Source: §II · p. 2
  • KEPLER adapts cascade shape regression to unconstrained faces and jointly trains keypoint locations, visibility, and 3D pose using only 2D color images.Source: §II · p. 2
  • In the architecture overview, f predicts visibility, pose, and next-stage corrections, while h forms the input representation for the next iteration.Source: Figure 2 · p. 2

III. KEPLER

KEPLER iteratively renders current keypoint locations, predicts corrections, and updates the points. It jointly estimates keypoint locations, pose, and visibility using a weighted combination of task losses.

III. KEPLER

  • KEPLER cycles through rendering current keypoints, predicting corrections with a CNN, and adding those increments to update their locations.The rendering module stacks fixed-variance Gaussian maps centered on keypoints with the raw RGB image; the resulting input feeds the next iteration.Source: §III · pp. 2–3
  • The method trains a separate correction function for each of five iterations, with the final iteration addressing bounding-box differences across datasets.The initial rendering uses the mean shape.Source: §III · p. 3
  • Keypoint localization uses visibility-weighted error, while pose and keypoint-visibility confidence are estimated with Euclidean losses.The paper considers 21 facial keypoints, each associated with visibility.Source: §III · p. 3
  • The network combines the localization, pose, and visibility losses as a weighted linear combination, with weights selected by iteration.The combined objective is λL1 + µL2 + νL3.Source: §III · p. 3

A. Network Architecture

KEPLER uses a pruned GoogLeNet-based convolutional architecture that pools and combines intermediate features. It jointly trains for pose, visibility, and bounded keypoint error.

A. Network Architecture

  • The network prunes GoogLeNet after inception 4c and replaces ReLU with PReLU before pooling intermediate features.The architecture is used in every KEPLER stage.Source: A. Network Architecture · pp. 3–4
  • After stage two, the visible predicted points follow the shape of the input face, as illustrated against ground-truth points.The figure marks predictions in green and ground truth in red.Source: Figure 4 · p. 4
  • The Channeled Inception module pools alternate-layer features, convolves and concatenates their outputs, then reduces dimensionality with 1×1 convolution.Intermediate feature maps are adjusted to a common 7×7 size before concatenation.Source: A. Network Architecture · p. 4
  • Joint training predicts pose, keypoint visibility, and bounded correction to model their relationship to the correction needed for a point at a given pose.The network uses convolution layers through its final fully connected output layer.Source: A. Network Architecture · p. 4

B. Iteration 1 and 2: Constrained Training

The first two training stages bound keypoint corrections so the network can learn shifts progressively. Pose and visibility are learned jointly with keypoint localization.

B. Iteration 1 and 2: Constrained Training

  • The bounded correction is computed from the ground-truth error vector and its unit direction, scaled to the correction bound when necessary.The first stage is trained from scratch, and the second-stage function is trained similarly with weights initialized from the first.Source: B. Iteration 1 and 2: Constrained… · p. 4
  • For the first two stages, corrections are capped at 20 pixels, so the network learns the direction of each shift before its magnitude.The bound limits corrections when the ground truth is far from the current keypoint location.Source: B. Iteration 1 and 2: Constrained… · p. 4
  • Pose and each point’s visibility are learned jointly with keypoint corrections, because a one-shot location prediction is difficult in the nonlinear input space.The paper also notes that learning small corrections should be easier during initial training.Source: B. Iteration 1 and 2: Constrained… · p. 4

C. Iteration 3: Variant of Euclidean loss

After stage two, repeating the same training approach produced vanishing gradients, which the authors relate to the small dataset. They introduce a loss term intended to maintain gradients during learning.

C. Iteration 3: Variant of Euclidean loss

  • A small dataset led to vanishing gradients when few examples in a batch remained misaligned, making repeated stage-two training inadequate.The authors report this despite GoogLeNet’s claim to avoid vanishing gradients.Source: C. Iteration 3: Variant of Euclidean… · p. 4
  • The revised loss combines prediction error with an added term intended to preserve gradients while minimizing error.The added term is part of the objective, not a weight regularizer; γ controls its gradient strength.Source: C. Iteration 3: Variant of Euclidean… · pp. 4–5
  • For stage three, Gaussian channels are omitted for points whose predicted visibility falls below 0.03, and correction magnitude is unconstrained.The threshold and γ = 0.2 were selected by cross-validation.Source: C. Iteration 3: Variant of Euclidean… · p. 5

D. Iteration 4: Hard sample mining

Iteration 4 targets residual local misalignment by mining hard training examples after most faces are already aligned.

D. Iteration 4: Hard sample mining

E. Iteration 5: Local Error Correction

The optional fifth stage uses local patches to refine fiducials, addressing bounding-box differences across datasets.

E. Iteration 5: Local Error Correction

  • AFLW provides larger boxes than AFW, and box inconsistency across datasets makes the final local-correction stage optional and test-set dependent.The first four stages use full images and produce global corrections.Source: E. Iteration 5: Local Error Correction · p. 5
  • After iteration 4, patches centered on predicted points are fed to a network that produces locally corrected predictions.Source: Figure 6 · p. 5
  • The local network jointly predicts visibility and fiducial corrections; training both tasks performs better than correction-only training.It uses all patches regardless of visibility, while preventing gradient propagation for invisible points.Source: E. Iteration 5: Local Error Correction · p. 5
  • Pose is not predicted from local patches because the small patch context may lead the network to learn improper weights.Source: E. Iteration 5: Local Error Correction · p. 5

IV. EXPERIMENTS AND COMPARISON

Experiments evaluate KEPLER on AFLW and AFW, using AFLW for training and testing both datasets under challenging in-the-wild conditions.

A. Datasets

  • AFLW is used for training and as a main test set, while AFW is used as a main test set for diverse poses and expressions.Source: A. Datasets · p. 5
  • AFLW has 24,386 faces, yaw up to 120°, substantial occlusion, and at most 21 landmarks; AFW has 468 faces with yaw up to 90°.AFLW omits invisible-landmark coordinates, while AFW requires predicting occluded-point locations.Source: A. Datasets · p. 6
  • The AFLW-PIFA test uses 1,000 images split evenly across three yaw ranges, and the full AFLW test set also contains 1,000 images.Additional AFLW variants rotate and flip images; AFW evaluation follows and uses 341 faces taller than 150 pixels.Source: A. Datasets · p. 6
  • Landmark error is averaged over annotated points and summarized by mean error or cumulative error distribution; Fig. 7 plots AFLW landmark-localization CED curves.The figure legend reports average normalized mean error normalized by face size.Source: A. Datasets · p. 6, Figure 7 · p. 6

B. Results

KEPLER outperforms other state-of-the-art methods on AFLW and all but CCL on AFW; an additional local-correction iteration improves results, especially on AFW.

B. Results

  • KEPLER outperforms all state-of-the-art methods on AFLW and all except CCL on AFW.Source: B. Results · p. 6
  • KEPLER is farther from ground truth on invisible points and propagates no loss for them under the supplied AFLW annotations.CCL manually annotates 19 AFLW landmarks including invisible ones, whereas KEPLER uses the dataset annotations as provided.Source: B. Results · p. 7
  • After iteration 4, AFLW NME falls from 0.0369 to 0.0299 after iteration 5, while AFW NME falls from 0.0757 to 0.0301.The paper reports gains of 18% on AFLW and close to 60% on AFW; iteration 5 is optional and intended for differing bounding-box annotations.Source: B. Results · p. 7

V. CONCLUSIONS AND FUTURE WORKS

KEPLER combines improved initialization and multitask training with additional channels to capture facial structure, and the paper reports state-of-the-art performance for keypoint and head-pose prediction. It also produces 3D facial pose for potential use in pose-dependent verification and 3D model generation.

V. CONCLUSIONS AND FUTURE WORKS

  • KEPLER's improved initialization and multitask training outperform state-of-the-art methods for both keypoint prediction and head-pose estimation.The paper attributes precise unconstrained-face localization to efficiently capturing face structure through additional channels.Source: §V · p. 7
  • The resulting 3D facial pose can be used in pose-dependent verification methods and 3D model generation.Source: §V · p. 7
  • AFW landmark-localization curves report the fraction of test faces with average error below 5% of face size.Source: Figure 8 · p. 7
  • AFW pose-estimation curves report the percentage of faces labeled within a ±15° error tolerance.Source: Figure 9 · p. 7

Related papers

Loading 1702.05085v1…