PaperlayerView exact revision on arXiv

Source-linked AI summary

DenseReg: Fully Convolutional Dense Shape Regression In-the-Wild

Riza Alp Guler, Yuxiang Zhou, George Trigeorgis, Epameinondas Antonakos, Patrick Snape, Stefanos Zafeiriou, Iasonas Kokkinos

arXiv:1803.02188v2cs.CV

TL;DR

DenseReg addresses dense 2D–3D correspondence for deformable objects in natural images. It learns deformation-free U-V coordinates with a fully convolutional, quantized-regression network and uses them to improve related localization and pose-estimation systems, while exposing limits from landmark marching and label granularity.

  • Problem

    Dense correspondence between deformable object surfaces and natural images requires a dense representation beyond regions or sparse landmarks.

  • Method

    DenseReg fits a 3D template using in-the-wild landmarks, then trains a fully convolutional network with quantized regression to predict per-pixel U-V coordinates.

  • Results

    DenseReg improves facial landmark localization and articulated pose estimation, including through cascaded architectures, dense supervision, and deformable-model initialization.

  • Takeaways & Limitations

    Dense correspondence serves as a shared representation for geometrically defined tasks and as privileged training information for pose-estimation CNNs.

  • Takeaways & Limitations

    Landmark marching can produce incorrect correspondences, especially for jaw landmarks affected by head-pose changes; quantization also reduces training data as label granularity increases.

Abstract

from arXiv · show

In this work we use deep learning to establish dense correspondences between a 3D object model and an image "in the wild". We introduce "DenseReg", a fully-convolutional neural network (F-CNN) that densely regresses at every foreground pixel a pair of U-V template coordinates in a single feedforward pass. To train DenseReg we construct a supervision signal by combining 3D deformable model fitting and 2D landmark annotations. We define the regression task in terms of the intrinsic, U-V coordinates of a 3D deformable model that is brought into correspondence with image instances at training time. A host of other object-related tasks (e.g. part segmentation, landmark localization) are shown to be by-products of this task, and to largely improve thanks to its introduction. We obtain highly-accurate regression results by combining ideas from semantic segmentation with regression networks, yielding a 'quantized regression' architecture that first obtains a quantized estimate of position through classification, and refines it through regression of the residual. We show that such networks can boost the performance of existing state-of-the-art systems for pose estimation. Firstly, we show that our system can serve as an initialization for Statistical Deformable Models, as well as an element of cascaded architectures that jointly localize landmarks and estimate dense correspondences. We also show that the obtained dense correspondence can act as a source of 'privileged information' that complements and extends the pure landmark-level annotations, accelerating and improving the training of pose estimation networks. We report state-of-the-art performance on the challenging 300W benchmark for facial landmark localization and on the MPII and LSP datasets for human pose estimation.

1 Introduction

DenseReg extends CNNs to dense 2D–3D surface correspondence by regressing deformation-free template coordinates throughout an image. The resulting representation supports multiple tasks and improves landmark localization and articulated pose estimation through cascades, initialization, and dense supervision.

  • Dense shape regression: DenseReg uses a discriminatively trained, fully convolutional CNN to establish dense correspondence between image pixels and deformation-free 3D surface coordinates.The target surface is represented in a 2D U-V parameterization of a deformable template.
  • Dense shape regression: Manual in-the-wild landmarks are combined with 3D template fitting to construct dense U-V correspondence fields for network supervision.The fitted template supplies correspondence from image pixels to the surface parameterization.
  • Applications: Cascading and dense supervision are complementary, yielding faster and improved convergence in the combined training approach.Dense correspondence acts as an auxiliary supervision signal or privileged information for pose-estimation CNNs.
  • Quantized regression: The work introduces quantized regression, combining semantic-segmentation-style classification with residual regression for dense shape coordinates.The contribution is positioned as a carefully designed fully convolutional shape-regression system.
  • Applications: Template coordinates transfer multiple annotations from a shared template system, allowing one network to address landmark localization, segmentation, and related object tasks.The paper presents dense shape regression as a common representation for geometrically defined problems.
  • Applications: Dense correspondence improves articulated pose estimation and facial landmark localization when used in cascaded architectures that jointly regress correspondences and sparse landmarks.The paper reports improved localization in both articulated body pose estimation and facial landmark localization.

2 Previous work

Prior work includes CNNs that model transformations and statistical deformable models that explicitly recover dense shape correspondence. DenseReg is motivated by combining CNN robustness to natural-image variation with deformable models’ representation of shape detail.

  • Motivation: The paper combines CNNs’ handling of image variation with statistical deformable models’ explicit capture of deformation details.The authors frame these complementary strengths as the starting point for DenseReg.
  • CNNs and shape: CNN research has enriched deep networks with explicit similarity or non-rigid deformation models for classification, fine-grained recognition, and face detection.These approaches model deformation within deep-learning systems rather than treating it solely as nuisance variation.
  • Statistical deformable models: Statistical deformable models recover dense correspondence between a deformation-free template and an observed image, preserving shape information for analysis and invariant decisions.Examples include Active Appearance Models and 3D Morphable Models.
  • Bridging paradigms: Earlier detection and deformable-model approaches bridge the two paradigms by using shape information as priors on relative part positions in pose estimation.This shape prior is represented through energy terms or probabilistic graphical models.

3 From Statistical Deformable Models to Network Supervision

The paper converts deformable-model fitting and landmark annotations into dense, deformation-free image-to-template supervision, then represents correspondence as UV fields for network regression.

  • Deformable templates factor category variation into shape-related deformations and appearance differences such as facial hair, skin color, and illumination.
  • Rather than learning a linear generative deformation model, the method treats image-to-template correspondence as a vector field regressed by the network.
  • The template is a 3D mesh whose vertices are mapped bijectively to a canonical two-dimensional U-V space.
  • 3.1 Supervision for the face template: Facial landmark annotations in the wild guide template fitting, allowing deformation-free coordinates from visible template vertices to transfer into image space as regression targets.
  • The resulting supervision encodes correspondence through template-derived UV coordinates transferred to image-domain pixels.
  • 3.2 Supervision for the human body template: For human bodies, the surface is partitioned into 25 planar patches, each assigned normalized deformation-free Index, U, and V fields.

4 Quantized Regression

DenseReg predicts dense U–V surface correspondences by combining classification-based quantization with region-specific residual regression. This design improves correspondence accuracy over plain regression and supports downstream pose and shape tasks.

  • DenseReg estimates U–V coordinates at face-region pixels while identifying non-face pixels through a dummy output.The network predicts horizontal and vertical correspondence components and distinguishes foreground from background.
  • Quantized regression first classifies each coordinate into a coarse bin, then refines it using a residual regressor specific to the selected region.The refined residual is added to the quantized prediction to produce a smooth correspondence field.
  • Facial template coordinates are uniformly quantized into K horizontal and vertical bins, balancing correspondence precision against training-data scarcity and label complexity.Larger K can reduce quantization error, but increasingly granular labels provide less training data and increase classification complexity.
  • The network uses two K-way classification branches for U and V and region-specific residual regressors, reducing complexity by separating the coordinate components.For K distinct U and V bins, classification uses two K-way classifiers, while regression uses 2K one-dimensional units.
  • 64?

5 Experiments

DenseReg is evaluated across face and body tasks using fitted template supervision, multiple CNN architectures, and an end-to-end cascade for joint correspondence and pose estimation.

  • DenseReg is evaluated on semantic segmentation, facial landmark localization, deformable tracking, human-body correspondence, and human-ear landmark localization.
  • Face training uses 3DDFA projection and 3DMM parameters to rasterize per-pixel ground-truth UV coordinates from the Basel + FaceWarehouse model.
  • Body supervision uses the Unite the People dataset, whose images are automatically fitted with the SMPL model containing dense shape and articulation components.
  • Pose experiments use MPII and LSP, comprising approximately 18k MPII training images and 7k testing images, with LSP fine-tuning on an extended 11k-image training set.
  • The experiments use ResNet101 with dilated convolutions and an Hourglass-type network, while the cascade stacks two hourglasses and applies heatmap regression to landmarks or body joints.The cascade balances dense-regression and heatmap losses during training.

5.2 Semantic Segmentation

DenseReg transfers template annotations to images for facial segmentation and landmark localization, outperforming comparison systems on the reported 300W evaluation.

  • Semantic Segmentation: Template-defined labeling functions can be transferred to image pixels using regressed coordinates, enabling facial-part segmentation from one template mask.
  • Semantic Segmentation: The facial template segmentation mask contains eight classes: eyes, eyebrows, upper and lower lips, nose, and other.
  • Semantic Segmentation: DenseReg outperforms DeepLab-v2 on Helen-testset IoU evaluation, with substantial improvements reported for several parts including eyebrows and lips.DeepLab-v2 uses the same ResNet-101 architecture and is trained specifically for semantic segmentation.
  • Landmark Localization: DenseReg estimates facial landmarks by locating image points whose deformation-free coordinates are closest to landmark coordinates defined on the template.
  • Landmark Localization: 2.67% failure rate: DenseReg cascade achieves a new 300W test-set performance record and largely outperforms the other tested methods.The 300W evaluation reports AUC and failure rate for 68-point landmark localization.

5.4 Deformable Tracking

DenseReg is tested for deformable face tracking on the 300VW benchmark, including severe occlusions and extreme illumination, and its cascaded variants outperform the competition winner.

  • The 300VW test set contains 114 videos and approximately 218k frames captured under arbitrary conditions, including severe occlusions and extreme illumination.
  • Tracking is evaluated from sparse landmark points using DenseReg, DenseReg + MDM, and DenseReg + Cascade against 300VW challenge participants.
  • DenseReg + Cascade and DenseReg + MDM outperform the winner of the 300VW competition on AUC and Failure Rate.The authors note that their approach was not fine-tuned for video and generally lacks temporal modeling used by competing methods.

5.5 Dense Correspondence for the Human Body

For human bodies, DenseReg provides dense correspondence and is evaluated through qualitative correspondence results and a cascaded articulated-pose architecture on MPII and LSP.

  • Because no prior dense 3D-human-model-to-image results exist, the paper demonstrates correspondence performance visually on a held-out partition of the Unite the People dataset.
  • DenseReg cascade is quantitatively evaluated for articulated body pose estimation on MPII and LSP.
  • MPII and LSP training images are augmented through cropping, flipping, rotation, and rescaling before model training.
  • Qualitative results show joint localization in extreme viewpoints, challenging poses, occlusions, self-occlusions, and ambiguities, alongside dense correspondence estimates.

5.6 Ear Landmark Localization

The paper extends dense correspondence to human ears and evaluates landmark localization using deformation-free ear coordinates and 55-point annotations. Qualitative examples and supplementary quantitative evaluation are provided for this experiment.

  • Data and correspondence construction: 602 ear images with sparse landmark annotations were brought into dense correspondence using Thin Plate Splines because no 3D ear model was available.The resulting deformation-free space supported the landmark-localization procedure and qualitative coordinate visualizations.
  • Evaluation: The ear experiment localized landmarks using the same procedure described for the paper’s main landmark experiments.Quantitative results were reported using AUC and failure rate based on normalized RMS point-to-point error.
  • Results and visualization: Supplementary material provides the ear landmark localization AUC values and additional qualitative comparisons between regressed and ground-truth deformation-free coordinates.These examples are illustrated in Fig. 13, while the ear template space is visualized in Fig. 14.
  • Evaluation: Table 6 reports 55-point human-ear landmark localization accuracy using AUC and the failure rate of the cumulative error distribution.The error is normalized by the RMS point-to-point distance.

6 Conclusion

The conclusion presents fully convolutional regression as a general approach for mapping natural-image objects to 3D template coordinates. It connects dense shape regression with articulated pose estimation and reports applications to faces, bodies, and ears.

  • Conclusion: The proposed fully convolutional method establishes dense correspondence fields between natural images and three-dimensional object templates.The paper focuses on face and body shapes while also evaluating human ears.
  • Conclusion: Dense correspondence is used for geometrically represented tasks, including face-part segmentation, facial landmark localization, and articulated body pose estimation.The conclusion describes dense shape regression and articulated pose estimation as unified within an end-to-end trainable architecture.
  • Conclusion: The method outperforms a state-of-the-art semantic segmentation approach for face-part segmentation when used out-of-the-box.As an initialization for Statistical Deformable Models, it achieves state-of-the-art results on the 300W landmark localization challenge.
  • Conclusion: Experiments on human body and human ear shapes demonstrate the generality of the correspondence method.The conclusion frames the approach as readily usable for face-related tasks and integrable into other correspondence problems.

A Network Structure for Hourglass-type DenseReg

The hourglass DenseReg network uses convolution, down-sampling, and up-sampling modules, with an inception-v2 replacement for the original convolution block. This modification improves training speed and slightly improves body pose estimation.

  • Architecture: The hourglass architecture consists of convolution, down-sampling, and up-sampling modules.These modules are assembled into the modified hourglass network used for DenseReg.
  • Architecture: The original three-layer convolution module is replaced by an inception-v2-type module.The change is presented as a modification to the hourglass architecture.
  • Effect of the modification: The inception-v2 modification yields a slight improvement in body pose estimation and an obvious improvement in training speed.The reported gains concern both task performance and optimization efficiency.
Loading 1803.02188v2…