Source-linked AI summary
Deep Deformation Network for Object Landmark Localization
Xiang Yu, Feng Zhou, Manmohan Chandraker
TL;DR
Landmark localization is difficult under non-rigid shape, appearance, and occlusion variation. DDN combines a low-rank shape-basis initialization with local thin-plate-spline refinement in a CNN cascade. It reports state-of-the-art accuracy across face, human-body, and bird benchmarks using the same architecture.
Problem
Landmark localization must handle non-rigid shape deformation, appearance variation, and occlusions across object types such as faces, bodies, and birds.
Source: §1 · p. 1Method
DDN cascades a low-rank shape basis network for global initialization with a point transformer network that applies local thin-plate-spline deformations.
Source: §1 · p. 2Results
DDN reports state-of-the-art accuracy across face, human-body, and bird benchmarks using the same CNN architecture and training mechanisms.
Source: §1 · p. 3Takeaways & Limitations
The same architecture covers three object types while learning part geometries without hard-coded part connections.
Source: §1 · pp. 2–3The low-rank truncation makes SBN insufficient for localization on its own.
Source: §3.2 · p. 7
Abstract
from arXiv · showhide
We propose a novel cascaded framework, namely deep deformation network (DDN), for localizing landmarks in non-rigid objects. The hallmarks of DDN are its incorporation of geometric constraints within a convolutional neural network (CNN) framework, ease and efficiency of training, as well as generality of application. A novel shape basis network (SBN) forms the first stage of the cascade, whereby landmarks are initialized by combining the benefits of CNN features and a learned shape basis to reduce the complexity of the highly nonlinear pose manifold. In the second stage, a point transformer network (PTN) estimates local deformation parameterized as thin-plate spline transformation for a finer refinement. Our framework does not incorporate either handcrafted features or part connectivity, which enables an end-to-end shape prediction pipeline during both training and testing. In contrast to prior cascaded networks for landmark localization that learn a mapping from feature space to landmark locations, we demonstrate that the regularization induced through geometric priors in the DDN makes it easier to train, yet produces superior results. The efficacy and generality of the architecture is demonstrated through state-of-the-art performances on several benchmarks for multiple tasks such as facial landmark localization, human body pose estimation and bird part localization.
1 Introduction
DDN addresses landmark localization under non-rigid shape, appearance, and occlusion variation with a two-stage cascade that embeds geometric constraints in a CNN. It reports state-of-the-art accuracy across faces, human bodies, and birds using the same CNN architecture.
1 Introduction
- Landmark localization remains challenging under non-rigid shape deformation, appearance variation, and occlusion across faces, bodies, and birds.Examples include facial expression and skin-tone changes, body articulation, and bird pose variation.Source: §1 · p. 1
- Prior cascaded CNN regressors face careful initialization and design, complex training, and a lack of learned geometric relationships.They map CNN features directly to landmark locations.Source: §1 · p. 2, Figure 1 · p. 2
- The cascade incorporates geometric structure at coarse and fine levels rather than learning direct feature-to-landmark regressors, and avoids hard-coded part connections.The authors describe the geometric priors as regularizing each stage and the framework as easier to train and test.Source: §1 · p. 2
- DDN uses a shape basis network for low-rank global shape initialization, then a point transformer network for local non-rigid refinement with thin-plate splines.The PTN uses the SBN output as its initialization and maps initialized landmarks to final positions.Source: §1 · pp. 2–3
- State-of-the-art accuracy is reported across face, human-body, and bird benchmarks using the same CNN architecture and training mechanisms.The framework is presented as general across three object types.Source: §1 · p. 3
2 Related Work
Related work covers domain-specific approaches to facial landmarks, human pose, and bird parts, alongside more general pose estimation and CNN-based transformation learning. The paper contrasts DDN with these methods on initialization, part relationships, and local versus global alignment.
Facial landmark localization
- Facial landmark methods range from PCA-based shape models and appearance-based variants to regression and cascaded CNN approaches.The paper contrasts DDN with facial cascades that require careful stage design or initialization.Source: §2 · p. 3
Human body pose estimation
- Human pose work spans pictorial structures, learned part relations, poselets, and deep networks; DDN is presented as easier to train and more efficient to evaluate.The cited deep approaches include cascaded CNN regressors, CNN-plus-graphical-model training, and dual-source networks.Source: §2 · p. 4
Bird part localization
- Bird part localization includes probabilistic models, symbiotic segmentation, exemplar methods, and CNN-based approaches such as part-based R-CNN and Deep LAC.Source: §2 · p. 4
General pose estimation; Learning transformations with CNNs
- General pose methods include regression frameworks sensitive to initialization, while CNN transformation work motivates spatial constraints and non-rigid alignment.DDN uses the SBN for global alignment and the PTN for finer local deformation, unlike WarpNet’s alignment role as described here.Source: §2 · p. 4
3 Proposed Method
DDN cascades a shape-basis initialization and TPS-based point refinement, placing geometric constraints within a CNN landmark-localization pipeline. Its feature extractor, learned low-rank shape representation, and control-point regularization address feature sensitivity, pose complexity, and TPS overfitting.
3 Proposed Method
- DDN combines convolutional features, an SBN landmark initialization, and a PTN that refines the landmarks using TPS transformations.The three components are trained end-to-end, with separate pretraining used for ease of training.Source: §3 · pp. 4–5, Figure 2 · p. 5
3.1 Convolutional Feature Extraction
- Four truncated VGG-16 stages produce 512-channel 7 × 7 maps; pooling is removed, and using three stages performs worse.The network uses 100 × 100 inputs for faces and 200 × 200 inputs for body and bird pose estimation.Source: §3.1 · p. 5
3.2 Shape Basis Network
- The SBN predicts a shape on the training samples’ low-dimensional manifold by combining CNN features with a mean shape and top-k orthogonal PCA bases.A nonlinear mapping produces basis weights, with k chosen to preserve 99% of training-shape covariance energy; the low-rank representation is insufficient for localization alone.Source: §3.2 · pp. 6–7
3.3 Point Transformer Network
- Control-point regularization adds synthesized landmarks to the TPS loss, helping reduce overfitting and produce more stable warps.The PTN deforms the SBN initialization toward ground-truth landmarks; the reported settings are γ = 1 and ϕ = ψ = 0.4.Source: §3.3 · pp. 7, 9
4 Implementation Details
DDN’s components are pretrained separately before joint training, and the full network can be trained end-to-end. Training images and landmarks are synthetically augmented.
4 Implementation Details
- The convolutional layers, SBN, and PTN can be trained end-to-end, but the authors pretrain SBN and PTN separately before joint training.Training images and landmarks are randomly cropped, rotated, and flipped; momentum is 0.9 and weight decay is 0.004.Source: §4 · p. 9
- SBN pretraining first updates its fully connected layers with VGG-16 convolutional weights fixed, then updates both after the error stops decreasing for 10 epochs.PTN pretraining uses the SBN-pretrained convolutional weights, initially training only its fully connected layers and then training both sets after 10 epochs.Source: §4 · p. 9
- After pretraining, SBN supplies shape input to PTN; joint training first updates PTN with SBN fixed, then updates the whole network jointly.The loss is evaluated at PTN’s output and propagated back through the network.Source: §4 · pp. 9–10
5 Experiments
The experiments evaluate DDN on facial landmarks, human body pose, and bird parts using PCK and facial normalized error. The reported results include improvements over prior methods, with different transformation and component comparisons across tasks.
5 Experiments
- The evaluation covers three landmark tasks, with PCK for faces, bodies, and birds and task-specific runtime measurements.Reported runtimes are 1.3 ± 0.5 ms for faces, 4.3 ± 3.6 ms for human body parts, and 7.6 ± 2.4 ms for bird parts on a Tesla K80 GPU.Source: §5 · p. 10
5.1 Facial Landmark Localization
- DDN outperforms the five reported state-of-the-art methods on 300-W, although its improvement over cGPRT is moderate.The authors also note that handcrafted SIFT features are competitive with CNN features for face alignment.Source: §5.1 · pp. 10–11
- On face datasets, the combined DDN consistently outperforms SBN and PTN evaluated independently, while TPS-based DDN outperforms its affine variant.SBN alone localizes poorly and PTN is limited to in-plane transformations.Source: §5.1 · p. 11, Table 2 · p. 11
5.2 Human Body Pose Estimation
- 10.9% better mean accuracy over all landmarks than the best reported result, with DDN improving over CNN-based methods on LSP.Figure 5 compares PCK across thresholds; Table 3 reports landmark PCK at α = 0.2, where DDN performs better on almost all articulated joints.Source: Figure 5 · p. 12, §5.2 · p. 12, Table 3 · p. 12
- For human pose, TPS brings a larger accuracy improvement over affine transformations than it does for faces; DDN degrades on highly occluded or folded body parts.The authors relate the larger TPS gain to greater non-rigidity in human pose.Source: §5.2 · pp. 12–13
5.3 Bird Part Localization
- On CUB200-2011, DDN achieves highly competitive and sometimes better performance than Zhang et al. at significantly lower computational cost.The comparison reports PCK for each of the 15 landmarks at α = 0.02, 0.05, 0.08, and 0.10.Source: §5.3 · pp. 13–14, Table 4 · p. 13
6 Conclusion
DDN combines low-rank shape initialization with local TPS deformation in an end-to-end cascade. The paper reports state-of-the-art results across face, human pose, and bird landmark localization.
6 Conclusion
- DDN uses a Shape Basis Network to initialize landmarks on a low-rank manifold, then a Point Transformer Network to refine them with non-rigid thin-plate spline warps.The end-to-end framework avoids handcrafted features and pre-specified part connectivities and achieves state-of-the-art results on face, human pose, and bird landmark tasks.Source: §6 · p. 14