Source-linked AI summary
Unsupervised Learning of Object Landmarks through Conditional Image Generation
Tomas Jakab, Ankush Gupta, Hakan Bilen, Andrea Vedaldi
TL;DR
The paper addresses learning object landmarks without manual annotations. It uses conditional image generation with a tight geometry bottleneck to combine source appearance and target geometry, and reports strong performance across faces, people, 3D objects, and digits.
Problem
The paper asks how to learn detectors for object landmarks without external annotations across varied visual object categories.
Method
The method reconstructs a target image from source appearance and target geometry encoded through a bottleneck that distils landmark-like spatial features.
Results
The method outperforms previous unsupervised and supervised facial-landmark methods and extends to people, 3D objects, and digits.
Takeaways & Limitations
Conditional generation can induce semantically consistent landmark detectors without manual supervision across synthetic deformations, videos, and diverse datasets.
Takeaways & Limitations
The generator uses an off-the-shelf network pre-trained by self-supervised grayscale-image colorisation for its perceptual loss.
Abstract
from arXiv · showhide
We propose a method for learning landmark detectors for visual objects (such as the eyes and the nose in a face) without any manual supervision. We cast this as the problem of generating images that combine the appearance of the object as seen in a first example image with the geometry of the object as seen in a second example image, where the two examples differ by a viewpoint change and/or an object deformation. In order to factorize appearance and geometry, we introduce a tight bottleneck in the geometry-extraction process that selects and distils geometry-related features. Compared to standard image generation problems, which often use generative adversarial networks, our generation task is conditioned on both appearance and geometry and thus is significantly less ambiguous, to the point that adopting a simple perceptual loss formulation is sufficient. We demonstrate that our approach can learn object landmarks from synthetic image deformations or videos, all without manual supervision, while outperforming state-of-the-art unsupervised landmark detectors. We further show that our method is applicable to a large variety of datasets - faces, people, 3D objects, and digits - without any modifications.
1 Introduction
The paper learns object landmarks without annotations by reconstructing a target image from source appearance and target geometry distilled through a tight bottleneck. Conditioning on both images reduces ambiguity enough for perceptual-loss training, enabling broad application without correspondence or optical-flow information.
- Motivation: The paper targets unsupervised landmark detectors for object categories including faces and human bodies.Examples include facial eyes, nose, and mouth, and human hands, shoulders, and head.
- Approach: Training uses source-target image pairs differing by viewpoint or deformation, obtained from videos or synthetic perturbations.The target supplies changing geometry, while appearance is obtained from the source image.
- Approach: A tight geometry bottleneck distils the target into spatial landmark information before the generator reconstructs the target from source appearance and encoded geometry.The architecture extracts heatmaps, estimates keypoint coordinates, renders 2D Gaussians, and combines them with source-image features.
- Contribution: The formulation works on more complex data, including highly articulated human bodies, and does not require correspondences, optical flow, or known transformations.It can learn from both synthetically generated deformations and raw videos.
- Approach: Conditioning generation on source and target images makes the task less ambiguous, allowing direct perceptual-loss optimization instead of GAN-based training.The paper reports that this still produces excellent image generation and semantically consistent landmarks without manual supervision.
2 Related work
Prior work learns object structure with equivariance, auto-encoding, deformation fields, or adversarial training, often relying on known correspondences or additional constraints. This paper instead shapes a conditional-generation bottleneck to resemble landmark-detector outputs without adversarial training.
- Landmark learning: Earlier landmark methods use equivariance and distinctiveness, but rely on correspondences from optical flow or synthetic transformations and cannot directly leverage video data.Equivariance is described as orthogonal to this paper’s approach and potentially usable as an additional cue.
- Representation learning: Representation-learning approaches impose latent structure through auto-encoders, GANs, or conditional encoders.The paper places its method in the conditional-encoder family rather than using an auto-encoder.
- Video generation: Related video-generation methods factor pose and identity or reconstruct and extrapolate frames using variational, recurrent, or adversarial architectures.These approaches include pose embeddings, fixed-length sequence representations, and spatio-temporal video GANs.
- Positioning: This paper explicitly shapes its bottleneck to resemble landmark-detector output and avoids adversarial training.That design distinguishes it from related conditional-generation and pose-factorization methods described in the section.
- Object structure: Concurrent object-structure methods learn appearance templates, deformation fields, or 3D keypoints, commonly using embedding controls, known transformations, or equivariance.One related landmark-generation method generates an image from itself using transported features and additionally requires equivariance to learn geometry.
3 Method
The method learns landmarks by extracting a tightly constrained geometric representation from a target image and using it with a source image to reconstruct the target. Its bottleneck converts heatmaps into Gaussian landmark representations, while conditional generation is trained with perceptual reconstruction loss.
- Conditional image generation: The model jointly learns an encoder Φ and generator Ψ so the source image and distilled target representation reconstruct the target image.The reconstruction objective minimizes expected loss between x′ and Ψ(x, Φ(x′)).
- Heatmaps bottleneck: The encoder produces K heatmaps, applies spatial Softmax, and condenses each heatmap into a landmark location using its spatial expected value.Each heatmap is then replaced by a Gaussian-like function centered at the extracted location.
- Qualitative visualization: Figure 2 visualizes source, target, reconstruction, landmark locations, and heatmaps for synthetic transformations and VoxCeleb video frames.The figure includes selected landmarks and corresponding score maps, where brighter pixels indicate higher confidence.
- Heatmaps bottleneck: The resulting tensor contains Gaussian heatmaps encoding the locations of K maxima while remaining equivalent to 2D landmark coordinates.The heatmap form is intended to provide a more useful input to the generator than coordinates alone.
- Heatmaps bottleneck: The bottleneck retains very little information from the input to avoid degenerate identity solutions, while Gaussian heatmaps preserve a generator-usable representation.The expected-value step is especially important for restricting retained information.
- Generator network using a perceptual loss: The generator uses a perceptual reconstruction loss comparing deep-network activations from reference and generated images rather than only raw pixels.The implementation combines reconstruction errors from multiple VGG-19 layers and uses an ℓ2 norm.
4 Experiments
Experiments evaluate the method across faces, human bodies, 3D objects, and digits using synthetic deformations or videos. The learned landmarks track semantic structure, achieve strong facial performance with few labels, and exhibit robustness to several nuisance factors, while retaining stated limitations.
- Experimental setup: The common architecture is evaluated on faces, human bodies, 3D objects, and digits using synthetic image transformations or video-derived pairs.CelebA uses random thin-plate-spline warps, while VoxCeleb supplies pairs of frames from the same video.
- Faces: Facial landmarks consistently track features across deformation and identity changes, while accurately reconstructing the target image.Qualitative examples include landmarks tracking the lower chin and the region between the eyes.
- Faces: At K = 30, errors are 2.58% on MAFL and 6.31% on AFLW, outperforming supervised and unsupervised methods under the reported comparison.The method also reports 2.58% versus 3.16% for Zhang et al. on MAFL and 2.58% versus 8.42% when their synthetic warps are removed.
- Human body: Human-body landmarks track actors across identities and poses, but the model cannot distinguish frontal from dorsal body sides.The stated explanation is weak visual cues and the absence of explicit consistency constraints.
- Human body: On BBC-Pose, the method trails the top two supervised methods but outperforms methods using 10k samples while using 1k training samples.Accuracy is measured as the percentage of points within a specified pixel distance d; performance is particularly good for elbows.
- 3D objects: On SmallNORB, stable keypoints are robust to illumination and elevation, invariant to azimuth changes up to ±80°, and localize similar regions across shape changes.The method fails to generalise beyond ±80° azimuth changes.
- Appearance and geometry: On SVHN, AFLW, and Human3.6M, the generator preserves geometry while substituting appearance, including for unrelated image pairs unseen during training.A reported example re-renders the geometry of digit 3 with the appearance of digit 5.
5 Conclusions
The method uses simple conditional image generation to induce unsupervised landmark detectors, outperforming prior methods on faces and extending to people, 3D objects, and digits.
- Conditional image generation induces object landmark detectors without manual supervision.
- The method outperforms previous unsupervised and supervised face-landmark methods.
- The approach extends to challenging people data and diverse 3D-object and digit datasets.
A MAFL
MAFL experiments vary the number of unsupervised landmarks and training data, reporting normalized mean squared error after regression to five annotated landmarks; TPS training excludes invalid border regions from reconstruction loss.
- MAFL evaluation varies K unsupervised landmarks learned from CelebA random-TPS warps or VoxCeleb face videos.
- The learned landmarks are regressed onto five manually annotated MAFL test landmarks using CelebA or MAFL training sets.
- Performance is reported as mean squared error normalized by inter-ocular distance.
- TPS reconstruction loss ignores pixels outside the original image and downweights a 10-pixel edge region followed by a 20-pixel smooth step.
C MAFL and AFLW Faces
The figure compares supervised regression from unsupervised landmarks on MAFL and AFLW test sets, using models trained with random-TPS image pairs and a perceptual-loss network.
- Five keypoints are linearly regressed from ten unsupervised landmarks on MAFL and AFLW test sets.
- White dots mark ground-truth locations, while dark dots show predictions.
- The models use random-TPS warped image pairs and a self-supervised perceptual-loss network.
D VoxCeleb
VoxCeleb video frames are used to train the conditional generation model, whose learned landmarks consistently track facial features.
- The figure presents source and target VoxCeleb frames alongside a generated target and landmarks superimposed on that target.
- The unsupervised landmarks consistently track facial features across the video frames.
E BBCPose
The method learns 50 unsupervised human-pose keypoints, with seven evaluated against annotations. Predicted landmarks are shown both directly and after linear regression.
- 50 unsupervised keypoints are learned for human pose.Seven annotated keypoints correspond to the head, wrists, elbows, and shoulders.
- The top rows show discovered keypoints matched maximally to seven annotations.Annotations are represented by empty circles, while predictions are solid circles.
- The bottom rows show linearly regressed points derived from the discovered keypoints.
F Human3.6M
On Human3.6M, unsupervised landmarks track major body parts across frames, viewpoints, and actors, while the model confounds frontal and dorsal sides. Related smallNORB experiments show viewpoint-, illumination-, and shape-invariant keypoints that generalize beyond five training instances per category.
- F Human3.6M: Human3.6M landmarks consistently track the legs, arms, torso, and head across frames, views, and actors.The model is evaluated on videos of two actors from the test set.
- F Human3.6M: The Human3.6M model confounds frontal and dorsal sides.
- F Human3.6M: SmallNORB trains object-category-specific keypoint detectors on five categories using image pairs that differ only in viewpoint.The paired images share the same object instance or shape and illumination.
- F Human3.6M: The learned smallNORB keypoints are invariant to viewpoint, illumination, and object shape.
- F Human3.6M: Despite only five training instances per category, the detectors generalize to novel test instances and correspond to structurally similar regions.
H Disentangling appearance and geometry
The generator transfers geometry from a target image while retaining appearance from a source image. This factorized behavior is demonstrated across SVHN digits, Human3.6M humans, and AFLW faces.
- The generator substitutes the target image’s appearance with the source image’s appearance.At inference, image pairs with different styles produce transfer across object categories.
- For SVHN digits, the digit shape is swapped while colours, shadows, and blur are retained.
- For Human3.6M humans, pose transfers across actors and viewpoints while shoes, clothing colour, and hats are retained.
- For AFLW faces, the source image is rendered with the target image’s pose while identity is retained.