Source-linked AI summary
Unsupervised learning of object landmarks by factorized spatial embeddings
James Thewlis, Hakan Bilen, Andrea Vedaldi
TL;DR
Learning viewpoint-independent object structure without manual supervision remains difficult. The paper factorizes image deformations through an intrinsic reference frame and learns equivariant landmark detectors, finding landmarks that align across instances and predict manual facial landmarks. It also shows strong supervised-landmark regression results and effective pretraining, while leaving very large out-of-plane rotations and a shared reference-shape assumption as scope boundaries.
Problem
Learning intrinsic, viewpoint-independent structure such as landmarks from images without manual supervision remains largely open.
Method
The method factorizes viewpoint and deformation effects through an intrinsic reference frame, training a deep network to detect landmarks equivariantly from related images.
Results
The method works for rigid and deformable objects and categories, produces landmarks predictive of manual ones, and achieves a 6.67 error rate versus 7.95 for TCDCN.
Takeaways & Limitations
Unsupervised landmarks can establish cross-instance structure and support effective pretraining for semantic landmark detectors with limited annotations.
Takeaways & Limitations
Very large out-of-plane rotations remain future work because handling partial landmark occlusions is not addressed, and category modeling assumes isomorphic object surfaces.
Abstract
from arXiv · showhide
Learning automatically the structure of object categories remains an important open problem in computer vision. In this paper, we propose a novel unsupervised approach that can discover and learn landmarks in object categories, thus characterizing their structure. Our approach is based on factorizing image deformations, as induced by a viewpoint change or an object deformation, by learning a deep neural network that detects landmarks consistently with such visual effects. Furthermore, we show that the learned landmarks establish meaningful correspondences between different object instances in a category without having to impose this requirement explicitly. We assess the method qualitatively on a variety of object types, natural and man-made. We also show that our unsupervised landmarks are highly predictive of manually-annotated landmarks in face benchmark datasets, and can be used to regress these with a high degree of accuracy.
1. Introduction
The paper addresses the open problem of learning viewpoint-independent object structure without manual supervision by factorizing image deformations and detecting equivariant landmarks. The method applies across object instances and categories, with learned landmarks predictive of manual facial landmarks.
- Learning intrinsic, viewpoint-independent object structure from images remains largely open without manual annotations.
- The approach factorizes image deformations by explaining them through motion of a viewpoint-independent reference frame attached to the object.
- A neural network learns landmarks from warped image pairs by requiring its outputs to be equivariant to the transformation, without manual supervision.
- Meaningful landmarks emerge for rigid and deformable objects and for object categories, with cross-instance alignment occurring without an explicit alignment constraint.
- The method is evaluated on shoes, animals, and faces, and its unsupervised landmarks predict manually annotated facial landmarks well enough to support semantic-landmark pretraining.
2. Related work
The related work spans geometric and semantic flow, unsupervised part discovery, deformation-prediction networks, and landmark detectors. The paper distinguishes its approach by explicitly learning one viewpoint-independent landmark set across images without manual landmark supervision.
- Flow: Flow methods match images through motion-induced deformations, while semantic flow defines correspondence beyond purely geometric warps.
- The paper's landmarks establish geometric and indirect semantic correspondences while providing a single viewpoint-independent set valid across images.
- Parts: Unsupervised part-learning methods model object structure through constellations, deformable parts, or learned parts matching instances and categories.
- Deformation-prediction networks: Deformation-prediction networks align image pairs by estimating transformations, whereas this paper explicitly trains a network to output equivariant keypoints without supervision.
- Landmark detection: Existing face methods include alignment, descriptors, part models, clustering, templates, direct coordinate regression, and deep networks, but the cited approaches do not learn landmarks from scratch.
3. Method
The method learns viewpoint-independent object structure by factorizing image deformations through equivariant landmark mappings, extending the formulation to deformable objects and object categories.
- 3.1. Structure from viewpoint factorization: Viewpoint factorization represents image warps by mapping pixels through intrinsic object points in a shared reference frame.The warp is decomposed by first identifying the object point corresponding to a pixel, then locating that point in the second image.
- 3.1. Structure from viewpoint factorization: A deep neural network is trained in a Siamese configuration so landmark detections remain equivariant under supplied or randomly synthesized transformations.Random transformations can generate training triplets from unannotated images when real viewpoint warps are unavailable.
- 3.1.1 Deformable objects: For deformable objects, a common reference space and isomorphisms tie corresponding surface points across shape variants, subject to no topological changes.The same equivariance principle accounts for both viewpoint-induced image deformations and deformations of the object surface.
- 3.1.2 Object categories: For object categories, sharing one learned mapping across instances makes semantically consistent correspondences emerge without an explicit alignment constraint.The paper argues that consistent deformation handling favors correspondences such as matching the right eye across different faces.
- 3.2. Landmark detection networks: The mapping is represented by K discrete reference locations whose outputs are K landmark positions, typically extracted from neural-network score maps with soft argmax.Discrete landmarks are preferred for robustness to occlusion and because they avoid assumptions about an allowable transformation family.
- 3.3. Learning formulation: Training compares landmark outputs across transformed image pairs, and probability-map losses provide a more direct and stable gradient signal than localization through soft argmax.The probability-map loss encourages corresponding maps to overlap and concentrate; it reaches zero when matched maps are delta functions aligned by the transformation.
4. Experiments
The experiments train unsupervised landmark detectors on shoes, cat faces, and human faces, then evaluate consistency under synthetic deformations, cross-instance alignment, and regression to supervised facial landmarks. The learned landmarks support accurate semantic landmark prediction, including performance competitive with or better than supervised detectors and useful results with limited annotations.
- 4.1. Implementation details: The detector uses six convolutional layers, a single early max-pooling layer, and spatial softmax to produce K landmark probability maps.Training uses randomly generated TPS-based image pairs and optimizes the unsupervised objective with Adam.
- 4.2. Qualitative results: Unsupervised landmarks are trained from scratch on shoes, cat faces, and CelebA faces, using 8-, 10-, or 30-landmark networks for qualitative evaluation.The datasets include UT Zappos50k shoes, cat heads, and more than 200,000 CelebA face images.
- 4.2. Qualitative results: Landmarks remain consistent under synthetic affine or TPS warps and across different object instances, despite no explicit cross-instance alignment constraint.Examples include shoes, cat faces, and faces, with facial landmarks aligning with features such as eyes and mouth corners.
- 4.3. Quantitative results: The learned landmarks can be linearly regressed to manually annotated facial landmarks, with 6.67 error on MAFL versus 7.95 for TCDCN.Using more unsupervised landmarks improves regression because some discovered points are likely to correlate strongly with the manually labeled landmarks.
- 4.3. Quantitative results: With 10 or 20 labeled images, regression from 30 unsupervised landmarks achieves errors of 8.5 and 8.06, compared with 7.15 using all 19,000 training images.The results support unsupervised pretraining for learning semantic landmarks with few annotations; on 300-W, errors are 9.30 without and 7.97 with unsupervised fine-tuning.
- 4.3. Quantitative results: The learned regressor assigns source-landmark contributions according to distance from target points and ignores a forehead landmark outside the target-point convex hull.The contribution structure is visualized as a directional graph between discovered and semantic landmarks.
5. Conclusions
The paper presents unsupervised object-structure learning as equivariant landmark detection, interpreting this as factorizing deformations through an intrinsic reference frame.
- 5. Conclusions: Equivariant landmark detectors reduce unsupervised object-structure learning to compatibility with image deformations.The approach instantiates deformation factorization by learning an intrinsic reference frame.
- 5. Conclusions: The technique works for rigid objects, deformable objects, and object categories.
- 5. Conclusions: The learned landmarks are highly predictive of manually annotated landmarks and can be used effectively for pretraining.
- 5. Conclusions: The work acknowledges support from the AIMS CDT and ERC 677195-IDIU.
Supplementary Material: Unsupervised learning of object landmarks by
The supplementary section identifies the paper by its title and lists its three Oxford-affiliated authors.
- Supplementary Material: The paper is titled “factorized spatial embeddings.”
- Supplementary Material: James Thewlis is affiliated with the University of Oxford.
- Supplementary Material: Andrea Vedaldi is affiliated with the University of Oxford.
- Supplementary Material: Hakan Bilen is affiliated with the University of Oxford and the University of Edinburgh.
1. Introduction
The supplementary material expands the experimental setup, compares against supervised training with few images, and provides qualitative results.
- 1. Introduction: The supplement details the experimental setup and hyperparameters.
- 1. Introduction: It compares the method quantitatively with a supervised network trained using small numbers of images.
- 1. Introduction: It presents numerous images to provide a qualitative view of the method’s performance.
- 1. Introduction: The supplement organizes these materials into additional details, quantitative comparison, and qualitative results sections.
2. Experimental details
The experiments generate paired Thin Plate Spline warps, sample their parameters from zero-mean Gaussians, and vary image preprocessing and pooling settings across datasets.
- 2. Experimental details: Paired warps are parameterized as Thin Plate Spline deformations with affine components.The warps model keypoint deformations together with an affine transformation.
- 2. Experimental details: Warp parameters are sampled from zero-mean Gaussians using dataset-specific standard deviations.
- 2. Experimental details: The affine component uses a similarity transform with rotation, translation, and scale parameters.
- 2. Experimental details: Faces and cats use 100 × 100 inputs cropped to 80 × 80 after warping, while MNIST uses resized and padded images.
- 2. Experimental details: Pooling before the diversity loss uses 5 × 5 windows except for MNIST and the AFLW 51-landmark network, which use 3 × 3 windows.
- 2. Experimental details: Table 1 reports standard deviations for sampled warp parameters, while Table 2 reports AFLW results across training-image counts.Table 2 compares supervised training from scratch with regression using unsupervised landmarks.
- 2. Experimental details: The pooling configuration is associated with denser coverage of the face area in the referenced figure.
3. Supervised Network Comparison
The section compares supervised training from scratch with the paper’s unsupervised-pretrained network under limited labeled data, using results summarized in Table 2 and Figure 1.
- Comparison setup: The comparison evaluates supervised training from scratch against unsupervised pretraining when only a limited number of labeled images are available.The adapted pretrained architecture adds a final pooling layer with stride 2 and a fully connected layer.
- Reported result: 23.85 error is achieved with 20 images using the adapted unsupervised architecture.
- Comparison setup: Figure 1 presents the same comparison as Table 2 in graphical form.
4. Qualitative Results
The qualitative experiments examine unsupervised landmarks across digits, cats, faces, illumination changes, and multiple landmark-network sizes, including target-dataset finetuning.
- Object and face categories: Additional qualitative results cover 20-landmark cat networks, 30-landmark CelebA networks, 300-W regression, and 51-landmark AFLW finetuning.
- MNIST: MNIST experiments train separate networks for digits 3, 5, and 6 using Thin Plate Spline and similarity-transform augmentation.The experiments use around 5000 training images per digit and show robustness across writing-style variation.
- Illumination variation: On Cropped Extended Yale B1, the CelebA network gives consistent landmarks under moderate lighting variation despite the domain shift to grayscale, tightly cropped faces.
- Illumination variation: Hard shadows cause failures where facial regions contain little resolvable detail.
- Illumination variation: Finetuning with synthetic warps or aligned different-lighting pairs produces qualitatively good landmark predictions across illumination variants.The aligned-pair option uses the identity transformation between images of the same subject.
5. Evaluating learned features
The learned landmark features are evaluated for transfer to face segmentation, where pretrained representations improve pixel accuracy and support adaptation beyond landmark regression.
- Transfer evaluation: Face segmentation on HELEN2 tests whether features from the pretrained 50-landmark CelebA network transfer to another face-based task.The first three layers are frozen and the last layer is replaced with a 10-way spatial classification layer.
- Transfer evaluation: 94.46% pixel accuracy is obtained with frozen pretrained layers versus 93.59% for the same configuration trained from scratch.
- Transfer evaluation: 94.82% pixel accuracy is reached when all pretrained layers are finetuned.
- Transfer evaluation: Pretraining converges faster and supports training with fewer images.The paper reports additional limited-data results for 20 and 50 images in Table 4.
- Qualitative inspection: The qualitative figures show landmark and regressor outputs for networks with 7, 10, 20, 30, 50, and 51 landmarks across several datasets.