Source-linked AI summary
Unsupervised Discovery of Object Landmarks as Structural Representations
Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, Honglak Lee
TL;DR
The paper addresses how to learn perceptible object structures without costly manual supervision. It introduces a differentiable landmark-based autoencoder with validity constraints, whose landmarks are more predictive of human annotations and enable attribute recognition and controllable image generation.
Problem
Object-category structures are difficult for deep networks to conceptualize compactly and perceptibly, while manual structure annotations are costly and scarce.
Method
A differentiable autoencoding framework discovers landmark coordinates and uses soft validity constraints and decoder feedback to learn explicit structural representations.
Results
The discovered landmarks outperform prior unsupervised methods in predicting manually annotated landmarks and perform comparably to fully supervised detectors.
Takeaways & Limitations
Landmark coordinates provide explicit representations that complement pretrained neural features and support controllable object-shape manipulation and structure-conditioned image generation.
Takeaways & Limitations
Cross-object semantic correspondence is not explicitly enforced and mainly relies on shared convolutional filters producing semantically similar visual patterns.
Abstract
from arXiv · showhide
Deep neural networks can model images with rich latent representations, but they cannot naturally conceptualize structures of object categories in a human-perceptible way. This paper addresses the problem of learning object structures in an image modeling process without supervision. We propose an autoencoding formulation to discover landmarks as explicit structural representations. The encoding module outputs landmark coordinates, whose validity is ensured by constraints that reflect the necessary properties for landmarks. The decoding module takes the landmarks as a part of the learnable input representations in an end-to-end differentiable framework. Our discovered landmarks are semantically meaningful and more predictive of manually annotated landmarks than those discovered by previous methods. The coordinates of our landmarks are also complementary features to pretrained deep-neural-network representations in recognizing visual attributes. In addition, the proposed method naturally creates an unsupervised, perceptible interface to manipulate object shapes and decode images with controllable structures. The project webpage is at http://ytzhang.net/projects/lmdis-rep
1. Introduction
The paper targets unsupervised discovery of perceptible object structures, introducing a differentiable landmark-based autoencoder with constraints for valid landmark representations. The learned landmarks support landmark prediction, attribute recognition, and controllable image manipulation.
- Object structures can provide appearance-invariant intermediate representations, but manual annotations or designs such as skeletons and semantic parts are costly and scarce.
- Deep neural networks learn rich latent representations for many vision tasks, yet little evidence shows they compactly and perceptibly conceptualize intrinsic object-category structures.
- The proposed differentiable autoencoder discovers landmark coordinates during image modeling, using decoder gradients and regularization to make landmarks informative while avoiding irrelevant or redundant latent information.
- The method uses soft constraints to enforce landmark validity and discovers visually meaningful landmarks without supervision across varied object categories.
- The discovered landmarks outperform previous unsupervised methods for predicting manually annotated landmarks and perform comparably to fully supervised detectors trained with substantial labeled data.
- Landmark representations also support strong visual-attribute recognition and controllable decoding for object-shape manipulation and structure-conditioned image generation.
2. Related work
Related work studies object structures through discriminative parts, dense correspondences, canonical coordinates, and unsupervised landmark discovery. These approaches differ in whether they optimize recognition, match images, or impose structural equivariance.
- Deformable part-based and constellation models learn object-part configurations for detection, while newer neural methods learn deformable mixtures of parts for pose estimation.
- Recurrent and spatial-transformer architectures have been used to discover and refine object parts for fine-grained image classification.
- Other methods discover TPS control points or dense canonical coordinates to match object images and represent object structures.
- Thewlis et al. use equivariance to train deep networks for unsupervised object landmark discovery rather than learning dense correspondence.
3. Autoencoding-based landmark discovery
The framework discovers landmarks as explicit, differentiable structural representations within an image autoencoder. Detection maps yield landmark coordinates and local descriptors, while regularizers encourage concentrated, separated, transformation-equivariant landmarks and the decoder reconstructs images from these representations.
- Landmark detector: An encoder converts an image into a multi-channel detection confidence map containing K landmark channels and one background channel.The landmark detector is implemented with an hourglass-style network and channel normalization encourages different detected patterns.
- Landmark detector: Each landmark coordinate is computed as the detection map’s weighted mean spatial coordinate, allowing gradients to pass through localization.The formulation is differentiable except in rare cases where detection mass is concentrated at one pixel or uniformly distributed.
- Landmark constraints: Three soft constraints promote valid landmarks by encouraging concentrated detection maps, spatial separation, and equivariance under image transformations.The concentration term favors peaked local distributions, the separation term avoids center-collapsed landmarks, and equivariance ties landmark motion to known transformations.
- Scope and representation: The model does not explicitly guarantee semantic correspondence across object instances, relying mainly on shared convolutional filters to produce semantically similar landmarks.For natural images, landmarks also require complementary latent representations because they cannot encode all visual content alone.
- Local latent descriptors: A shared feature map is pooled with each landmark’s soft detection mask, then landmark-specific linear operators produce compact local descriptors.Each descriptor encodes a particular local pattern in limited dimensions, and a separate background descriptor can also be extracted.
- Landmark-based decoder: The decoder approximately reconstructs confidence maps from landmark coordinates and uses them as soft switches to unpool landmark and background descriptors into a feature map.Gaussian maps centered at the coordinates provide the approximate inverse representation before shared-space transformations and nonlinear activation.
4. Experiments
Experiments evaluate unsupervised landmark discovery across diverse object datasets, then test landmark quality, attribute recognition, and controllable image decoding. The method discovers semantically meaningful structures across varied poses, viewpoints, object categories, and limited-label settings, while outperforming prior unsupervised approaches on annotated-landmark prediction.
- Experimental setup: The evaluation spans faces, cat heads, cars, shoes, human poses, MNIST, and animal images across multiple datasets.Quantitative experiments focus on landmark discovery, annotated-landmark prediction, shape-related facial attributes, and image decoding.
- 4.1. Landmark discovery on multiple datasets: The model discovers semantically stable landmarks across faces, cats, cars, shoes, and human bodies, including under occlusion, pose variation, and limited training data.Examples include facial features, cat ears and noses, car wheels and doors, shoe locations, and stable human head, back, waist, and leg landmarks.
- 4.2. Prediction of ground truth landmarks: Less than 200 labeled samples suffice to achieve less than 4% mean error on the MAFL testing set, outperforming TCDCN and MTCNN.Removing any training loss reduces performance, while reconstruction loss further improves detection accuracy.
- 4.3. Visual attribute recognition: Landmark coordinates complement pretrained FaceNet features for recognizing shape-related facial attributes.FaceNet convolutional features perform slightly better alone but have much higher dimensionality; combining them with landmarks improves accuracy.
- 4.4. Controllable image decoding: The decoder produces realistic facial and human-body images whose shapes follow synthetically adjusted landmarks while largely preserving input appearance factors.The experiment fixes landmark latent descriptors and changes landmark locations through synthesized flows.
5. Conclusion
The paper presents unsupervised landmark discovery as an explicit structural component of image representations, supporting manipulation and discriminative recognition. It reports visually meaningful, quantitatively relevant, and complementary landmark representations.
- The method uses fully differentiable landmark discovery with soft constraints enforcing valid landmark representations.
- Discovered landmarks form an explicit structural representation disentangled from other appearance-related latent factors.
- Landmark representations provide an interface for manipulating image generation and appear complementary to pretrained deep-neural-network features for discriminative tasks.
- The work was supported by ONR, NSF CAREER, and a Sloan Research Fellowship.
Appendices
The appendices identify the paper, its authors and affiliations, and provide online access to project materials and supplementary videos.
- The paper is titled “Unsupervised Discovery of Object Landmarks as Structural Representations.”
- The authors are Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee.
- The authors are affiliated with the University of Michigan and Google Brain.
- Code and results are available on the project webpage.
- Supplementary appendix videos are available as a downloadable archive.
A. More details and results on face manipulation using unsupervised landmarks
The appendices demonstrate controllable landmark-based manipulation across faces and digits, including whole-object edits, local facial edits, digit morphing, and optical-flow-assisted Human3.6M experiments.
- Face manipulation: The decoder generates realistic facial images from extracted landmark descriptors and alternative landmark configurations.
- Face manipulation: Face experiments target discovered landmarks from other images and interpolations or extrapolations between two images.
- Face manipulation: The 30-landmark model supports manipulation of all landmarks, three mouth landmarks, and mouth-and-jaw landmarks.
- MNIST: MNIST landmarks remain consistent within each digit despite shape variation when models are trained independently by digit.
- MNIST: A single model trained across ten digits discovers semantically consistent corresponding landmarks across digit categories.
- MNIST: The all-digit model enables geometrically meaningful cross-category morphing, with landmark coordinates serving as the full MNIST image representation.
- Human3.6M: Using optical flow reduced ground landmark prediction error from 4.91 without optical flows to 4.14 with optical flows.
C.2. Quantitative results
On Human3.6M, the method is evaluated by mapping 16 discovered landmarks to annotated landmarks and is compared with both unsupervised and supervised baselines. The reported qualitative and quantitative comparisons favor the proposed method over the prior unsupervised approach, while the supervised model remains stronger.
- Both unsupervised methods discover 16 landmarks and use the full training set to learn a linear mapping to annotated landmarks.
- The proposed method significantly outperforms Thewlis et al.’s unsupervised method on annotated-landmark prediction.
- Both unsupervised methods perform worse than Newell et al.’s supervised stacked hourglass network.
- Some wearable-marker annotations are close together, making the effective number of distinct annotated locations around 16 rather than 32.
- The Human3.6M table reports prediction error as a percentage of image edge length.
- Qualitative comparisons on six actions show the proposed method visually outperforming Thewlis et al.’s method.
D. Results on animals of mixed species
On mixed animal images spanning five species, the method discovers consistent landmarks despite substantial appearance differences.
- The model finds recurring landmarks across antelope, deer, moose, horse, and zebra images, including hoof, back, and buttock locations.These landmarks remain consistent even when multiple species are mixed.
E. More qualitative results on human faces, cat heads, cars, and shoes
Qualitative experiments show landmark discovery across faces, cat heads, cars, and shoes, with discovered points often aligned to meaningful visual features. Comparisons on annotated landmarks favor the proposed method over Thewlis et al. [59].
- Human faces: On CelebA, discovered landmarks are highly aligned with facial features such as mouth corners, eye corners, and the nose.
- Human faces: The method discovers 10 or 30 landmarks on CelebA and AFLW, including unaligned CelebA images.
- Annotated-landmark prediction: For annotated-landmark prediction, the method shows more low-error landmarks than Thewlis et al. [59] on faces, cat heads, and cars.The comparisons use five facial, seven cat-head, and six car annotated landmarks.
- Cat heads: On cat heads, the method places landmarks on meaningful regions including the nose, eyes, and base of the earlobe.
- Cars and shoes: The method also discovers 10 or 24 landmarks on PASCAL-VOC 3D cars and eight landmarks on shoes.
F.1. Number of labeled samples for annotated-landmark prediction
The discovered landmarks support accurate prediction of manually annotated landmarks with few labeled examples, while training dynamics and TPS-control-point choices affect how the system is optimized.
- Training dynamics: During training, confidence maps evolve from diffuse responses producing centered landmarks to spatially concentrated responses.The separation loss helps move each channel’s peak toward a different location.
- TPS control points: The equivariance constraint alternates regular-grid and discovered-landmark TPS control points, using the two sets with a 7:3 chance.Regular-grid control points are used initially because early discovered landmarks cluster at the image center.
G.1. Data preprocessing
The supplementary implementation specifies dataset-specific preprocessing, padded inputs for random TPS transformations, hourglass-style networks, and optimization details for landmark discovery and reconstruction.
- Dataset preprocessing: Inputs are padded with edge values so random TPS transformations retain sufficient image margins.
- Dataset preprocessing: Animal preprocessing keeps left-facing profile views, flips right-facing animals horizontally, and excludes frontal and back views.
- Network architecture: The model uses hourglass-style mirrored encoding and decoding with same-resolution skip-links fused by element-wise addition.Max-pooling reduces feature-map size during encoding, while nearest interpolation upsamples during decoding.
- Network architecture: A 1×1 convolution produces K + 1 raw detection-score channels, while output responses are mapped to reconstruction intensity and pixel-wise standard deviation.
- Optimization: Training uses Adam, random TPS warps for equivariance, Gaussian reconstruction likelihood, and dataset-specific hyperparameters.The reconstruction loss uses pixel values scaled to [0, 1] and fixed σ_color = 0.05.
- Structure-conditioned decoding: The landmark decoder can generate facial images by conditioning on detected landmarks while sampling an isotropic Gaussian latent representation.