Unsupervised Discovery of Object Landmarks as Structural Representations

• Intrinsic object structures can support high-level visual understanding while reflecting physical state and remaining invariant to appearance changes, but manual annotations are costly and deep networks do not naturally represent them compactly and perceptibly.Source: §1 · p. 1
• Earlier unsupervised landmarks targeted stable, spatially equivariant visual patterns but did not explicitly favor critical image-modeling locations; their non-differentiable formulation also prevented decoder gradients.Source: §1 · p. 1
• A differentiable landmark autoencoder lets decoder gradients train landmark detection, while soft constraints encourage valid landmarks and prevent irrelevant or redundant information.Source: §1 · p. 1
• Without supervision, the method discovers visually meaningful landmarks that outperform the state-of-the-art in predicting manual annotations and perform comparably to fully supervised detectors trained with significant labeled data.Source: §1 · p. 1
• The discovered landmark coordinates provide discriminative features for visual-attribute recognition and support object-shape manipulation and structure-conditioned image generation.Source: §1 · p. 2

Discriminative part learning

• Unlike discriminative part-learning methods, this work learns landmarks for generic image modeling rather than a specific recognition or pose-estimation task.Prior approaches learn part configurations for detection or deformable mixtures of parts for pose estimation.Source: §2 · p. 2

Learning structural representations

• Existing structural-representation methods encode factors in non-perceptible latents or require supervision for human-designed outputs such as landmarks, masks, and depth [65].These approaches either disentangle factors such as viewpoint, motion, and identity or predict conceptualized structures as additional outputs.Source: §2 · p. 2

Learning explicit structures for image correspondence

• The discovered landmarks fall between object parts and 3D points in density, and are unsupervised and sensitive to precise locations.Prior correspondence methods range from coarse colocalization [9] to object parts, TPS image matching [24], and annotation-based 3D skeleton discovery [67].Source: §2 · p. 2

Landmark discovery with equivariance

• Unlike equivariance-based discovery [59], this method uses equivariance to validate landmarks and differentiably incorporates coordinates into image modeling, yielding more predictive landmarks across broader object categories.The paper reports greater predictive power for manual landmarks than Thewlis et al. [59].Source: §2 · p. 2

Image modeling with landmarks

• Unlike landmark-conditioned GAN and PixelCNN generators [46] [47], this method uses a stacked-autoencoder framework to discover landmarks informative for unsupervised image modeling.The related work also surveys SAE, variational autoencoders [25], GANs [43], and autoregressive models.Source: §2 · p. 2

Landmark detection

• The authors use discovered landmarks to predict manual annotations and compare their method with recent supervised landmark-detection models.The related work describes supervised detectors for faces, bodies, and birds.Source: §2 · p. 2

3.1. Architecture of landmark detector

• An hourglass-style detector produces softmax-normalized landmark and background maps, whose weighted mean coordinates define differentiable landmark locations.The map has K landmark channels plus a background channel; coordinate gradients can pass through the weighted means except when a detection map’s mass is entirely concentrated in one pixel or uniformly distributed.Source: §3.1 · p. 3

3.2. Visual concept of landmarks

• Regularizers encourage concentrated, spatially separated landmarks that move equivariantly under image transformations, while cross-instance semantic correspondence is not explicitly ensured.Transformations can be simulated with thin plate splines using affine changes and perturbed control points, or with dense motion flow for video.Source: §3.2 · pp. 3–4

3.3. Local latent descriptors

• Low-dimensional local descriptors supplement landmarks for natural images, using landmark-centered soft-mask pooling and landmark-specific linear maps to limit each descriptor’s information.The shared feature map is pooled with a Gaussian approximation of each detection map; each descriptor is mapped into an individual space with C < S channels.Source: §3.3 · p. 4

3.4. Landmark-based decoder

• The decoder reconstructs images from landmark-derived confidence maps and descriptors, using Gaussian localization and differentiable unpooling to preserve gradient flow through coordinates.Multiple decoder Gaussian variances balance broader coordinate gradients early in training with sharper descriptor localization later.Source: §3.4 · p. 5

3.5. Overall training objective

• The squared-error image reconstruction loss drives training of the entire autoencoder.Source: §3.5 · p. 5

4.1. Landmark discovery on multiple datasets

• It discovers stable semantic landmarks on faces and shoes, consistent cat landmarks under significant occlusion, meaningful car landmarks, and roughly consistent Human3.6M landmarks across poses, with arms less consistent.The car dataset has 567 training images and 63 test images. For faces, landmarks are located more robustly against pose variations and occlusions than Thewlis et al.’s method [59].Source: §4.1 · pp. 6–7

4.2. Prediction of ground truth landmarks

• Our 30-landmark detector reduces annotated-landmark error 55% vs Thewlis et al. [59] and 45% vs Thewlis et al. [58], and outperforms prior unsupervised and pretrained fully supervised models on MAFL and AFLW.The regression targets are five annotated landmarks; on AFLW, only the five always-visible landmarks are used.Source: Table 1 · p. 7, §4.2 · p. 7
• The method also predicts annotated landmarks on cat, car, and Human3.6M datasets; fewer than 200 labels yield under 4% MAFL mean error, while removing any loss hurts, especially the separation loss.Adding reconstruction loss further improves accuracy.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.4. Image manipulation and generation

• Adjusting landmarks while fixing their latent descriptors yields realistic face and body images matching the new shapes, with other appearance factors visually unchanged.The results indicate that unsupervised landmarks can condition image decoding as an explicit representation separated from other variation factors.Source: Figure 10 · p. 8, §4.4 · p. 8

4.3. Visual attribute recognition

• On CelebA shape-related facial attributes, 60-dimensional landmarks outperform FaceNet top-layer features for most attributes; combining them with FaceNet improves accuracy.FaceNet conv-layer features perform slightly better than landmarks but have much higher dimensionality.Source: §4.3 · p. 8, Table 3 · p. 9

4.4. Image manipulation and generation

• Reference landmarks combined with randomly sampled latent codes condition face generation; BEGAN [3] and cyclic landmark loss encourage generated images to retain the reference landmarks.The results provide additional evidence that discovered landmarks are useful for image modeling.Source: Figure 11 · p. 9, §4.4 · p. 9

• A fully differentiable network discovers object landmark coordinates and uses soft constraints to enforce their validity.Source: §5 · p. 9
• The discovered landmarks are visually meaningful and quantitatively more relevant to human-designed landmarks.Source: §5 · p. 9
• Landmarks form an explicit part of learned image representations and are disentangled from latent representations of other appearance factors.Source: §5 · p. 9
• Landmark-based representations provide an interface for manipulating the image generation process.Source: §5 · p. 9
• Landmark coordinates appear complementary to pretrained deep-neural-network features for solving discriminative tasks.Source: §5 · p. 9

Appendices

• The authors are affiliated with the University of Michigan, Ann Arbor, and Google Brain.The passage lists authors with superscript 1 for the University of Michigan and 2,1 for Honglak Lee; the affiliations define 1 as the University of Michigan and 2 as Google Brain.Source: Appendices · p. 12
• The project webpage provides the code and results at http://ytzhang.net/projects/lmdis-rep.Source: Appendices · p. 12
• Supplementary videos referred to in the appendices are available at http://ytzhang.net/files/lmdis-rep/supp-videos.tar.gz.Source: Appendices · p. 12

A. More details and results on face manipulation using unsupervised landmarks

• On MAFL, the decoder generates realistic faces from image-derived descriptors and altered landmarks, including manipulations of all 30 landmarks and of mouth and jaw landmarks.The 30-landmark model’s targets include landmarks from other images and interpolations or extrapolations between images’ landmarks.Source: A. More details and results on face… · p. 13, Figure 12 · p. 14, Figure 14 · p. 16

B. Our model without landmark descriptors on MNIST

• The MNIST model omits the landmark-descriptor pathway and is trained either separately for each digit or jointly on all ten digits.The joint model supports morphing between different digits, such as 2 to 9.Source: B. Our model without landmark… · p. 20

B.1. Models for individual digits

• Models trained independently for each MNIST digit discover landmarks that remain consistent despite within-digit shape variations.The section shows a model discovering seven landmarks per digit.Source: §B.1 · p. 20, Figure 18 · p. 20

B.2. Models for all digits

• A single MNIST model discovers landmarks consistent across images of the same digit and semantically corresponding across different digits.Examples include landmarks at the digit’s middle, bottom, and most-left-top region.Source: §B.2 · p. 21, Figure 19 · p. 21

B.3. Digit morphing using discovered landmarks

• For MNIST, landmark coordinates form the full image representation and enable geometrically meaningful cross-category digit morphing.The morphing process linearly interpolates landmark coordinates between source and target digits.Source: §B.3 · p. 22, Figure 20 · p. 22

C. Details and more results on Human3.6M

• On Human3.6M, the study evaluates prediction of 32 annotated landmarks using models trained on six actions, both jointly and independently.The mixed-action model is also evaluated qualitatively.Source: C. Details and more results on Human3.6M · p. 24

C.1. Optical flow as self-supervision for equivariance

• For Human3.6M, optical-flow equivariance uses Farneback flow at 5-frame intervals, accumulated to 10-frame intervals alongside random TPS, and discourages background landmarks.Weak-flow regions can yield lower equivariance loss, so the method encourages correlation between landmark heatmaps and flow magnitude.Source: §C.1 · p. 24
• Ground landmark prediction error fell from 4.91 (without optical flows) to 4.14 (with optical flows).The model uses optical flow as equivariance self-supervision in all Human3.6M experiments.Source: §C.1 · p. 24

C.2. Quantitative results

• Both models discover 16 landmarks, and a linear mapping trained on the whole training set predicts the annotated landmarks.To handle frontal-versus-back-view ambiguity, evaluation takes the lower error against the original annotations or their left-right-flipped counterpart, flipping the whole-body annotations together.Source: §C.2 · p. 24
• Our method significantly outperforms Thewlis et al. [59] in annotated-landmark prediction, although both unsupervised methods perform worse than Newell et al. [39]’s supervised model.The paper notes that its method is unsupervised and uses a smaller neural-network architecture than the supervised model.Source: §C.2 · p. 24, Table 4 · p. 25
• The linear-regressor strategy first heuristically identifies frontal views, trains on those images, then iteratively decides whether to flip other annotations and retrains on all training images.The frontal-view heuristic marks an image frontal when more than two-thirds of left-side annotated landmarks lie to the right of right-side landmarks; iteration continues until convergence.Source: §C.2 · p. 24
• Although the annotations contain 32 markers, close pairs mean the effective annotated locations are fewer than 32, approximately 16.Source: §C.2 · p. 24

C.3. Qualitative results

• In side-by-side predictions, our method visually outperforms Thewlis et al. [59], with more low-error landmarks shown in deep blue.The comparison predicts 32 annotated landmarks on Human 3.6M.Source: §C.3 · p. 25, Figure 22 · p. 26
• Landmark-discovery examples show cases where our method substantially outperforms Thewlis et al. [59] on the Human 3.6M test set.The examples illustrate discovery of 16 landmarks.Source: §C.3 · p. 25, Figure 23 · p. 27

D. Results on animals of mixed species

• Landmarks align with consistent anatomical locations: the yellow cross marks the hoof, the orange cross lies above the back, and the light green cross marks the buttock.Source: D. Results on animals of mixed species · p. 28
• These landmarks remain consistently detected despite significant variation in species, pose, and individual appearance.Source: D. Results on animals of mixed species · p. 28
• The method discovers 10 landmarks across profile images of antelope, deer, moose, horse, and zebra.The figure shows the landmarks on mixed-species animal images.Source: D. Results on animals of mixed species · p. 28, Figure 24 · p. 29

E. More qualitative results on human faces, cat heads, cars, and shoes

• The displayed test images are randomly sampled and include landmark discovery and ground-truth prediction comparisons with Thewlis et al. [59].Source: E. More qualitative results on human… · p. 30

E.1. CelebA

• On CelebA, discovered landmarks align with facial features, and the figures show results with 10 or 30 landmarks, including on unaligned images.Source: Figure 25 · p. 30, Figure 26 · p. 31, Figure 28 · p. 33
• On CelebA, the method shows more low-error predictions of 5 annotated landmarks than Thewlis et al. [59].Source: Figure 27 · p. 32

E.2. AFLW

• On AFLW, the figures show discovered landmarks with 10 and 30 landmarks.Source: Figure 29 · p. 34, Figure 30 · p. 35

E.3. Cat heads

• On cat heads, 10-landmark discoveries include nose, eye, and earlobe-base locations, while predictions of 7 annotated landmarks show more low-error results than Thewlis et al. [59].Source: Figure 31 · p. 36, Figure 32 · p. 37, Figure 33 · p. 38

E.4. Cars

• On cars, the figures show discoveries with 10 and 24 landmarks, and predictions of 6 annotated landmarks show more low-error results than Thewlis et al. [59].Source: Figure 34 · p. 39, Figure 35 · p. 39, Figure 36 · p. 40

E.5. Shoes

• On shoes, the figure shows landmark discovery using 8 landmarks.Source: Figure 37 · p. 42

F.1. Number of labeled samples for annotated-landmark prediction

• Fewer than 200 labeled samples yield under 4% mean error on MAFL testing and outperform two popular off-the-shelf models.This suggests a high-accuracy landmark detector can be trained with few labeled samples when sufficient unlabeled data train the unsupervised model.Source: §F.1 · p. 43, Figure 38 · p. 43
• The learned detector’s MAFL prediction performance is evaluated across different numbers of labeled samples, with fewer than 200 enough to meet the reported error threshold.Figure 38 plots prediction errors on the MAFL testing set as the number of labeled training samples varies.Source: §F.1 · p. 43, Figure 38 · p. 43

F.2. Evolution of detection confidence map during training

• Initially, random confidence maps produce landmarks at image centers; during training, their values become spatially concentrated.Landmarks are the mean coordinates weighted by the confidence maps.Source: §F.2 · p. 43, Figure 39 · p. 43
• The separation loss moves each confidence-map channel’s peak to a different location, preventing the degenerate outcome in which every peak remains centered.Without the separation loss, all channels can peak at the image center.Source: §F.2 · p. 43

F.3. TPS control points

• For TPS equivariance, regular-grid points and current discovered landmarks alternate as control points with a 7:3 chance, a ratio kept fixed across experiments.The ratio is not exhaustively tuned.Source: §F.3 · p. 44, Table 5 · p. 44
• Model performance is fairly insensitive to the TPS control-point ratio, while including discovered landmarks as control points provides a benefit.The passage refers to Figure 5 for the performance comparison; Table 5 describes the ratio-impact evaluation setup.Source: §F.3 · p. 44, Table 5 · p. 44
• Because discovered landmarks cluster at image centers early in training and are poor TPS control points, the method starts with regular-grid points and adds the ratio-based mixture after several thousand iterations.The delay reflects the separation constraint’s effect on early landmark locations.Source: §F.3 · p. 44

G.1. Data preprocessing

• Random TPS transformations required image margins large enough to keep foreground content in frame; Table 6 summarizes dataset image sizes before and after edge-value padding.Source: §G.1 · p. 44, Table 6 · p. 44
• CelebA and AFLW faces, cat heads, and shoes were cropped or resized to standardized inputs, with shoes padded and their color range compressed for sigmoid outputs.CelebA images were scaled before center cropping; AFLW boxes were enlarged, cat crops used landmark-derived boxes, and shoes were padded with white margins.Source: §G.1 · p. 44
• Cars were cropped using landmark-based boxes; AwA animals were squared and filtered to left-facing views, while Human3.6M bodies were aligned using 3D landmarks and masked backgrounds.AwA crops were scaled to 64×64; Human3.6M crops were scaled to 128×128.Source: §G.1 · pp. 44–45

G.2. Network architectures

• The hourglass networks use mirrored encoding and decoding, convolutional skip-links fused by addition, max-pooling and nearest-neighbor upsampling; image outputs predict sigmoid intensities and positive pixelwise standard deviations.Batch normalization follows each linear layer, with LeakyReLU as the default activation.Source: §G.2 · p. 47

G.3. Training strategy

• Training used Adam with staged learning-rate decay and random color jitter; batch-normalization statistics came from a random training subset, and batch sizes 16 or 32 performed similarly.Source: §G.3 · p. 47
• Equivariance training warped paired inputs with random TPS transformations, while the reconstruction-loss weight was increased tenfold twice to stabilize early landmark and decoder learning.Source: §G.3 · p. 47

G.4. Hyper-parameters

• The model was not sensitive to minor hyperparameter changes, though dataset-specific tuning could improve performance slightly; loss weights were set using normalized landmark coordinates.Source: §G.4 · p. 48, Table 7 · p. 48

G.5. Details about face generations using unsupervised landmarks

• Face generation fixed the landmark detector, sampled Gaussian latent codes, fused landmark-confidence and latent feature maps, and trained a BEGAN discriminator plus an L1 landmark-consistency loss.The experiment aimed to demonstrate that discovered landmarks were accurate and meaningful enough for controllable image generation.Source: §G.5 · p. 48