Source-linked AI summary
Unsupervised Part-Based Disentangling of Object Shape and Appearance
Dominik Lorenz, Leonard Bereska, Timo Milbich, Björn Ommer
TL;DR
The paper tackles unsupervised disentangling of shape and appearance in object images with substantial articulation and local variation. It learns consistent parts using invariance and equivariance constraints, and reports improvements in unsupervised shape learning alongside favorable comparisons with supervised methods across transfer and video tasks.
Problem
Unsupervised learning must separate shape and appearance from their superposition in images while representing large object articulation with flexible parts.
Method
The model learns a part-based object representation by jointly disentangling local shape and appearance through generative training with invariance and equivariance constraints.
Results
The approach significantly improves prior unsupervised object shape learning and performs favorably against supervised methods on shape and appearance transfer, with smooth video-to-video retargeting.
Takeaways & Limitations
Consistent local parts can support unsupervised pose prediction, image synthesis, appearance transfer, and video retargeting across diverse object categories.
Abstract
from arXiv · showhide
Large intra-class variation is the result of changes in multiple object characteristics. Images, however, only show the superposition of different variable factors such as appearance or shape. Therefore, learning to disentangle and represent these different characteristics poses a great challenge, especially in the unsupervised case. Moreover, large object articulation calls for a flexible part-based model. We present an unsupervised approach for disentangling appearance and shape by learning parts consistently over all instances of a category. Our model for learning an object representation is trained by simultaneously exploiting invariance and equivariance constraints between synthetically transformed images. Since no part annotation or prior information on an object class is required, the approach is applicable to arbitrary classes. We evaluate our approach on a wide range of object categories and diverse tasks including pose prediction, disentangled image synthesis, and video-to-video translation. The approach outperforms the state-of-the-art on unsupervised keypoint prediction and compares favorably even against supervised approaches on the task of shape and appearance transfer.
1. Introduction
The paper addresses unsupervised disentangling of shape and appearance for articulated objects, where variation may be global or local. It proposes a part-based representation learned with invariance and equivariance constraints and evaluates it across diverse tasks.
- Motivation: Unsupervised learning must separate intertwined shape and appearance factors such as pose, articulation, texture, and color from unlabeled object images.Object deformation can also change which pixels depict foreground and background.
- Applications: The learned representation supports unsupervised pose estimation, image synthesis, video-to-video translation, and part-level appearance transfer.These applications are summarized in Figure 1.
- Motivation: Existing supervised approaches depend on pose labels and therefore do not apply broadly to arbitrary object categories with unlabeled data.Prior unsupervised methods had mainly demonstrated results on rigid objects or required multiple instances of the same person.
- Approach: The proposed model learns a local, part-based disentangled representation for articulated objects without supervision or prior class information.It uses a generative, fully differentiable autoencoding framework with explicit invariance and equivariance constraints.
- Results: The approach improves unsupervised landmark regression across rigid and articulated datasets, with particularly large gains under strong articulation.The authors also report favorable comparison with supervised results and smooth frame-by-frame video-to-video retargeting.
2. Related Work
Prior work includes supervised and unsupervised shape-appearance disentangling, discriminative part learning, and unsupervised landmark methods. The paper differs by learning articulated part-based shape and appearance jointly while incorporating part extent into equivariance constraints.
- Shape and appearance disentangling: Supervised disentangling methods condition generative models on specified shape representations, whereas this paper learns shape and appearance without supervision.Earlier unsupervised frameworks used holistic models and focused on relatively rigid objects or simple datasets.
- Part-based representation learning: Part-based representations are common in computer vision, but discriminative methods can ignore spatial arrangement and articulation when solving recognition tasks.This image-modelling approach instead requires parts to encode spatial structure and visual appearance accurately.
- Landmark learning: Unsupervised landmark methods enforce equivariance under artificial image transformations, but equivariance alone can disregard explanatory coverage of the object.A later autoencoder approach adds reconstruction, yet its separation constraint can produce an artificial grid-like layout that does not scale to complex articulations.
- Landmark learning: The paper factorizes each object into local parts with separate shape and appearance descriptions rather than using landmarks only for spatial structure.This lets learned parts capture object variance meaningfully.
- Novelty: Unlike prior landmark methods, the approach accounts for part extent in equivariance and explicitly introduces invariance constraints for part-level shape-appearance disentangling.These are presented as distinctions from the cited landmark-learning approaches.
3. Approach
The approach learns a part-based representation that separates local shape and appearance while enforcing invariance and equivariance through reconstruction of transformed image pairs. A two-stream architecture extracts part activations and appearances, then decodes them into the original image without predefined part structure.
- Two-stream architecture: The two-stream architecture learns part shape and appearance separately, then uses corresponding activations to localize appearance features before decoding the image.A shape stream predicts part activation maps, an appearance stream extracts localized part features, and a U-Net reconstructs x.
- Part-based representation: Images are represented as local parts whose appearance features and shape activation maps vary independently under corresponding transformations.Each part is encoded as φi(x) = (αi(x), σi(x)), with appearance as a feature vector and shape as an activation map.
- Invariance and equivariance: Appearance changes preserve shape maps, while spatial shape changes preserve appearance features and transform shape maps equivariantly.The constraints are αi(x◦s) = αi(x), σi(a(x)) = σi(x), and σi(x◦s) = σi(x)◦s.
- Learning objective: Training combines reconstruction with an equivariance loss computed from transformed image pairs, discouraging unstable pixel-level matching and uniform activations.The model reconstructs x from encoded parts while comparing activation-map moments under transformations.
- Unsupervised training: The model automatically structures objects into meaningful parts without part annotations or manually specified relations, and uses brightness, contrast, hue, TPS, and video-frame transformations during training.The number of parts is reported as noncritical because the model is robust to different choices.
- Decoder: Part locations are approximated by activation-map means and covariances, while appearance vectors are projected into those localized regions for reconstruction.The decoder receives approximated activations and appearance vectors without localization, combining them into a localized appearance encoding.
4. Experiments
The experiments evaluate unsupervised shape learning, disentangled image generation, and video-to-video translation across diverse rigid and articulated object datasets. The approach discovers consistent, flexible parts and performs strongly against unsupervised, semi-supervised, and supervised baselines.
- Datasets: The evaluation covers unsupervised landmark discovery on faces, cats, birds, articulated humans, running dogs, and multiple video datasets.The experiments include CelebA, Cat Head, CUB-200-2011, BBC Pose, Human3.6M, Penn Action, Dogs Run, Deep Fashion, and related visualizations.
- Unsupervised Shape Learning: On CUB-200-2011, the method improves object coverage and landmark consistency over [60] while allowing more flexible part placement.The comparison attributes [60]'s rigid grid-like layout to its separation constraint, whereas localized part modeling captures articulations more closely.
- Unsupervised Shape Learning: The learned representation predicts landmarks consistently across diverse instances and captures meaningful full-body structure under articulation.It tracks fine-grained parts such as arms across heavy articulations, viewpoint changes, blurred limbs, and cluttered backgrounds.
- Quantitative Evaluation: On BBC Pose, the method outperforms by 6.1% and narrows the gap to supervised methods, while also outperforming other unsupervised and semi-supervised methods on Human3.6M and Dogs Run.The BBC Pose metric measures the percentage of points within 6 pixels of the ground-truth location.
- Disentangled Synthesis: Disentangled shape and appearance enable unsupervised image synthesis with independently controlled pose, clothing, and selected object parts.Part appearance transfer includes local swaps such as shirts and pants, unlike holistic representations that do not guarantee localization.
- Video-to-Video Translation: Frame-by-frame video-to-video translation produces smooth sequences and robust pose transfer despite having no temporal consistency constraints.The model captures realistic transitions including out-of-plane rotation and complex 3D articulation of hands and fingers.
5. Conclusion
The paper presents an unsupervised approach that disentangles shape and appearance through compositional parts, invariance, and equivariance constraints. Experiments show significant improvement over previous unsupervised methods.
- The approach learns compositional object parts by disentangling shape from appearance without supervision.
- The generative framework incorporates invariance and equivariance constraints to discover consistent parts without prior assumptions.
- Experiments show significant improvements over previous unsupervised methods.
A. Supplementary Material
The supplementary material includes an extension of the Fashion dataset results from Figure 8 in the main paper.
- Fashion dataset results extend Figure 8 from the main paper.
- The supplementary material reports an extension to the main-paper Fashion results.
- The extension is identified as supplementary material accompanying Figure 8.
A.1. Disentangled Representation
The supplementary experiments illustrate disentangled representation through local appearance transfer, video-to-video translation, and generated pose sequences on BBC Pose.
- Local Appearance Transfer: Local appearance transfer successively swaps the appearance of individual parts using different source images.
- Local Appearance Transfer: Figure 9 shows six examples of successively altering part appearances, starting from the original appearance.
- Video-to-Video Translation: Video-to-video translation presents frame-to-frame appearance-shape transfer sequences on the BBC Pose dataset.
- Video-to-Video Translation: The supplementary material extends the main paper’s Figure 7 with BBC Pose sequences.
- Video-to-Video Translation: Figure 10 generates sequences from a target pose sequence and target appearances.
A.2. Landmark Discovery
The supplementary material presents landmark-discovery results across animal, human, face, and action datasets, including Cat Head, Dogs Run, CUB-200-2011, CelebA, Human3.6M, and Penn Action.
- Landmark Discovery: Unsupervised landmark discovery is shown on Cat Head, Dogs Run, and CUB-200-2011.
- Landmark Discovery: Unsupervised landmark discovery is shown on CelebA and Human3.6M.
- Landmark Discovery: Penn Action includes 12 of 16 displayed landmarks in the supplementary results.
A.3. Implementation Details and Settings
The implementation uses stacked-hourglass encoders, multiscale decoding, approximated part activation maps, and an optional patch-based adversarial task, with settings summarized in Table 7.
- Network architecture: The Eσ and Eα encoders use a stacked hourglass architecture, producing part activation maps and localized appearance encodings at 64 × 64 resolution.Eσ reaches 4 × 4 resolution, while Eα reaches 32 × 32; residual blocks use 256 feature channels.
- Network architecture: The decoder is a U-Net variant operating at the input resolution, passing approximate part activations through skip connections and distributing appearance encodings through a multiscale bottleneck.The bottleneck spans resolutions from 4 × 4 to 16 × 16, and the first upsampling stage uses 512 feature channels.
- Decoder approximation: Two covariance settings approximate part activation maps: an identity matrix or the covariance of normalized activation locations.The identity setting produces more confined part shapes and is used for keypoint regression experiments.
- Adversarial task: A patch-based adversarial task trains a discriminator to classify N × N image patches as real or fake, conditioned on corresponding approximated part activations.Patches of size 49 × 49 are extracted around mean part locations from real and generated images.
- Experimental settings: Table 7 summarizes the experimental settings, including landmark counts, input resolution, Adam learning rate, and adversarial-task configuration.The cited table is an overview of settings across the different experiments.