Source-linked AI summary
Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance
Zhixin Shu, Mihir Sahasrabudhe, Alp Guler, Dimitris Samaras, Nikos Paragios, Iasonas Kokkinos
TL;DR
The paper addresses unsupervised disentangling of image shape and appearance, including the challenge of modeling shape variation and its importance for image understanding and manipulation. It introduces Deforming Autoencoders that represent shape as deformation from a canonical template and appearance in template coordinates. The method enables group-wise alignment, image manipulation, landmark localization, and unsupervised shading-albedo decomposition in face images.
Problem
Unsupervised models need to separate shape, represented as domain transformations, from appearance while handling the greater difficulty of shape variation.
Method
Deforming Autoencoders predict a smooth, invertible template-to-image deformation together with template-aligned appearance, using affine and non-rigid deformation fields.
Results
The method supports unsupervised group-wise image alignment, image interpolation and manipulation, landmark localization, and shading-albedo decomposition; landmark localization outperforms recent self-supervised correspondence methods.
Takeaways & Limitations
Canonical template coordinates enable broader unsupervised disentangling, including separating facial shading and albedo and manipulating latent shading representations.
Abstract
from arXiv · showhide
In this work we introduce Deforming Autoencoders, a generative model for images that disentangles shape from appearance in an unsupervised manner. As in the deformable template paradigm, shape is represented as a deformation between a canonical coordinate system (`template') and an observed image, while appearance is modeled in `canonical', template, coordinates, thus discarding variability due to deformations. We introduce novel techniques that allow this approach to be deployed in the setting of autoencoders and show that this method can be used for unsupervised group-wise image alignment. We show experiments with expression morphing in humans, hands, and digits, face manipulation, such as shape and appearance interpolation, as well as unsupervised landmark localization. A more powerful form of unsupervised disentangling becomes possible in template coordinates, allowing us to successfully decompose face images into shading and albedo, and further manipulate face images.
1 Introduction
Deforming Autoencoders bring deformable-template modeling into unsupervised autoencoders to disentangle image shape from appearance. The approach supports image manipulation, deformation modeling, and unsupervised landmark localization.
- Shape variation is difficult because it transforms a function’s domain, creating complex nonlinear optimization problems even for simple supervised additive models.
- Deforming Autoencoders represent object instances as deformations of a prototypical template, separating spatial transformations from appearance variation.
- The architecture predicts a smooth, invertible template-to-image transformation alongside template-aligned appearance, then deforms the synthesized appearance to reconstruct the input.
- The decoder composes a global affine field with a learned non-rigid field and constrains the result to be a diffeomorphism, which simplifies training and improves accuracy.
- Disentangling deformation from appearance improves expression, pose, and identity interpolation compared with standard autoencoders and supports shading-albedo decomposition and re-shading.
- The method outperforms recent self-supervised correspondence-estimation methods on landmark localization, despite being trained only to reconstruct pixel intensities.
2 Deforming Autoencoders
Deforming Autoencoders reconstruct images by synthesizing appearance in template coordinates and applying a learned deformation, separating shape from appearance. The architecture constrains deformations for stable modeling and extends the representation to class-aware and intrinsic decomposition settings.
- Deforming Autoencoders: The model reconstructs an observed image by warping synthesized template-coordinate appearance with an estimated deformation field.Appearance is synthesized at coordinate p and sampled through W(p) using a differentiable spatial transformer.
- Deforming Autoencoders: Independent appearance and deformation decoders receive separate latent codes, ZT and ZS, to represent appearance and shape distinctly.A joint encoder produces Z = [ZT, ZS], which feeds the two decoder networks.
- Deformation field modeling: Global deformation is modeled with an affine field, while a convolutional decoder learns the remaining non-rigid deformation.The shape code is divided into ZA for affine deformation and ZW for the learned non-rigid component.
- Deformation field modeling: The differential decoder predicts ∇xWx and ∇yWy instead of the local warp directly, enabling control over folding and excessive stretching.ReLU enforces positive offsets between adjacent pixels, and spatial integration reconstructs smooth warping fields that avoid self-crossings.
- Class-aware Deforming Autoencoder: Class information can be added to the latent representation and supplied to both decoders, helping model multi-modal deformation distributions.The class-aware variant introduces ZC and a classification subnetwork, as illustrated on MNIST.
- Intrinsic Deforming Autoencoder: Intrinsic-DAE decodes shading and albedo separately, forms texture as T = S ◦ A, and then warps it; smooth shading regularization supports decomposition of unaligned faces.A standard autoencoder with intrinsic decomposition fails to produce plausible shading and albedo components without the learning-to-align property.
3 Experiments
Experiments show that Deforming Autoencoders learn canonical appearances, meaningful deformation representations, unsupervised alignment, and intrinsic image decompositions across digits, faces, and hands.
- Datasets and tasks: The experiments cover Deformed MNIST, MUG facial expressions, MAFL and CelebA faces, and uncontrolled in-the-wild face variation.They evaluate alignment, appearance inference, latent manifolds, intrinsic decomposition, and landmark detection.
- Unsupervised appearance inference: With appearance latent dimensions of 1 or 0, the model infers canonical digit or facial-expression appearances and places expression variation in shape.Interpolating deformation representations produces meaningful facial-expression morphs.
- Unsupervised appearance inference: Class-aware Deforming Autoencoders improve multi-modal appearance learning by generating a canonical appearance for each implicitly classified class.Misclassified inputs can be decoded into an incorrect class.
- Unsupervised alignment: On MAFL faces and 11k Hands, the network aligns images to common texture spaces while retaining lighting, gender, facial hair, and other input information.The resulting correspondences support applications such as computer-graphics UV mapping.
- Latent manipulation: Deformation-only traversal changes face shape and pose while preserving texture, whereas texture traversal transfers pose-aligned appearance; joint traversal produces smooth image transitions.Compared with a plain autoencoder, the deforming model generates sharper facial features and fewer artifacts.
- Intrinsic decomposition: Intrinsic-DAE generates convincing shading and albedo estimates without direct supervision, while a baseline intrinsic design fails to produce plausible components.Manipulating the shading latent representation simulates interpolated lighting directions without explicitly learning geometry.
4 Conclusion and Future Work
The paper concludes that Deforming Autoencoders disentangle shape and appearance, support unsupervised alignment and landmark localization, and enable canonical-coordinate image manipulation.
- Conclusion: Deforming Autoencoders disentangle shape and appearance in latent space and support unsupervised groupwise image alignment.Experiments include expression morphing, shape and appearance interpolation, and unsupervised landmark localization.
A.1 Convolutional Encoders and Decoders
The convolutional implementation uses matching encoder and decoder families composed of convolutional, normalization, activation, and output-thresholding layers.
- Notation: Conv layers produce feature maps, ConvT layers provide transposed convolutions, BN denotes batch normalization, Nz is latent dimension, and Nc is output-channel count.The decoder uses transposed convolutions to reconstruct the output image.
A.2 DenseNet-style Encoders and Decoders
The DenseNet-style variant replaces plain convolutional stacks with dense encoder and decoder blocks connected by transition operations, with tensor dimensions documented in accompanying tables.
- Dense encoder: The dense encoder combines DBE blocks with TBE transition blocks and max-pooling, ending in a Nz-dimensional Sigmoid latent representation.The listed transitions expand channels from 32 through 256 before producing Nz latent channels.
- Dense decoder: The dense decoder mirrors this structure with DBD blocks and TBD transition blocks, ending in Nc output channels followed by Threshold(0,1).Decoder transitions use 4 × 4 convolutions with stride 2.
- Tensor sizes: Tables 3 and 4 specify intermediate tensor shapes for the convolutional and dense encoder-decoder architectures.Each output shape is represented as height × width × channels.
B Ablation Study
The ablation study varies the texture-latent dimensionality on deformed digits and in-the-wild faces, revealing how this choice affects alignment and appearance representation.
- The experiments use single deformed MNIST 3 images and unmasked in-the-wild MAFL faces to assess the effect of varying ZT.
- On deformed MNIST 3 images, increasing texture-latent dimensionality changes the learned balance between shared texture, pose alignment, and stroke-width variation.At 0 dimensions, texture is identical across images; at 1 dimension, pose aligns while stroke width varies, with further increases learning more appearance variation.
B.2 Methods for deformation modeling
The deformation-modeling ablation compares direct non-rigid warping with affine and integral warping modules, showing how global and local alignment roles combine.
- Affine warping handles global pose variation but cannot align local non-rigid facial structures such as eyes, noses, and landmarks.
- Combining affine and integral warping improves non-rigid face alignment over either module’s isolated behavior.
- Compared with directly predicting a non-rigid warping field from a convolutional decoder, the proposed affine-plus-integral design yields better reconstructions and visually plausible texture maps.
- Integral warping aligns faces non-rigidly, but can exaggerate deformation and create texture smears around features such as eyebrows.
C Latent Manifold Traversal
Latent-manifold traversal experiments compare Deforming Autoencoders with plain autoencoders and demonstrate smooth manipulation of facial pose, shape, texture, and expression.
- Compared with a plain autoencoder, Deforming Autoencoders produce sharper, more realistic face interpolations that remain on the face manifold.
- The deforming autoencoder also provides better reconstructions and a better learned face manifold than the plain autoencoder.
- Interpolating ZT and/or ZW yields smooth transitions in facial pose, shape, and skin texture.
- On the MUG facial-expression dataset, interpolating deformation representations generates sharp images and smooth transitions between source and target expressions.
D Intrinsic Decomposition with DAE
Intrinsic-DAE separates face appearance into canonical-view shading and albedo while retaining deformation information for reconstruction and lighting manipulation.
- Intrinsic-DAE maps faces under varying pose and illumination to a canonical view, then separates canonical shading and albedo with independent decoders.
- Interpolating the learned shading representation simulates smooth, reasonable lighting-direction changes without explicitly modeling geometry.
- Intrinsic-DAE is trained on 200,000 CelebA images with 16-dimensional albedo and shading latents and a 128-dimensional deformation representation.
- The learned deformation warps aligned shading and albedo back to the original input view.