Source-linked AI summary
Disentangling factors of variation in deep representations using adversarial training
Michael Mathieu, Junbo Zhao, Pablo Sprechmann, Aditya Ramesh, Yann LeCun
TL;DR
The paper addresses disentangling label-associated and unspecified variation when unspecified-factor labels or aligned observations are unavailable. It introduces a conditional VAE-GAN model using same-class distinctions and adversarial training, and reports disentanglement, analogy resolution, and generalization across synthetic and real datasets, while noting that evaluation remains limited to constrained examples.
Problem
Real-world observations combine hidden factors, but rich labels for their separate influences and aligned unspecified factors are usually unavailable.
Method
A conditional generative model separates label-associated variation from remaining variability using complementary codes, VAEs, GANs, and same-class observations.
Results
Across synthetic and real datasets, the model separates specified and unspecified information, resolves analogies, and generalizes to unseen classes or variabilities in tested settings.
Takeaways & Limitations
The learned representations support single-image analogies and transfer of factors such as identity, position, and illumination without requiring matching observations.
Takeaways & Limitations
The initial study tests the model only on rather constrained examples, and adding GAN loss for additional samples made the system unstable without perceptual improvement.
Abstract
from arXiv · showhide
We introduce a conditional generative model for learning to disentangle the hidden factors of variation within a set of labeled observations, and separate them into complementary codes. One code summarizes the specified factors of variation associated with the labels. The other summarizes the remaining unspecified variability. During training, the only available source of supervision comes from our ability to distinguish among different observations belonging to the same class. Examples of such observations include images of a set of labeled objects captured at different viewpoints, or recordings of set of speakers dictating multiple phrases. In both instances, the intra-class diversity is the source of the unspecified factors of variation: each object is observed at multiple viewpoints, and each speaker dictates multiple phrases. Learning to disentangle the specified factors from the unspecified ones becomes easier when strong supervision is possible. Suppose that during training, we have access to pairs of images, where each pair shows two different objects captured from the same viewpoint. This source of alignment allows us to solve our task using existing methods. However, labels for the unspecified factors are usually unavailable in realistic scenarios where data acquisition is not strictly controlled. We address the problem of disentanglement in this more general setting by combining deep convolutional autoencoders with a form of adversarial training. Both factors of variation are implicitly captured in the organization of the learned embedding space, and can be used for solving single-image analogies. Experimental results on synthetic and real datasets show that the proposed method is capable of generalizing to unseen classes and intra-class variabilities.
1 Introduction
Disentangling factors of variation is difficult because real-world observations combine influences that are rarely richly labeled. The paper seeks representations that separate task-relevant variation from other variation rather than discarding the latter.
- 1 Introduction: Real-world sensory observations combine multiple hidden factors whose influences are difficult to label explicitly.Examples include speaker attributes, intonation, and words in speech, and object properties and viewing conditions in images.
- 1 Introduction: Supervised representations often become invariant to task-irrelevant variation, discarding information that may matter for other tasks.
- 1 Introduction: Generative applications require preserving all factors of variation while disentangling them for operations such as voice transfer and image restoration.
- 1 Introduction: The paper introduces a deep conditional generative model that separates label-associated factors from other variability using only same-label observation distinctions.The approach combines Variational Auto-Encoders and Generative Adversarial Networks.
2 Related work
Prior work separates content and style or learns invariant representations, often using specialized supervision or synthetic data. The paper positions its approach as avoiding matching observations and unspecified-factor labels.
- 2 Related work: Earlier approaches use bilinear models, autoencoders, regularization, restricted Boltzmann machines, and VAEs to separate or preserve different variation factors.
- 2 Related work: Computer-graphics methods recover abstract attributes such as orientation and lighting from renderings, but benefit from synthetic data that enables strong supervision.
- 2 Related work: Fair representation methods seek invariance to nuisance factors while retaining remaining information, including through independence between latent factors.
- 2 Related work: Unlike an analogy method requiring matching examples, this approach requires only class identities and no matching observations or unspecified-factor labels.
3 Background
The background combines VAE latent-variable modeling with conditional GAN training. VAEs encode observations into approximate posteriors and optimize reconstruction plus prior regularization, while GANs use a discriminator-generator game for realistic conditional samples.
- 3 Background: VAEs model observations with independent latent variables through a prior and a decoder-defined conditional likelihood.
- 3 Background: A learned encoder defines an approximate Gaussian posterior, and training minimizes reconstruction error together with KL regularization toward the prior.
- 3 Background: GAN training forms a min-max game in which a discriminator distinguishes real from generated samples and a generator attempts to confuse it.
- 3 Background: Conditioning both discriminator and generator on class labels enables generation of samples belonging to a specified class or possessing a desired property.
4 Model
The model uses a specified code for label-associated variation and a continuous latent code for remaining variability. It prevents collapse of the specified code by combining VAE objectives with adversarial training on swapped components without requiring aligned unspecified factors.
- 4 Model: The conditional model represents specified variation with s and remaining variability with an independent continuous latent variable z.The specified code is continuous rather than one-hot to support generalization to unseen identities.
- 4 Model: A decoder generates x from s and z, while sampling z from N(0, I) produces an instance compatible with a given specified component.
- 4 Model: The model obtains s from same-class images with a deterministic encoder and uses another encoder to parameterize q(z | x, s).The encoders may share parameters, and a single encoder can output both specified and unspecified components.
- 4 Model: Without additional constraints, information can flow entirely through z, allowing the decoder to ignore s and mapping same-class images to different latent regions.
- 4 Model: Adversarial regularization replaces unavailable aligned-factor reconstruction after swapping components, combining the variational objective with λLgan.The procedure swaps components across observations and alternates model and adversary updates.
- 4 Model: The GAN term compensates for the lack of aligned training data and also produces sharper images as a side effect.Applying GAN loss to additional samples made the system unstable without improving perceptual quality.
5 Experiments
Experiments evaluate disentanglement qualitatively through image swapping, retrieval, interpolation, and sampling, and quantitatively through classification of specified and unspecified representations. Across synthetic and real datasets, the model resolves analogies and can generalize to unseen sprites, while limitations appear with small or highly variable NORB identities.
- Evaluation: The evaluation combines qualitative image generation and analogy tests with quantitative classification-based disentanglement analysis.Qualitative settings include swapping, retrieval, interpolation, and conditional generation; quantitative evaluation predicts structured labels, object categories, or subject identity from learned components.
- MNIST: MNIST experiments transfer handwriting style without matched examples while learning a smooth digit representation.The specified component is the digit class, and the experiments cannot test generalization to unseen identities.
- Sprites: Sprites experiments show successful unseen-sprite generation, sharp images, resolved single-image analogies, and smooth transitions between identities or positions.The dataset contains 672 characters with 20 animations each and seven sources of variation; interpolation can show abrupt changes because positions are discrete.
- NORB: NORB experiments resolve analogies, but reconstruction quality degrades for highly variable classes and symmetric objects, plausibly because training identities are few.Training and testing use distinct object identities, making generalization more difficult; the authors conjecture that more data could eliminate these problems.
- Extended-YaleB: Extended-YaleB experiments satisfactorily resolve analogies, transferring position and illumination even when those positions were absent during training.The dataset contains images of 28 individuals, but its small number of identities prevents testing generalization to unseen identities.
- Influence of components: Removing adversarial regularization prevents the model from learning disentangled representations, whereas pretrained identity embeddings retain disentanglement but compromise image and analogy quality.Joint end-to-end training also uses fewer parameters through encoder parameter sharing.
6 Conclusions and discussion
The paper presents a VAE–GAN conditional generative model that disentangles label-associated and unspecified variation without strong supervision. Results are promising but limited by constrained evaluations, high sample complexity, and no within-component disentanglement.
- The model separates factors specified by categorization from unspecified variability using a conditional generative model combining VAE and GAN.
- The approach resolves analogies consistently across several datasets with minimal parameter or architecture tuning.
- The initial evaluation covers relatively constrained examples despite motivation for more realistic settings.
- Under weak supervision, the method appears to require training samples covering the full ranges of specified and unspecified variation.
- The model does not attempt to disentangle variations within either the specified or unspecified component.
Network architectures
The encoder uses a shared convolutional sub-network that splits into two separate branches.
- The encoder consists of a shared sub-network that splits into two separate branches.
Image generation
Image generation extracts the specified component from a data sample and samples the unspecified component to generate within-category variation.
- The specified part is extracted from a data sample, while the unspecified part is sampled from a Gaussian distribution.
- The generated samples vary within the category defined by the specified part.
- Figure 8 illustrates generation by combining the specified part from left images with a sampled unspecified part for right-hand images.
Interpolation
Interpolation results independently interpolate the specified and unspecified parts extracted from two images.
- The specified and unspecified parts extracted from two images are interpolated independently.
Using a pre-trained embedding
The pre-trained embedding scheme first learns the specified code with supervised classification, then adds an encoder for the unspecified code while freezing the specified encoder. Its behavior is examined through interpolation and swapping grids, including a comparison with joint training.
- Two-step training: The specified encoder is first trained with a two-layer classifier and classification loss, then its weights are frozen.This establishes a supervised embedding for the samples’ classes before learning the unspecified code.
- Two-step training: An additional encoder is then added to produce the unspecified code while the specified encoder remains fixed.The system is subsequently trained as before with the new unspecified-code encoder.
- Embedding visualizations: Interpolation grids on YaleB vary the specified part along lines and the unspecified part along columns.The figure uses only the top-left and bottom-right real test faces as endpoints.
- Embedding visualizations: Generation grids visualize swapping the specified and unspecified parts, following the arrangement used in an earlier figure.
- Embedding visualizations: Swapping grids compare pre-training the specified encoder on a supervised task with jointly training the whole system.The comparison is presented as two alternatives for learning the specified and unspecified parts.
Training procedure
The full procedure alternates generative-model and adversary training over sampled labeled examples. It encodes same-label and different-label samples, reconstructs and generates observations, and backpropagates reconstruction and adversarial losses.
- Overview: Algorithm 1 summarizes the full model-training procedure and refers to Sections 3 and 4 for notation definitions.
- Generative-model training: Each iteration begins by sampling a triplet in which x1 and x′1 share a label while x2 has another label.
- Generative-model training: The encoder computes µ, σ, and s codes for all three samples, and latent variables are sampled from the corresponding Gaussian parameters.
- Generative-model training: The decoder reconstructs x1 using latent content with either specified code s1 or s′1, and reconstruction losses are backpropagated.
- Generative-model training: The decoder generates samples by combining z1 with s2 or a random latent z, while adversarial losses are backpropagated with the adversary frozen.
- Adversary training: The adversary-training stage samples a pair of labeled observations before computing its training quantities.