PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders

Ananya Harsh Jha, Saket Anand, Maneesh Singh, V. S. R. Veeravasarapu

arXiv:1804.10469v1cs.CV

TL;DR

Disentangled representations could support targeted data augmentation, but learning them is difficult when factor labels are hard to obtain. This paper introduces cycle-consistent VAEs that separate specified and unspecified factors using pairwise similarity labels; across three datasets, the proposed model is reported to leak the least specified-factor information into the unspecified subspace among the compared models.

  • Problem

    Learning disentangled image representations for targeted data augmentation is challenging when factors of variation are difficult or cumbersome to label.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces a cycle-consistent VAE that partitions latent variables into specified and unspecified subspaces and uses pairwise similarity labels for weak supervision.

    Source: §1 · p. 2, §3.2 · pp. 6–7

  • Results

    Across three datasets and latent-space dimensionalities, the proposed architecture leaks the least specified-factor information into the unspecified subspace among the three compared models.

    Source: §4.1 · p. 10

  • Takeaways & Limitations

    The model separates task-associated factors from remaining variation while supporting controlled generation through its specified and unspecified latent subspaces.

    Source: §3.2 · p. 6, §5 · p. 13

    VAE-based generative models, including this model, produce less sharp images than GANs.

    Source: §5 · p. 13

Abstract

from arXiv · show

Generative models that learn disentangled representations for different factors of variation in an image can be very useful for targeted data augmentation. By sampling from the disentangled latent subspace of interest, we can efficiently generate new data necessary for a particular task. Learning disentangled representations is a challenging problem, especially when certain factors of variation are difficult to label. In this paper, we introduce a novel architecture that disentangles the latent space into two complementary subspaces by using only weak supervision in form of pairwise similarity labels. Inspired by the recent success of cycle-consistent adversarial architectures, we use cycle-consistency in a variational auto-encoder framework. Our non-adversarial approach is in contrast with the recent works that combine adversarial training with auto-encoders to disentangle representations. We show compelling results of disentangled latent subspaces on three datasets and compare with recent works that leverage adversarial training.

1 Introduction

Disentangled representations can support targeted data augmentation, but nuisance factors and costly or difficult supervision complicate learning them. The paper proposes cycle-consistent VAEs that use pairwise similarity labels to separate specified and unspecified factors, and reports avoiding degenerate solutions while remaining robust to latent-space dimensionality choices.

1 Introduction

  • Disentangled latent representations can support targeted data augmentation by enabling generation from the latent subspace relevant to a task.Nuisance factors can distract a prediction model, while invariant representations or data augmentation are described as ways to address them.Source: §1 · p. 1
  • Vanilla GANs do not support inference over latent variables, while fully supervised DNA-GANs require labels for each factor that can be cumbersome to acquire.Source: §1 · p. 2
  • Prior adversarial auto-encoder methods can produce good-quality novel images, but may leave leakage control to the decoder rather than training the encoder to restrict it.Source: §1 · p. 2
  • A prior non-adversarial approach relies on severely restricting unspecified-space dimensionality, and its results are highly sensitive to that choice.Even slightly plausible results require careful selection of the unspecified-space dimensionality.Source: §1 · p. 2
  • Cycle-consistent VAEs use pairwise similarity labels to disentangle specified and unspecified factors without requiring actual class labels.The specified factors are associated with available supervision; the remaining factors are modeled in the unspecified subspace.Source: §1 · p. 2
  • The authors report that their architecture avoids degenerate solutions and is robust to choices of dimensionality for both latent subspaces.A degenerate solution occurs when the decoder ignores specified latent variables and instead takes image identity from the unspecified variables.Source: §1 · p. 2

2 Related Work

Related work spans variational and adversarial auto-encoders, cycle-consistent architectures, and methods for separating image factors. The proposed approach builds on prior disentangling architectures while using non-adversarial training and weaker pairwise-similarity supervision.

2 Related Work

  • Variational auto-encoders use an encoder-parameterized approximate posterior and decoder-parameterized likelihood, optimizing a lower bound with a likelihood term and a KL regularizer.The KL term aligns the approximate posterior with the latent prior; sampling from the prior and decoding can generate samples.Source: §2 · pp. 3–4
  • GANs train a generator and discriminator in a min-max game, and DCGANs use CNNs to model complex image distributions.GANs can generate high-quality samples but require training tricks to stabilize learning and avoid mode collapse.Source: §2 · p. 4
  • Cycle-consistency has been applied to machine translation, image correspondences, depth estimation, image-to-image translation, and domain adaptation.This paper applies cycle-consistency in the unspecified latent space to train the encoder to reduce leakage of specified-factor information.Source: §2 · p. 4
  • Prior disentangling research includes invariant representations, identity-and-pose models, and image-generation approaches that lack inference over latent variables.The cited examples include methods using nuisance-variable factoring, modified GANs, and siamese networks over DC-GANs.Source: §2 · p. 5
  • The proposed work builds on Mathieu et al. and Szabó et al. [8], replacing their adversarial training and single-source supervision with a non-adversarial approach using pairwise similarity labels.Source: §2 · p. 5

3 Cycle-Consistent Variational Auto-Encoders

The model partitions a VAE latent space into specified and unspecified factors and trains it with forward and reverse cycles. Pairwise reverse consistency is introduced to prevent degenerate representations.

3.1 Cycle-Consistency

  • Cycle-consistency trains forward and reverse mappings to compose approximately to identity, combining KL-regularized reconstruction with a reverse-cycle loss.The model alternates between forward and reverse losses.Source: §3.1 · pp. 5–6

3.2 Model Description

  • A conditional VAE separates specified factors in s from unspecified factors in z, enabling control through s and sampling from z.The model assumes marginal independence between the two subspaces and uses a standard normal prior for z.Source: §3.2 · p. 6
  • For the forward cycle, the model swaps s between same-class image pairs and reconstructs them, requiring only pairwise similarity labels rather than class labels.The forward cycle alone can permit a degenerate solution in which the decoder reconstructs using only unspecified factors.Source: §3.2 · p. 7, Figure 3 · p. 7

3.3 Preventing a Degenerate Solution

  • For the reverse cycle, one sampled z is decoded with specified factors from independently sampled images, and the generated images’ z embeddings should be close.This constraint promotes marginal independence and encourages z to retain only information about unspecified factors.Source: §3.3 · p. 8, Figure 4 · p. 8
  • The pairwise reverse loss addresses non-unique mappings of similar unspecified factors and was easier to minimize than the absolute counterpart in loss value and disentanglement.The pairwise loss penalizes large distances between generated images’ unspecified latent embeddings without requiring them to remain close to the sampled z.Source: §3.3 · p. 8

4 Experiments

The experiments evaluate disentanglement and image generation across MNIST, 2D Sprites, and LineMod. The proposed model shows the least specified-factor leakage into z across datasets and latent-space dimensions, alongside image-generation evaluations using latent swapping, interpolation, and prior sampling.

4 Experiments

  • Evaluation covers MNIST digit identity, 2D Sprites character identity, and synthetic LineMod object identity, including tests on sprite characters unseen during training.The evaluation also includes image-generation comparisons with prior works and [8].Source: §4 · p. 9, §4.2 · p. 12

4.1 Quality of Disentangled Representations

  • Two-layer classifiers on s and z embeddings assess factor information; ideal disentanglement means perfect specified-factor accuracy in s and near-chance accuracy in z.The experiments also examine changes in latent-space dimensionality.Source: §4.1 · pp. 9–10
  • Across all three datasets and latent-space dimensions, Cycle-Consistent VAE has the lowest specified-factor classification accuracy in z and comparable accuracy in s.The authors attribute reduced leakage to the reverse cycle-consistency loss.Source: §4.1 · p. 10, Table 1 · p. 10
  • For MNIST t-SNE, adversarial-only embeddings cluster by digit [8], while Mathieu et al. and this model show similar class confusion; the quantitative table better captures performance differences.The plots color points by digit identity, the specified factor.Source: §4.1 · pp. 10–11, Figure 5 · p. 11
  • The authors conjecture adversarial joint updates may leave encoders less disentangled; VAE reparameterization promotes class confusion, while reverse-cycle loss further reduces specified-factor information in z.They also note that Szabó et al. [8] can generate novel images despite specified-factor information remaining in z.Source: §4.1 · p. 11

4.2 Quality of Image Generation

  • Swapped-latent grids combine z from the first column with s from the top row; the proposed model avoids degeneracy at substantially higher latent dimensions than base values.For LineMod, test images are held constant across model grids for direct comparison.Source: Figure 6 · p. 12, §4.2 · pp. 12–13
  • The experiments also use linear interpolation to inspect both latent manifolds and conditionally generate images by fixing s and sampling z from its Gaussian prior.The figures show interpolation and prior-sampling examples for the model.Source: §4.2 · pp. 12–13
  • On LineMod, unspecified-factor transfer cannot be interpreted directly as viewpoint transfer because objects are not consistently aligned and viewpoint is not the only such factor.The authors keep test images constant across model grids to compare transfer directly.Source: §4.2 · pp. 12–13

5 Conclusion

The paper concludes that cycle-consistency enables weakly supervised disentanglement with compelling results across three datasets. It also notes a sharpness limitation of VAE-generated images and identifies possible directions for improvement and further study.

5 Conclusion

  • The proposed cycle-consistent architecture uses pairwise weak supervision, avoids degenerate solutions, and is robust to latent-space dimensionality choices.The authors report compelling quantitative results on three datasets and good image-generation capabilities.Source: §5 · p. 13
  • VAE-based generations, including this model, are less sharp than GAN-generated images.The authors suggest adversarial decoder training with reverse cycle-consistency loss as a possible sharpness improvement that may preserve encoder disentanglement.Source: §5 · pp. 13, 15
  • For LineMod, transfer of unspecified factors does not directly correspond to viewpoint transfer.The paper also notes that the assumed viewpoint-only variation does not hold for these objects.Source: Figure 8 · p. 14
  • The paper identifies unsupervised disentanglement as a direction for further exploration.It also proposes adversarial decoder training after cycle-consistent VAE training as a possible way to improve image sharpness.Source: §5 · p. 15

Algorithm

Training alternates forward and reverse cycles. The forward cycle reconstructs paired images, while the reverse cycle checks recovery of unspecified latents after decoding with shared random latents.

Algorithm

  • The forward cycle samples a pair according to similarity labels, encodes each image, and samples its unspecified latent from the encoder’s approximate posterior.The encoded specified and unspecified embeddings are computed for both images.Source: Algorithm · p. 17
  • Each forward-cycle reconstruction combines one image’s sampled unspecified latent with the other image’s specified embedding, then incurs KL-divergence and L2 reconstruction losses.The losses are computed independently for the two posterior parameter pairs and between each reconstruction and its source image.Source: Algorithm · p. 17
  • The forward-cycle gradients train both the encoder and decoder, whereas the reverse-cycle gradients train only the encoder.The reverse-cycle training uses the computed unspecified embeddings and an L1 reconstruction loss on the re-encoded means.Source: Algorithm · p. 17
  • The reverse cycle decodes a shared random unspecified latent with each image’s specified embedding, then re-encodes the reconstructions to obtain unspecified latent embeddings.It samples two images and a latent from a standard normal distribution.Source: Algorithm · p. 17

Network Architectures

The encoder branches from a shared convolutional trunk to produce specified and unspecified embeddings. The decoder combines those embeddings before upsampling to an image.

Network Architectures

  • The encoder uses a shared convolutional trunk that splits into fully connected branches for specified and unspecified factors.The unspecified branch outputs mean and variance to parameterize its approximate posterior.Source: Network Architectures · p. 17
  • The decoder takes z and s through separate fully connected branches, concatenates and reshapes them, then upsamples through convolutional blocks.The convolutional blocks use convolution, instance normalization, and ReLU layers.Source: Network Architectures · p. 18
  • MNIST uses three encoder and decoder convolutional blocks; 2D Sprites and LineMod use four each, with 5-pixel filters and stride 2.MNIST images are 28x28x1, while 2D Sprites and LineMod images are 64x64x3.Source: Network Architectures · p. 18

t-SNE Plots

The t-SNE plots visualize unspecified latent spaces with points colored by specified-factor labels. They contrast label clustering and class confusion across methods and datasets.

t-SNE Plots

  • The plots show unspecified latent spaces for 2D Sprites and LineMod, with points color-coded by specified-factor labels.Source: t-SNE Plots · p. 18, Figure 2 · p. 19
  • On 2D Sprites, class clusters in the baseline plot indicate specified-factor information in the unspecified space, while the other two plots show class confusion.The 480 characters are grouped into 10 broad classes based on gender and body type.Source: t-SNE Plots · p. 18
  • For LineMod, adversarial training alone yields clusters by class identity, while Mathieu’s re-parameterization and the proposed reverse-cycle loss show class confusion in z.The proposed approach combines re-parameterization with reverse-cycle loss to create confusion regarding specified factors.Source: Figure 1 · p. 18
  • The 2D Sprites baseline’s unspecified space has high classification accuracy despite lacking clear visual clusters, suggesting the classes remain separable.The plot’s sparseness contrasts with the classification result.Source: t-SNE Plots · pp. 18–19

Related papers

Loading 1804.10469v1…