Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models

Jesse Engel; Matthew Hoffman; Adam Roberts

Source-linked AI summary

Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models

Jesse Engel, Matthew Hoffman, Adam Roberts

arXiv:1711.05772v2Nov 15, 2017cs.LGcs.NEstat.ML

TL;DR

The paper addresses the cost of adding new controls to conditional generative models by learning latent constraints after training an unconditional model. Its critics and optimization or actor mechanisms produce realistic conditional samples, preserve identity during transformations, and support zero-shot rule-based generation, including for discrete musical sequences.

Problem
Adding new conditional controls commonly requires expensive retraining, while users need fine-grained attribute control and identity-preserving modification.
Method
The method learns post-hoc latent critics for desired attributes and realism, then uses gradient optimization or an amortized actor to shift latent samples.
Results
The approach generates realistic conditional VAE images, supports identity-preserving transformations, and performs zero-shot rule-based generation without labeled data or differentiable rewards.
Takeaways & Limitations
Unconditional generative models can serve as reusable latent foundations for new conditional behaviors and controls.

Abstract

from arXiv · show

Deep generative neural networks have proven effective at both conditional and unconditional modeling of complex data distributions. Conditional generation enables interactive control, but creating new controls often requires expensive retraining. In this paper, we develop a method to condition generation without retraining the model. By post-hoc learning latent constraints, value functions that identify regions in latent space that generate outputs with desired attributes, we can conditionally sample from these regions with gradient-based optimization or amortized actor functions. Combining attribute constraints with a universal "realism" constraint, which enforces similarity to the data distribution, we generate realistic conditional images from an unconditional variational autoencoder. Further, using gradient-based optimization, we demonstrate identity-preserving transformations that make the minimal adjustment in latent space to modify the attributes of an image. Finally, with discrete sequences of musical notes, we demonstrate zero-shot conditional generation, learning latent constraints in the absence of labeled data or a differentiable reward function. Code with dedicated cloud instance has been made publicly available (https://goo.gl/STGMGx).

1 INTRODUCTION

The paper proposes enforcing new user-defined constraints after training an unconditional generative model by operating in its latent space. This supports conditional sampling, realism control, identity-preserving transformations, and zero-shot rule-based generation.

1 INTRODUCTION: Post-hoc latent constraints let pre-trained unsupervised generative models enforce new behaviors without retraining for each constraint set.The method separates reconstruction learning from later behavioral constraint learning in the exposed latent space.
1 INTRODUCTION: Critics identify latent regions associated with desired attributes or realism, while gradient optimization or an amortized actor generates high-value samples.The approach can operate even with a nondifferentiable decoder, including discrete sequences.
1 INTRODUCTION: A universal realism constraint addresses the VAE tradeoff between reconstruction quality and sample quality without sacrificing diversity.It requires latent samples to resemble encoded training data rather than prior samples.
1 INTRODUCTION: Minimal latent adjustments enable identity-preserving attribute transformations, such as changing a person’s expression or hair while retaining recognizability.The paper contrasts this behavior with pure GAN-based transformations, which often fail to preserve identity.
1 INTRODUCTION: Rule-based constraints enable zero-shot conditional generation from VAE-generated exemplars without labeled data.The actor-critic pair can learn to satisfy user-specified constraints from reward signals.

2 BACKGROUND

VAEs and GANs provide complementary generative capabilities but face limitations in conditional control, mode coverage, blurriness, and reconstruction quality. The paper motivates latent realism and attribute constraints as a way to improve samples while retaining useful reconstructions and controllability.

2 BACKGROUND: Figure 2 illustrates that σx = 0.1 maximizes the ELBO and improves reconstruction fidelity but reduces sample realism, while actor-based realism shifting restores realistic samples without sacrificing sharpness.The figure also shows that the method maps samples toward nearby latent points satisfying realism and attribute-preservation constraints.
2 BACKGROUND: VAEs often trade sharp reconstructions for sensible samples, while GANs can produce realistic outputs but suffer from mode collapse and limited manipulation coverage.The VAE tradeoff depends on choices such as latent dimensionality and likelihood scale.
2 BACKGROUND: Figure 3 evaluates realism critics in influential low-variance latent dimensions, showing prior samples directed toward realistic digits and held-out data near a critic local maximum.The critic is trained to distinguish encoded data from prior samples and generalizes to held-out encodings.
2 BACKGROUND: Conditional GANs and VAEs require attribute labels for the full training set and generally need retraining to adapt to new attributes.They also inherit mode-collapse or blurriness problems from their unconditional counterparts.
2 BACKGROUND: The proposed approach trains an unconditional VAE for good reconstruction, then learns realism and attribute critics to guide latent samples toward desired regions.Generation can use latent optimization or an actor that shifts samples while satisfying the critics.

3 THE “REALISM” CONSTRAINT: SHARPENING VAE SAMPLES

The method defines realism using the VAE’s marginal posterior rather than its prior, then uses critics and optimization or amortized generation to move latent samples toward realistic regions.

Using q(z), the marginal posterior of encoded data, as the realism target closes the gap between reconstruction and sample quality without sacrificing diversity.
A realism critic distinguishes latent samples from q(z) and p(z), while gradient optimization moves prior samples toward points judged more realistic.
Figure 4 combines attribute and realism constraints, with latent-distance regularization producing smaller shifts and greater cross-row diversity but less cross-column diversity.
Amortized actor networks replace the inner optimization loop by learning to shift latent inputs toward high-value regions.
Identity-preserving optimization starts from reconstructed data points and adjusts latent codes using separate attribute and realism critics.

4 ATTRIBUTE CONSTRAINTS: CONDITIONAL GENERATION

Attribute constraints are imposed in the latent space through conditional actor-critic models, enabling controlled generation after the VAE is trained. Distance regularization trades some attribute accuracy and realism for smaller latent shifts and greater diversity.

Conditional latent-space actor-critic models use attribute labels to generate samples that satisfy both desired attributes and the realism constraint.
The smaller CGAN uses about 2884× fewer FLOPs per iteration than the VAE and achieves only slightly worse results than the larger CGAN.
Distance regularization yields slightly worse accuracy, smaller latent shifts, and greater image diversity than the unregularized actor.
No distance penalty produces higher attribute-classification accuracy but shifts samples farther in latent space.
The approach can also produce reasonably good conditional samples from classical autoencoders, although high-frequency spatial artifacts remain.

5 IDENTITY-PRESERVING TRANSFORMATIONS

The paper transforms reconstructed images by jointly optimizing attribute and realism critics in latent space. Because reconstructions remain close to the originals, the resulting attribute changes preserve much of the original identity.

Gradient optimization changes attributes while maintaining realistic images by jointly optimizing attribute and realism constraints from reconstructed latent points.
Held-out-image transformations preserve much of the original structure because the VAE reconstructions remain close to the input images.
Figure 6 controls musical pitch class and note density in 16-bar pianorolls, placing notes within C Major while optionally increasing density.

6 RULE-BASED CONSTRAINTS: ZERO-SHOT CONDITIONAL GENERATION

The method enables zero-shot conditional generation by learning latent constraints from user-supplied rule-based rewards, without labeled data or differentiable decoders.

A critic approximates reward values in latent space while an actor shifts samples toward high-value states, using a diversity-promoting distance penalty.
The approach supports nondifferentiable reward functions and VAE decoders because the critic learns a value function that guides actor training.
An LSTM VAE represents melodies as categorical sequences and applies reward functions for pitch-class membership and minimum note density.
The actor controls pitch class and note density while eventually settling in high-value latent states requiring only small movements.Figure 6 and Table 2 support the constraint-control result; training alternates between exploration of high-reward modes and contraction toward nearby states.

7 RELATED WORK

Prior work conditions generative models during training or constrains latent representations with classifiers and optimization, whereas this paper uses latent-space actor-critic generation with broader post-hoc flexibility.

Conditional GANs and VAEs require attribute labels during training and generally need retraining for new attributes, while also inheriting mode-collapse and blurriness issues.
Table 2 reports conditional melody generation with high constraint satisfaction, including 70.8% satisfaction for the C Major constraint under the single-constraint actor.
Related methods use latent classifiers, iterative optimization, auxiliary reinforcement losses, or image-space processing to impose properties on generated outputs.
This work differs by learning an amortized generator and discriminator directly in the latent space, extending conditional generation beyond prior optimization-based approaches.

8 DISCUSSION AND FUTURE WORK

The paper presents latent-space constraint enforcement as a general approach and suggests extending it to alternative decoder, density-estimation, and sampling architectures.

The demonstrated approach constrains the latent space of an unconditional generative model to obtain conditional generation.
Future extensions include autoregressive or adversarial decoder costs, explicit constrained-distribution models, and ancestral sampling with autoregressive priors.

9 APPENDIX

The appendix describes image and melody datasets used to evaluate conditional generation, including large-scale MIDI extraction and categorical sequence encoding.

Image experiments use 28 × 28 grayscale MNIST digits and 64 × 64 RGB CelebA images, with digit classes and selected facial attributes as controls.
The melody corpus contains over 3 million unique 16-bar sequences extracted from more than 1.5 million public MIDI files.
Each melody is encoded as 256 categorical variables with 130 states representing pitches, holds, and rests.

9.2 MODEL ARCHITECTURES

The experiments use dataset-specific VAEs plus feed-forward actor and critic networks to impose latent constraints. Actors shift latent vectors toward desired attributes, while critics score realism or attribute satisfaction.

9.2 MODEL ARCHITECTURES: Training uses Adam for encoders, decoders, and classifiers, while critic and actor training adds a gradient penalty and a 10:1 critic-to-generator update ratio.The actor-critic optimization uses separate Adam hyperparameters and the gradient penalty is 10.
9.2.1 MNIST FEED-FORWARD VAE: MNIST uses a feed-forward VAE whose encoder parameterizes a 1024-dimensional diagonal Gaussian latent distribution and whose decoder reconstructs 28×28 images.The encoder and decoder each use three 1024-unit ReLU layers.
9.2.2 CELEBA CONVOLUTIONAL VAE: CelebA uses a convolutional VAE with four stride-2 convolutional encoder layers and transposed-convolutional decoding to produce images.The encoder parameterizes a 1024-dimensional diagonal Gaussian latent distribution.
9.2.3 MELODY SEQUENCE VAE: Melody modeling uses a hierarchical LSTM sequence VAE, while all actor experiments use a deep feed-forward network.The hierarchical decoder models structure first at the bar level and then at the note level.
9.2.4 ACTOR FEED-FORWARD NETWORK: The actor transforms latent vectors by predicting shifts and gates, interpolating between the original z and a predicted δz.This makes the network predict latent adjustments rather than entirely new latent vectors.
9.2.5 CRITIC FEED-FORWARD NETWORK: The critic maps each latent vector to a scalar sigmoid score, with attribute labels optionally concatenated to the latent input.The same feed-forward critic design is used for conditional and unconditional scoring.

9.3 SUPPLEMENTAL FIGURES

Supplemental figures visualize additional conditional samples, latent-space optimization, architectures, and training behavior. They also compare model sizes, latent constraints on a vanilla autoencoder, and decoder-variance effects.

Optimization and training: Latent optimization progressively updates prior samples over 100 steps to satisfy both realism and attribute constraints.Images are shown at steps 0, 10, 30, 50, and 100.
Identity-preserving transformations: Removing the distance penalty lets a CGAN actor match attributes while changing identity, whereas the procedure without such a penalty can return the reconstructed image.The figure contrasts identity distortion with attribute matching.
Melody generation: The melody actor-critic training curves track a pitch-class constraint for melodies in C major.The figure labels the actor as G and critic as D.
Model-size comparison: Smaller actor and critic networks produce comparable-quality samples with 85× fewer parameters when λdist = 0.0.The smaller models use three ReLU layers of 256 units each.
Alternative autoencoder: Applying latent constraints to a vanilla autoencoder yields samples similar in quality to VAEs with σx = 0.1, but with less diversity and more high-frequency artifacts.The comparison concerns sample quality, diversity, and visual artifacts.
Latent variance: Smaller decoder standard deviations produce lower-variance encoder posteriors, and distance regularization is scaled by the square inverse of variance per latent dimension.The figure orders dimensions from lowest to highest variance and links tighter posteriors to greater latent-dimension utilization.