Source-linked AI summary
Guided Image Generation with Conditional Invertible Neural Networks
Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, Ullrich Köthe
TL;DR
The paper addresses conditional natural image generation where outputs should remain diverse and realistic despite ambiguous conditioning inputs. It introduces cINNs, which combine invertible generative modeling with a learned conditioning network and stable joint maximum-likelihood training. Across MNIST generation and ImageNet colorization, cINNs produce diverse, sharp images and support intuitive latent-space manipulation.
Problem
Conditional image generation requires modeling diverse plausible outputs while avoiding the mode collapse and blurriness associated with common conditional generative approaches.
Method
cINNs combine an invertible neural network with an unconstrained feed-forward conditioning network, jointly optimized with maximum likelihood.
Results
cINNs generate diverse, sharp conditional images for MNIST and ImageNet colorization while enabling latent-space exploration and manipulation.
Takeaways & Limitations
The bidirectional cINN architecture provides a conditional generative model that supports diverse image synthesis and intuitive latent-space manipulation.
Takeaways & Limitations
Effective conditioning in complex settings may require extracting higher-level features rather than feeding the condition directly into each cINN block.
Abstract
from arXiv · showhide
In this work, we address the task of natural image generation guided by a conditioning input. We introduce a new architecture called conditional invertible neural network (cINN). The cINN combines the purely generative INN model with an unconstrained feed-forward network, which efficiently preprocesses the conditioning input into useful features. All parameters of the cINN are jointly optimized with a stable, maximum likelihood-based training procedure. By construction, the cINN does not experience mode collapse and generates diverse samples, in contrast to e.g. cGANs. At the same time our model produces sharp images since no reconstruction loss is required, in contrast to e.g. VAEs. We demonstrate these properties for the tasks of MNIST digit generation and image colorization. Furthermore, we take advantage of our bi-directional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style in an intuitive way.
1 INTRODUCTION
The paper proposes cINNs for conditional image generation, combining invertible generative modeling with learned conditioning features. The approach targets stable, diverse, sharp generation while enabling latent-space exploration and manipulation.
- Advantages over alternatives: INNs avoid reconstruction losses because reconstructions are perfect by design, preventing the blur associated with reconstruction-based objectives.Each data point maps to exactly one latent representation, avoiding posterior regions that can be disjoint or overlapping.
- Advantages over alternatives: cINNs aim to retain training stability and sample diversity while producing higher-quality images than autoencoder architectures.The introduction contrasts these properties with cGAN mode collapse and cVAE-induced blurriness.
- Applications: The conditioning architecture alleviates INN restrictions on standard components by preprocessing conditional inputs with a standard feed-forward network.The paper demonstrates cINNs on conditional MNIST generation and diverse ImageNet colorization.
- Architecture and training: cINNs combine an invertible neural network with an unconstrained feed-forward conditioning network.The conditioning network preprocesses inputs into features and is jointly optimized with the INN.
- Architecture and training: Maximum likelihood training jointly optimizes the INN and conditioning network using a stable procedure.
- Applications: The bidirectional architecture supports exploring and manipulating emergent latent-space properties for MNIST generation and image colorization.
2 RELATED WORK
Related work spans GANs, VAEs, autoregressive models, CRFs, and invertible networks for conditional generation. The paper positions cINNs as a flow-based alternative that produces diverse, spatially coherent colorizations with tractable likelihood training.
- Conditional generative models: cGANs often struggle to maintain diversity for fixed image conditions, while large-scale remedies can require substantial computational resources.BigGANs address diversity with very large networks and batch sizes, requiring up to 512 TPUs.
- Conditional generative models: cVAEs commonly produce blurry images because Gaussian variational distributions make the reconstruction penalty equivalent to squared loss.
- Invertible neural networks: The paper argues that maximum likelihood alone can train high-quality generative models and presents coupling layers for conditional image generation.The coupling-layer architecture provides tractable Jacobian determinants for maximum likelihood training.
- Colorization: Existing diverse-colorization methods independently sample local pixel-color histograms, requiring heuristic post-processing to avoid spatial artefacts.
- Conditional generative models: Autoregressive models and CRFs can address conditional generation but are computationally expensive and often cannot scale to realistic image sizes.
- Colorization: Prior encoder-generator colorization work was limited to cars and three latent dimensions, yielding global but not local diversity.
- cINN positioning: The flow-based cINN generates diverse colorizations in one feed-forward pass, models all pixels jointly, and supports meaningful latent-space manipulations.
3 METHOD
The cINN extends invertible affine coupling blocks with conditional inputs and a feed-forward conditioning network, enabling tractable maximum-likelihood training and conditional generation. Architectural and training adjustments target stable optimization, effective conditioning, and diverse, sharp samples without mode collapse.
- 3.1 CONDITIONAL INVERTIBLE TRANSFORMATIONS: Affine coupling blocks split u into u1 and u2, apply alternating affine transformations, then concatenate [v1, v2] for subsequent blocks.The internal s_j and t_j networks need only forward evaluation, including when the block is inverted.
- 3.1 CONDITIONAL INVERTIBLE TRANSFORMATIONS: Conditioning data c enters the non-inverted s_j and t_j subnetworks, preserving invertibility while adapting the coupling block to conditional generation.A separate feed-forward network h can transform c into features that are supplied to the cINN.
- 3.2 MAXIMUM LIKELIHOOD TRAINING OF CINNS: The cINN assigns conditional likelihoods through a change-of-variables formula using a prescribed latent distribution and the Jacobian determinant.The resulting objective combines maximum-likelihood loss with L2 weight regularization, and generation samples z before applying the inverse g.
- 3.2 MAXIMUM LIKELIHOOD TRAINING OF CINNS: Maximum-likelihood training makes mode collapse virtually impossible because low-probability training modes produce latent vectors far outside the prescribed normal distribution and incur large loss.This directly penalizes ignoring a mode rather than relying on the weaker discriminator signal described for GANs.
- 3.3 CONDITIONING NETWORK: A conditioning network h extracts higher-level features from c so each coupling block need not relearn them, and h can be pretrained or optimized jointly with the cINN.This is useful when effective conditioning requires global semantic information, such as information from an image.
4 EXPERIMENTS
Experiments evaluate cINNs on MNIST digit generation and natural-image colorization, including latent-space manipulation, architectural choices, and comparisons with conditional GANs. The results show shared style representations, strong diversity, and training improvements that support stable generation.
- Class-conditional MNIST generation: On MNIST, fixing latent code z while changing digit condition c produces different digits in the same writing style.The learned latent space also contains interpretable axes for width, thickness, and slant, enabling style analysis and transfer.
- Class-conditional MNIST generation: MNIST style transfer encodes a validation image into z and decodes it with different digit conditions to generate other digits in the same style.This uses the cINN’s bidirectional mapping between images and latent codes.
- Diverse ImageNet colorization: For ImageNet colorization, the cINN conditions coupling blocks on VGG-derived features and uses multistage resolution changes with early latent-code splitting.Haar wavelet downsampling reduces spatial dimensions while separate convolutional heads tailor conditioning to individual coupling blocks.
- Diverse ImageNet colorization: The cINN achieves the best diversity and best-of-8 accuracy among compared methods, while the cGAN shows no significant diversity because it apparently ignores the latent code.The model is trained on 256 × 256 grayscale conditioning images and generates 64 × 64 color information.
- Diverse bedrooms colorization: On LSUN bedrooms, the cINN trains in under 4 hours on one GTX 1080Ti, whereas colorGAN requires over 24 hours and multiple restarts for stable convergence.The colorGAN results are generally worse, although its pixel-wise color variance is slightly higher and may reflect unrealistic colorful outputs.
- Ablation of training improvements: At Adam learning rates of 10^-3, clamping and Haar wavelet downsampling are strictly necessary for stable training, while noise augmentation and permutations yield the largest final improvements.Initialization improves the final result only slightly but accelerates convergence noticeably.
5 CONCLUSION AND OUTLOOK
The proposed cINN enables guided generation of diverse, realistic images, while its conditioning network supports broader computer-vision applications.
- cINNs enable guided generation of diverse images with high realism.The authors suggest that the non-invertible conditioning network makes cINNs suitable for tasks such as diverse semantic segmentation.