Generative Models of Visually Grounded Imagination

Ramakrishna Vedantam; Ian Fischer; Jonathan Huang; Kevin Murphy

Source-linked AI summary

Generative Models of Visually Grounded Imagination

Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, Kevin Murphy

arXiv:1705.10762v8May 30, 2017cs.LGcs.CVstat.ML

TL;DR

Generating images for novel, partly specified visual concepts requires models that support compositional abstraction and meaningful diversity. This paper extends VAEs with a product-of-experts inference network and 3 C’s evaluation, outperforming prior methods on MNIST-A and CelebA.

Problem
Most conditional image models assume fully specified attributes, limiting generation from abstract concepts with unspecified attributes.
Method
The method extends multimodal VAEs with a new training objective, product-of-experts inference for missing attributes, and correctness, coverage, and compositionality metrics.
Results
The proposed method outperforms previous methods on the MNIST-with-attributes and CelebA datasets.
Takeaways & Limitations
The paper demonstrates generative models that imagine compositionally novel concrete and abstract visual concepts.
Takeaways & Limitations
A compared method can generate mean images rather than diverse concept instances, with broader coverage requiring a tradeoff against its KL penalty.

Abstract

from arXiv · show

It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C's). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et.al. and the BiVCCA method of Wang et.al.) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset.

1 INTRODUCTION

The paper develops a multimodal VAE framework for visually grounded imagination over compositional attribute concepts, including partially specified concepts. It introduces the TELBO objective, product-of-experts inference, 3 C’s evaluation criteria, and experiments showing improvement over prior methods on MNIST-A and CelebA.

Motivation: The paper frames visual understanding as generating mental images from descriptions of visual concepts and testing whether those images match the intended concept.A listener represents a description in a shared latent space and produces images that can be compared with the concept.
Problem Setting: Concepts are encoded as fixed-length discrete attribute vectors, enabling compact representation of exponentially many concepts across a compositional abstraction hierarchy.Partially specified vectors represent abstract concepts, while fully specified vectors represent concrete concepts.
Method: The method extends multimodal VAEs with the TELBO training objective and inference networks that embed images and descriptions into a shared latent space for cross-modal translation.It models p(x, y, z) = p(z)p(x|z)p(y|z) and supports paired training with image-only or description-only inference at test time.
Method: A product-of-experts attribute inference network handles missing attributes, producing broader image distributions for abstract concepts and narrower distributions as more attributes are specified.With no specified attributes, the posterior equals the prior; conditioning on additional attributes narrows the posterior.
Evaluation: The paper evaluates visual imagination using correctness, coverage, and compositionality, then compares its method with prior approaches on MNIST-A and CelebA, where it outperforms them.MNIST-A varies digit location, orientation, and size on a 64x64 canvas, while CelebA contains over 200k face images annotated with 40 binary attributes.

2 METHODS

The method extends VAEs to jointly model images and attributes, trains aligned and unimodal inference networks with a TELBO objective, and handles missing attributes using a Gaussian product-of-experts posterior. As more attributes are observed, the posterior becomes narrower, while observing none recovers the prior.

Standard VAEs: The VAE backbone uses a Gaussian standard-normal prior and an encoder optimized through the evidence lower bound, with β controlling pressure toward disentangled latent factors.The default setting is β = λ = 1, while β > 1 encourages proximity to the factorial prior.
Joint VAEs and the TELBO: The joint VAE models images and attributes conditionally independently from a shared latent variable, using a DCGAN image decoder and an MLP attribute decoder.Its training objective is the joint ELBO over paired image-attribute data.
Joint VAEs and the TELBO: The TELBO combines joint and unimodal inference objectives, with likelihood terms scaled by λ and γ and generative decoder terms frozen when retrofitting unimodal inference networks.This permits joint optimization while fitting image-only and attribute-only encoders from the aligned generative model.
Handling missing attributes: For missing attributes, each observed attribute contributes a Gaussian expert, whose product with the prior yields a closed-form Gaussian posterior over the shared latent space.The prior acts as a universal expert, and the posterior remains well-conditioned as attributes are partially specified.
Handling missing attributes: As more attributes are observed, summed precision matrices narrow the posterior; with no observed attributes, the posterior reduces to the prior.This makes posterior specificity increase with the number of observed constraints.

3 EVALUATION METRICS: THE 3C’S OF VISUAL IMAGINATION

The 3C evaluation framework measures visual imagination through correctness, coverage, and compositionality. A fixed attribute classifier evaluates generated images, while compositional splits test generalization to unseen attribute combinations.

Evaluation procedure: A fixed multi-label attribute classifier converts generated images into predicted attributes and serves as a human-observer proxy across evaluated methods.The classifier is trained on a large image-attribute dataset and held constant across methods.
Correctness and coverage: Correctness measures the fraction of specified attributes matched by each generated image, while coverage measures diversity across unspecified attributes.Coverage averages 1 − JS divergence between generated and training-set distributions for missing attributes.
Correctness and coverage: Correctness and coverage are reported separately, with JS-overall optionally combining them for convenient hyperparameter selection.Coverage compares generated and training-set attribute distributions using Jensen–Shannon divergence.
Comparison with inception score: Unlike inception score, the 3C metrics check consistency with the specified concept and adapt diversity to the concept’s abstraction level.Inception score evaluates unconditional image generation through class confidence and marginal diversity.
Compositionality: Compositionality is assessed on test concepts differing from training concepts, requiring correctness on novel attribute combinations that may be impossible.This compositional split is substantially harder than a standard iid split.

4 RELATED WORK

The paper situates its approach among conditional and joint generative models, methods for missing inputs, Gaussian embeddings, and compositional concept representations. It emphasizes representing abstract concepts as latent-space regions while addressing limitations of existing joint and attribute-based methods.

Conditional models: Prior conditional image models generate images from labels, attributes, sentences, or other images, whereas this work learns a shared latent space from descriptions and images.
Joint models: BiVCCA can generate mean images for concepts with multiple associated images, while increasing its compensating parameter reduces the broad posterior needed for concept coverage.
Joint models: JMVAE’s aggregated posterior covers embeddings of all images associated with a concept, but its lack of a prior-matching KL term slightly reduces diversity for novel concepts.
Joint models: SCAN treats attribute vectors as atomic symbols, preventing inference of unseen attribute combinations unless those combinations are paired with images during training.
Handling missing inputs: Existing missing-input strategies can rely on imputation, but conditioning on imputed values differs from omitting inputs, which should instead increase posterior uncertainty.
Gaussian embeddings and abstraction: Unlike point-based image–text embeddings and image-set concept representations, this method maps concepts to latent-space regions that support abstraction and generate new images.

5 EXPERIMENTAL RESULTS

Experiments on MNIST-A and CelebA evaluate TELBO, BiVCCA, and JMVAE using the 3 C’s and missing-attribute generation. TELBO generally provides strong correctness and superior coverage, while all methods face challenges on compositional generalization and rare attributes.

MNIST-A: For familiar concrete concepts, TELBO achieves about 82% correctness on fully specified MNIST-A concepts, with coverage undefined because no attributes are missing.These test concepts were seen during training in the iid split.
MNIST-A: For novel abstract concepts, TELBO maintains correctness, while its coverage exceeds JMVAE and BiVCCA because of its KL regularization.JMVAE correctness drops from about 85% to about 81.5%, whereas TELBO and BiVCCA remain steady.
MNIST-A: On MNIST-A, TELBO and JMVAE outperform BiVCCA on correctness for compositionally novel concrete concepts, although both decline from about 82% to about 75%.The compositional split requires strong generalization; BiVCCA reaches about 70% correctness.
CelebA: On CelebA, TELBO and JMVAE can sometimes generate fairly reasonable images for the unseen concept “bald female,” whereas BiVCCA results are uniformly poor.The reported CelebA examples are cherry picked.
CelebA: When eyeglasses are unspecified on CelebA, nearly all generated samples omit glasses because the attribute is rare, with a prior probability of about 6%.This illustrates how the dataset’s attribute frequency biases missing-attribute generation.

6 CONCEPT NAMING WITH IMAGINATION MODELS

This section evaluates imagination models for concept naming: assigning a concept label to image sets, including cases with partially specified attributes. Concept-Latent inference substantially outperforms Concept-NB, with the best reported accuracy reaching 54.66 ± 4.92%.

Task setup: Concept naming assigns a label to a set of images representing a concept and tests how far to generalize within a compositional abstraction hierarchy.The task is evaluated using sets of positive image examples for concepts with different levels of attribute specification.
Methods: Concept-Latent selects the concept whose latent distribution is closest to the image-set distribution using KL divergence, approximated from a Gaussian mixture.The image-set posterior is formed by combining per-image latent distributions, allowing inference in a lower-dimensional representation.
Evaluation: The MNIST-A study samples five images per concept and measures accuracy across three splits and multiple attribute-missingness patterns.Fully specified labels are used while attributes are dropped to represent different abstraction levels, excluding the case with no specified attributes.
Results: 54.66 ± 4.92% accuracy is achieved by Concept-Latent with JMVAE, versus around 18% for the best Concept-NB approach and 0.28% random chance.The evaluation uses the MNIST-A iid split and compares the best TELBO, JMVAE, and BiVCCA versions.

7 CONCLUSIONS AND FUTURE WORK

The paper demonstrates generative models that imagine compositionally novel concrete and abstract visual concepts, and identifies richer descriptions and multi-object scenes as future directions.

7 CONCLUSIONS AND FUTURE WORK: The models imagine compositionally novel concrete and abstract visual concepts.Future work will explore natural-language descriptions and compositional scene descriptions with variable numbers of objects.

A APPENDIX · A.1 ANALYSIS OF JMVAE OBJECTIVE

This appendix analyzes JMVAE’s description-conditioned KL term by rewriting it over image indices associated with each attribute vector. The analysis shows that JMVAE makes the description inference network cover the image-induced latent embeddings for that concept.

A.1 ANALYSIS OF JMVAE OBJECTIVE: JMVAE’s objective includes a KL term comparing the joint image-description inference network with the description-only inference network.
A.1 ANALYSIS OF JMVAE OBJECTIVE: For each unique training label y_i, X_i denotes its associated image indices and N_i = |X_i| denotes the number of associated images.
A.1 ANALYSIS OF JMVAE OBJECTIVE: The average KL can be reformulated by treating the image index n as a random variable with uniform prior q(n|y_i) = 1/N_i.
A.1 ANALYSIS OF JMVAE OBJECTIVE: Under this notation, the likelihood q(z|n, y_i) is defined as the posterior q_φ(z|x_n, y_i) induced by the corresponding image and description.
A.1 ANALYSIS OF JMVAE OBJECTIVE: The description-conditioned distribution q_φy(z|y_i) represents the average of the posteriors associated with all images sharing attribute vector y_i.
A.1 ANALYSIS OF JMVAE OBJECTIVE: Because image-conditioned posteriors are nearly delta functions, JMVAE effectively requires q_φy(z|y_i) to cover every image embedding associated with that concept.

A.2 DETAILS ON THE MNIST-A DATASET

MNIST-A augments each MNIST image with sampled scale, orientation, and location attributes, then applies corresponding transformations to create binary 64×64 images. Repeating this process ten times per original image yields 700,000 examples split into train, validation, and test sets.

A.2 DETAILS ON THE MNIST-A DATASET: MNIST-A samples discrete scale, orientation, and location labels for each original MNIST image before converting them into continuous transformation parameters.Scale labels are big or small; orientation labels are clockwise, upright, or anti-clockwise; location labels specify four canvas regions.
A.2 DETAILS ON THE MNIST-A DATASET: The dataset generates each image by rotating, scaling, and translating a 28×28 digit onto an empty black 64×64 canvas, followed by binarization.Scaling and resizing use bicubic interpolation, and the paper provides example binary images in Figure 7.
A.2 DETAILS ON THE MNIST-A DATASET: Each original MNIST image produces 10 transformed images, with attribute labels sampled uniformly from their corresponding categorical spaces.This expands the original 70,000-image dataset to 700,000 MNIST-A images.
A.2 DETAILS ON THE MNIST-A DATASET: The resulting MNIST-A dataset is divided into train, validation, and test sets using 85%, 5%, and 10% splits, respectively.The supplied passage describes these splits for the generated 700,000-image dataset.

A.3 β-VAE vs.JOINT VAE

On MNIST-2bit, ignoring semantic attributes with β-VAE fails to organize the latent space around high-level concepts, whereas joint VAE learning captures the four attribute-defined concepts with convex, linearly separable regions.

A.3 β-VAE vs.JOINT VAE: The comparison uses MNIST digit images annotated by parity and magnitude, forming four binary-attribute-defined concept types in a two-dimensional latent space.The attributes are odd versus even and digit value below 5 versus at least 5.
A.3 β-VAE vs.JOINT VAE: β-VAE trained without annotations produces a latent space in which the region for large, even digits is almost nonexistent.The reported β-VAE sweep selects β = 10 as the best-looking latent space, and Figure 8 visualizes one generated digit per latent-space point.
A.3 β-VAE vs.JOINT VAE: Joint VAE captures all four MNIST-2bit concepts with convex, linearly separable latent-space regions, while autoencoding the images well.The joint model is trained by optimizing the joint ELBO over images and attributes, with colors derived from p(y|z).

A.4 DETAILS OF THE NEURAL NETWORK ARCHITECTURES

The models use three inference networks and two decoders built primarily with ELU nonlinearities, with MNIST-A and CelebA architectures adapted to their respective image and attribute spaces. The MNIST-A evaluation additionally uses a convolutional observation classifier for the 3C metrics.

MNIST-A model architecture: The joint model comprises encoders q(z|x,y), q(z|x), and q(z|y), decoders p(x|z) and p(y|z), and ELU nonlinearities across models.The image decoder follows DCGAN, while the label decoder factorizes across attributes and uses two-layer MLPs with 128 hidden units per layer.
MNIST-A model architecture: The joint MNIST-A encoder separately processes images and labels, concatenates their features, and predicts latent Gaussian means and log standard deviations through multilayer perceptrons.The image pathway uses convolutional layers with 32, 64, 128, and 16 feature maps and strides 1, 2, 2, and 2; the image-only encoder reuses this extractor.
MNIST-A Observation Classifier Model: The MNIST-A observation classifier is a convolutional neural network used to evaluate correctness, coverage, and compositionality.Its initial layers use 5×5 convolutions with 32 and 64 channels, interleaved with 2×2 pooling layers of stride 2.
CelebA model architecture: CelebA retains the MNIST-A encoder and DCGAN decoder designs while using RGB inputs and outputs, an 18-dimensional latent space, and a Quantized Normal image likelihood.The label encoder processes 18 embedded categorical labels with one hidden layer before concatenation and two afterward; the joint encoder likewise accepts 18 labels.

A.5 OUTPUTS OF OBSERVATION CLASSIFIER ON GENERATED IMAGES … A.9 MORE RESULTS ON CELEBA

The appendix validates generated images with an observation classifier, examines hyperparameter sensitivity and compositional generalization on MNIST-A, and provides further CelebA dataset and generation analyses. Results emphasize the importance of likelihood scaling, identify labeling caveats, and show that TELBO can produce accurate and diverse images for specified or novel concepts.

A.5 OUTPUTS OF OBSERVATION CLASSIFIER ON GENERATED IMAGES: The observation classifier generally assigns generated MNIST-A images reasonable labels that humans would also accept, making it a useful proxy for human evaluation.Figure 11 displays sampled TELBO images alongside classifier-predicted attributes.
A.6 HYPERPARAMTER CHOICES FOR TELBO, JMVAE, BIVCCA ON MNIST-A: At λy=1, TELBO correctness falls to 60.47 (± 0.34), compared with 82.08 (± 0.56) at λy=50, showing λy is critical for performance.The same passage reports that analogous best settings are λy=10 for BiVCCA and λy=50 for JMVAE.
A.7 COMPOSITIONAL GENRALIZATION ON MNIST-A: QUALITATIVE RESULTS AND DETAILS: For compositional MNIST-A queries, TELBO and JMVAE perform well, whereas BiVCCA is substantially poorer according to classifier-detected incorrect generations.The models reuse the best iid-split parameters and are evaluated on unseen compositional queries.
A.6 HYPERPARAMTER CHOICES FOR TELBO, JMVAE, BIVCCA ON MNIST-A: TELBO correctness for fully specified queries changes from 82.08 (±0.56) at γ=50, while γ has a smaller effect than λy.The appendix sweeps γ over {1, 50, 100} and describes its impact as smaller than the common λy scaling term.
A.6 HYPERPARAMTER CHOICES FOR TELBO, JMVAE, BIVCCA ON MNIST-A: For JMVAE, α=1.0 performs best; reducing α to 0.1 or 0.01 lowers correctness from 85.63 (±0.29) to 77.58 (±0.23) and 74.57 (±0.44), respectively.For BiVCCA, μ=0.7 is best, although performance differences across the tested μ values are small.
A.8 DETAILS ON CELEBA: CelebA uses 18 visually distinctive binary attributes with aligned, cropped, 64 x 64 images, split into approximately 182K training and 20K test examples.The resulting dataset spans 3690 visual concepts, and the test split is iid with respect to attribute vectors.
A.9 MORE RESULTS ON CELEBA: TELBO and JMVAE generate accurate CelebA images for fully specified attribute queries, while TELBO produces more diverse samples than JMVAE.Figure 14 compares generations with all attributes specified, and the accompanying discussion attributes JMVAE’s reduced diversity on unseen concepts to its aggregate-posterior reasoning.
A.8 DETAILS ON CELEBA: The CelebA concept bald=1 and male=0 appears in nine training examples, but all are incorrectly labeled, and TELBO’s generations differ qualitatively from them.This caveat limits the claim that the concept is entirely absent from training while supporting that the model did not simply memorize those examples.