Source-linked AI summary
Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space
Liwei Wang, Alexander G. Schwing, Svetlana Lazebnik
TL;DR
Image captioning needs models that capture multiple valid descriptions and uncertainty, but fixed-prior CVAEs can produce insufficiently diverse captions. The paper structures latent space with content-specific Gaussian components using GMM and Additive Gaussian priors, and reports that both models improve diversity and accuracy over LSTM and vanilla CVAE baselines, with AG-CVAE offering the clearest advantage in diversity and controllability.
Problem
Image captioning is open-ended, yet fixed-prior CVAEs can collapse to a single mode and produce little diversity in captions sampled for an image.
Method
The paper proposes GMM-CVAE and AG-CVAE, structuring latent space around content-specific Gaussian components and combining component means for images containing multiple objects.
Results
Both GMM-CVAE and AG-CVAE generate captions more diverse and accurate than standard LSTM baselines; AG-CVAE has a clear edge in diversity and controllability.
Takeaways & Limitations
AG-CVAE provides a richer, more flexible, and interpretable latent space for generating and controlling image captions.
Takeaways & Limitations
The proposed conditioning relies on object labels whose supervisory information must be provided separately from the sentences.
Abstract
from arXiv · showhide
This paper explores image caption generation using conditional variational auto-encoders (CVAEs). Standard CVAEs with a fixed Gaussian prior yield descriptions with too little variability. Instead, we propose two models that explicitly structure the latent space around $K$ components corresponding to different types of image content, and combine components to create priors for images that contain multiple types of content simultaneously (e.g., several kinds of objects). Our first model uses a Gaussian Mixture model (GMM) prior, while the second one defines a novel Additive Gaussian (AG) prior that linearly combines component means. We show that both models produce captions that are more diverse and more accurate than a strong LSTM baseline or a "vanilla" CVAE with a fixed Gaussian prior, with AG-CVAE showing particular promise.
1 Introduction
Image captioning must generate descriptions that are both accurate and diverse, but conventional LSTM and fixed-prior CVAE approaches struggle with this ambiguity. The paper introduces GMM-CVAE and AG-CVAE models that structure latent space around image-content components to improve caption generation and controllability.
- Motivation: Image captioning is an open-ended task because one image can support many descriptions and may be difficult for machines to interpret accurately.The desired posterior should capture both linguistic variability and uncertainty about image content.
- Motivation: LSTM captioning methods produce relatively short, generic captions and are poorly suited to sampling diverse candidates for one image.Their training objective maximizes the likelihood of reference human descriptions.
- Proposed approach: AG-CVAE uses its additive latent structure to provide an interpretable way to control descriptions through image-content information.Changing object labels changes semantic-component weights, shifts the sampling mean, and modifies generated descriptions.
- Motivation: Fixed-prior CVAEs sample different latent vectors, but image-captioning posteriors can collapse to one mode and yield little caption diversity.The vanilla prior is a zero-mean, unit-variance Gaussian independent of image content.
- Proposed approach: The paper proposes GMM and Additive Gaussian priors whose components represent different image-content types and can combine when several objects appear.AG-CVAE encourages the encoder mean to approach a weighted linear combination of component means.
- Results: Both proposed models are reported to outperform LSTM and vanilla CVAE baselines on MSCOCO, with AG-CVAE showing marginally higher accuracy and the strongest diversity results.Figure 1 illustrates more diverse and accurate AG-CVAE captions than the LSTM baseline.
2 Background
VAEs model data through lower-dimensional latent encodings, while CVAEs condition both encoding and decoding on image content for tasks such as captioning. Training maximizes a conditional likelihood lower bound using reconstruction and KL-divergence terms, typically estimated with sampled latent vectors.
- VAE: A VAE models data likelihood by representing observations near a low-dimensional manifold and decoding latent embeddings into samples.The decoder models p(x|z), while the encoder approximates the intractable posterior p(z|x) with q(z|x).
- VAE: The VAE objective uses a tractable lower bound on log-likelihood because the non-negative posterior KL-divergence is difficult to compute directly.The bound can be maximized with respect to both encoder and decoder parameters.
- CVAE: A CVAE models descriptions conditioned on image-content representation c by conditioning both encoder and decoder distributions on c.The target distribution is p(x|c), where x denotes the description.
- CVAE: CVAE training maximizes a lower bound on conditional data likelihood using decoder parameters θ and encoder parameters φ.The encoder is qφ(z|x,c), and the decoder is pθ(x|z,c).
- Optimization: The stochastic objective estimates the reconstruction expectation with N samples drawn from qφ(z|x,c), often using one sample in practice.The reparameterization trick enables backpropagation when the encoder distribution is Gaussian.
3 Gaussian Mixture Prior and Additive Gaussian Prior
The models make the CVAE prior depend on image content, using either mixture components or an additive combination of component means to represent multiple objects. GMM-CVAE samples from one component, whereas AG-CVAE encourages embeddings toward the weighted combination of relevant object means.
- Motivation: The prior shapes the learned latent space, so a content-independent fixed Gaussian is inadequate for image captioning.The KL-divergence term encourages the encoder distribution to remain close to the prior.
- Content-conditioned modes: The framework represents image content with K modes whose nonnegative weights sum to one, using object categories in MSCOCO.The formulation can also use other mode definitions, including unsupervised latent topics.
- GMM-CVAE: GMM-CVAE models the conditional latent distribution as a Gaussian mixture whose component weights are the image-content weights and whose means represent content types.All components use the same standard deviation in practice.
- GMM-CVAE: GMM-CVAE trains by sampling a component according to the image-content distribution and then sampling the latent vector from that Gaussian component.The resulting objective is optimized for encoder and decoder parameters with stochastic gradient descent, while prior parameters are kept fixed.
- GMM-CVAE: At test time, GMM-CVAE conditions each description on one sampled component, so images with multiple objects are not jointly represented in each description.AG-CVAE addresses this limitation by structuring the latent space around object cooccurrence.
- AG-CVAE: AG-CVAE uses an additive Gaussian prior that encourages the encoder mean to approach the weighted linear combination of the means for an image’s objects.Figure 3 contrasts switching among cluster centers in GMM-CVAE with approaching the average of object means in AG-CVAE.
- Architecture: The encoder maps image, content, and caption inputs to latent-distribution parameters, while the decoder samples z and generates the caption word by word.The encoder is used only during training; at test time, the decoder generates until an end symbol.
4 Experiments
Experiments evaluate the models on MSCOCO using LSTM and fixed-prior CVAE baselines, oracle and consensus re-ranking metrics, and diversity and qualitative analyses. GMM-CVAE and especially AG-CVAE improve caption quality, diversity, and controllability, while re-ranking remains a substantial bottleneck.
- Experimental setup: Experiments use the MSCOCO train/validation/test split with 118,287 training, 4,000 validation, and 1,000 testing images.Each image has five reference captions in the dataset release described by the paper.
- Experimental setup: The comparisons include a strong LSTM baseline, a vanilla CVAE with a fixed Gaussian prior, and models evaluated with or without object-vector conditioning.The LSTM generates candidates with beam search, whereas CVAE variants sample latent z vectors.
- Caption quality: AG-CVAE obtains the highest oracle upper bound, while GMM-CVAE and AG-CVAE considerably exceed the LSTM baseline and vanilla CVAE.Oracle evaluation takes the maximum relevant metric over generated candidates; increasing z samples expands CVAE candidate sets more easily than increasing LSTM beam width.
- Caption quality: After consensus re-ranking, GMM-CVAE and AG-CVAE outperform the baselines in almost all metrics, with AG-CVAE slightly higher than GMM-CVAE.The paper also reports a large gap between upper-bound and re-ranked performance, identifying re-ranking as an important future direction.
- Diversity: CVAE has little diversity, GMM-CVAE is much better, and AG-CVAE has the decisive advantage in unique candidate captions.CVAE produces fewer distinct candidates, so its absolute number of novel sentences is much lower than for GMM-CVAE and AG-CVAE.
- Qualitative and controllability analysis: Qualitative results show AG-CVAE generating more varied and often more accurate descriptions, while omitting detected objects when the decoder cannot accommodate combinatorially complex sentences.Changing object vectors also makes AG-CVAE more responsive and controllable than GMM-CVAE, although it struggles to mention more than two or three objects in unusual combinations.
5 Discussion
GMM-CVAE and AG-CVAE outperform standard LSTM baselines in caption diversity and accuracy, while AG-CVAE offers stronger diversity and controllability. The discussion contrasts these CVAE advantages with GANs and identifies broader conditioning information as future work.
- GMM-CVAE and AG-CVAE generate captions that are more diverse and more accurate than standard LSTM baselines.
- AG-CVAE has a clear advantage over GMM-CVAE in diversity and controllability despite similar bottom-line accuracy.The diversity advantage is supported quantitatively and qualitatively.
- Unlike GANs, the additive prior provides interpretable control over the latent-space sampling process.The paper also notes that GAN training is difficult for discrete sentence generation.
- The paper favors its simpler vector-space latent structure over more general graphical models because those alternatives incur substantially greater complexity.
- Future work will explore conditioning signals beyond separately supervised object labels, including clustered nouns, noun phrases, or sentence representations.