Source-linked AI summary
Towards Diverse and Natural Image Descriptions via a Conditional GAN
Bo Dai, Sanja Fidler, Raquel Urtasun, Dahua Lin
TL;DR
Existing captioning methods often produce rigid, low-variability descriptions because training and evaluation favor wording overlap with reference captions. This paper uses a conditional GAN with Policy Gradient and early feedback to jointly learn image-conditioned generation and evaluation, producing more natural, diverse, and relevant captions across two datasets.
Problem
Existing captioning systems and metrics emphasize detailed wording overlap, while naturalness and diversity receive less attention.
Method
A conditional GAN jointly trains an image-conditioned generator and evaluator, using Policy Gradient and early feedback for discrete sequence generation.
Results
On MSCOCO and Flickr30k, the method produced more natural, diverse, and semantically relevant descriptions than a state-of-the-art MLE model, while its evaluator was more consistent with human evaluation.
Takeaways & Limitations
Adversarial training offers an alternative to wording-focused captioning by targeting semantic relevance together with naturalness and diversity.
Abstract
from arXiv · showhide
Despite the substantial progress in recent years, the image captioning techniques are still far from being perfect.Sentences produced by existing methods, e.g. those based on RNNs, are often overly rigid and lacking in variability. This issue is related to a learning principle widely used in practice, that is, to maximize the likelihood of training samples. This principle encourages high resemblance to the "ground-truth" captions while suppressing other reasonable descriptions. Conventional evaluation metrics, e.g. BLEU and METEOR, also favor such restrictive methods. In this paper, we explore an alternative approach, with the aim to improve the naturalness and diversity -- two essential properties of human expression. Specifically, we propose a new framework based on Conditional Generative Adversarial Networks (CGAN), which jointly learns a generator to produce descriptions conditioned on images and an evaluator to assess how well a description fits the visual content. It is noteworthy that training a sequence generator is nontrivial. We overcome the difficulty by Policy Gradient, a strategy stemming from Reinforcement Learning, which allows the generator to receive early feedback along the way. We tested our method on two large datasets, where it performed competitively against real people in our user study and outperformed other methods on various tasks.
1. Introduction
Existing image-captioning systems can describe visual content faithfully yet still produce rigid, low-variability language. The paper proposes conditional GANs, Policy Gradient, and early feedback to improve fidelity, naturalness, and diversity.
- State-of-the-art captions may describe images faithfully while sounding rigid, dry, and lacking vitality.
- Maximum-likelihood training encourages resemblance to training n-grams, limiting variability in generated wording.
- The proposed framework targets semantic fidelity, human-like naturalness, and notably different expressions for the same image.
- Conditional GAN training jointly learns an image-conditioned description generator and an evaluator of description quality.
- Policy Gradient treats word production as sequential actions, enabling feedback to reach the nondifferentiable sequence generator.
- Early feedback uses approximate future rewards from Monte Carlo rollouts to mitigate training difficulties from end-only evaluation.
2. Related Work
Earlier captioning methods largely generate and evaluate descriptions through recurring n-gram patterns. The paper instead jointly trains a conditional GAN generator and evaluator to emphasize naturalness, diversity, and image relevance.
- Generation: Early systems detected visual concepts and then used templates or sentence retrieval to produce image descriptions.
- Generation: Encoder-and-Decoder models commonly extract image features and generate words sequentially under maximum-likelihood training.
- Generation: Maximum-likelihood models tend to repeat frequent n-grams, especially after generating a prefix matching a common training pattern.
- Evaluation: BLEU, ROUGE, METEOR, and CIDEr primarily rely on n-gram matching with ground-truth captions.
- Our Alternative Way: The proposed adversarial approach jointly trains a generator for natural descriptions and an evaluator that distinguishes irrelevant or artificial descriptions.
- Our Alternative Way: Conditional GANs address image-description generation by conditioning text on the given image, unlike unconstrained text GANs.
3. Framework
The framework uses a conditional GAN with a generator for image-conditioned descriptions and an evaluator for naturalness and semantic relevance. Policy Gradient and expected future rewards make discrete sequence generation trainable with early feedback, while hierarchical designs extend the approach to paragraphs.
- Overall Formulation: The framework jointly learns a generator G for image-conditioned descriptions and an evaluator E for assessing their visual relevance.The generator produces sentences or paragraphs, while the evaluator scores how well descriptions fit the image.
- Overall Formulation: Unlike MLE-based training, the conditional GAN objective targets semantic relevance and naturalness rather than word-by-word resemblance alone.The generator aims to produce descriptions indistinguishable from human descriptions, while the evaluator distinguishes generated from human descriptions.
- Training G: Policy Gradient & Early Feedback: Policy Gradient treats each generated word as an action, allowing gradient-based training despite the nondifferentiability of discrete token sampling.The policy conditions on image features, a random vector, and preceding words, then samples the next token or sentence-end indicator.
- Training G: Policy Gradient & Early Feedback: Because complete-sentence rewards cause vanishing gradients and slow convergence, the framework supplies early feedback through expected future rewards estimated by Monte Carlo rollouts.Partial sentences are completed repeatedly, and the evaluator scores are averaged to approximate the expected future reward.
- Training E: Naturalness & Relevance: The evaluator increases scores for human descriptions, suppresses generated descriptions, and penalizes descriptions mismatched with the image.Separate objective terms encourage naturalness and semantic relevance, with coefficients α and β selected on validation data.
- Extensions for Generating Paragraphs: For paragraphs, hierarchical LSTMs first generate sentence topics and then words, while the evaluator reverses this process before scoring paragraph-image compatibility.The paragraph score is computed from the dot product between paragraph and image embeddings and converted to a probability.
4. Experiment
Experiments compare G-GAN with human descriptions and an MLE-based generator across conventional and evaluator-based metrics, user judgments, retrieval, and qualitative diversity. G-GAN produces more natural, diverse, and semantically relevant descriptions, while E-GAN better aligns with human evaluation than conventional metrics.
- Evaluation metrics: G-MLE surpassed human descriptions on nearly all conventional n-gram-based metrics, while E-GAN ranked human descriptions highest.These results indicate that conventional metrics reward wording overlap rather than overall sentence quality.
- User study: G-GAN was preferred over G-MLE by human evaluators in 61% of comparisons.The study involved 30 evaluators and approximately 3,000 responses.
- Qualitative comparison: G-GAN descriptions were more distinctive for similar images, whereas G-MLE often produced nearly identical descriptions.Varying the random vector z also demonstrated G-GAN's ability to generate diverse descriptions while preserving semantic relatedness.
- Evaluation metrics: E-GAN aligned more consistently with human evaluations than CIDEr and SPICE, with Kendall correlations of 0.14, -0.30, and -0.25, respectively.E-GAN also produced a larger human-versus-generator score gap than E-NGAN.
- Evaluation by retrieval: G-GAN outperformed G-MLE by a large margin across all tested retrieval cutoffs when generated descriptions were used to identify their source images.The retrieval experiment used 5,000 MSCOCO validation images and evaluated recall at k = 1, 3, 5, and 10.
- Paragraph generation: On paragraph generation, G-GAN again produced more diverse and natural descriptions than G-MLE.G-MLE tended to follow similar patterns across sentences.
5. Conclusion
The paper presents conditional GAN-based image description generation as an alternative to methods focused mainly on detailed wording. Jointly training a generator and evaluator with policy gradients and early feedback yields descriptions that are more natural, diverse, and semantically relevant than an MLE-based model, while the evaluator better matches human judgments.
- Conclusion: The proposed approach targets semantic relevance, naturalness, and diversity rather than primarily matching detailed wording.These properties are described as overlooked by previous efforts.
- Conclusion: The framework jointly trains a generator and evaluator using conditional GANs, policy gradients, and early feedback.Early feedback addresses technical challenges in end-to-end sequence-generator training.
- Conclusion: On MSCOCO and Flickr30k, the method produced descriptions that were more natural, diverse, and semantically relevant than a state-of-the-art MLE-based model.The conclusion cites user studies, qualitative examples, and retrieval applications as evidence.
- Conclusion: The framework also provides an evaluator that is more consistent with human evaluation.This extends the framework's role beyond description generation.