Source-linked AI summary
Discriminability objective for training descriptive captions
Ruotian Luo, Brian Price, Scott Cohen, Gregory Shakhnarovich
TL;DR
Image captions often lack discriminability, making it difficult to distinguish images from their descriptions. The paper adds a retrieval-derived discriminability loss to caption-generator training and reports more discriminative captions alongside improvements on standard caption metrics. The approach is modular, but combining objectives can create optimization and fluency difficulties.
Problem
Generated captions often lack discriminability, while conventional objectives can produce overly general descriptions that fail to distinguish visually different images.
Method
The paper incorporates a discriminability loss derived from image-caption retrieval into caption-generator training, including image-to-caption and caption-to-image matching.
Results
The approach produces more discriminative captions and improves caption quality across standard metrics, including BLEU, METEOR, ROUGE, CIDEr, and SPICE.
Takeaways & Limitations
A retrieval-based discriminability objective can be plugged into existing captioning models and provide richer training signals beyond surface-form matching.
Takeaways & Limitations
Optimizing discriminability alone does not yield human-friendly captions, and combined objectives can become stuck or abruptly reduce fluency as its weight increases.
Abstract
from arXiv · showhide
One property that remains lacking in image captions generated by contemporary methods is discriminability: being able to tell two images apart given the caption for one of them. We propose a way to improve this aspect of caption generation. By incorporating into the captioning training objective a loss component directly related to ability (by a machine) to disambiguate image/caption matches, we obtain systems that produce much more discriminative caption, according to human evaluation. Remarkably, our approach leads to improvement in other aspects of generated captions, reflected by a battery of standard scores such as BLEU, SPICE etc. Our approach is modular and can be applied to a variety of model/loss combinations commonly proposed for image captioning.
1. Introduction
Image captioning seeks fluent, informative descriptions, but conventional objectives often produce overly general captions that fail to distinguish similar images. The paper adds an explicit discriminability objective to caption training and reports improvements in discriminative and standard caption-quality measures.
- Caption quality requires both fluent language and informative descriptions of visual scenes.
- Standard evaluation compares generated captions with human references using metrics including BLEU, CIDEr, and SPICE.
- Caption discriminability measures whether a recipient can identify the matching image from two or more candidates.
- Traditional MLE and CIDEr objectives can encourage safe, overly general captions that repeat across visually different images.
- The proposed discriminability loss uses image-caption retrieval ability to encourage correct matches over incorrect image or caption pairings.
- The loss is modular, applies to gradient-based caption-generator training, and improves recently proposed models across evaluated metrics, including discriminative captioning.
2. Related work
Prior work explored captioning architectures, auxiliary objectives, referring expressions, and visual dialogue, but remained tied to matching human caption surface forms. This paper introduces an explicitly discriminative objective that combines image and caption retrieval with caption generation and can plug into existing models.
- Modern captioning systems commonly encode images with CNNs and generate arbitrary-length word sequences with RNNs, including attention-based variants.
- Referring-expression methods generate discriminative descriptions for image regions, whereas this work targets natural scenes in their entirety.
- Existing methods remained tied to matching human-caption surface forms rather than using an explicitly discriminative training objective.
- The proposed method is the first described here to incorporate both image retrieval and caption retrieval into caption-generation training.
3. Models
The models combine a retrieval network with a caption generator so retrieval-based discriminability signals can guide caption training across modular architectures.
- Collaborative framework: The approach combines a retrieval model that scores image-caption pairs with a caption generator that maps images to captions.Retrieval scores guide generator training, and the framework can use different retrieval models or caption generators.
- Retrieval model: The retrieval model embeds images and captions into a shared semantic space where their compatibility is scored.Image and caption features are projected into a joint space before similarity is computed.
- Retrieval model: Contrastive training learns caption embeddings and projection maps while keeping the pretrained CNN image encoder fixed.The loss uses correct image-caption pairs and mismatches from a batch, encouraging correct pairs to exceed incorrect pairs by margin α.
- Discriminability loss: The discriminability loss uses a pretrained retrieval model as a proxy for human judgments because direct human evaluation is costly and slow.Caption-generator training applies the retrieval contrastive loss to captions sampled from the generator’s posterior distribution.
- Caption generation models: Two caption generators are used: an FC sequence model initialized with visual features and an attention model using spatial image features.Both generate word sequences with LSTM-based mechanisms; the attention model conditions each output on a weighted average of spatial features.
- Caption generation models: Both generators represent caption probabilities as a product of conditional next-word probabilities.The models produce a posterior distribution over caption sequences.
4. Learning to reward discriminability
The paper trains caption generators to reward discriminability while retaining conventional captioning objectives. Because direct optimization is difficult and discriminability alone can reduce fluency, it uses REINFORCE and combines discriminability with MLE or CIDEr.
- Direct gradient descent is unavailable because the discriminability loss samples captions and is therefore non-differentiable.
- The authors use REINFORCE to optimize the expected reward of the stochastic captioning policy.A baseline is used for variance reduction without increasing bias.
- Using only negative discriminability loss can produce non-human-friendly captions because it does not directly penalize poor fluency.The authors therefore combine discriminability with traditional objectives.
- The standard MLE reward is the caption log-likelihood, optimized directly by gradient ascent.CNN weights remain fixed after vision-task pretraining, while word embeddings and LSTM weights are updated.
- Training the combined objective was difficult: small λ values appeared stuck in local minima, while larger values abruptly reduced fluency.The authors report that discriminability cooperated better with CIDEr than with log-likelihood and observed better validation performance across many metrics.
- The combined reward weights CIDEr against discriminability loss, with λ controlling their tradeoff.This weighted-sum formulation differs from prior CIDEr-reward training by adding the discriminability term.
5. Experiments and results
Experiments evaluate discriminability loss across captioning models, objectives, automatic metrics, and human image-discrimination judgments. The loss improves discriminability and can also improve descriptive quality, with λ = 1 providing the strongest overall tradeoff while larger weights increase discriminability at fluency or metric cost.
- Experimental setup: Experiments combine FC or ATTN captioning models with MLE or CIDEr objectives, with or without discriminability loss, on COCO.The split contains 113,287 training images, 5,000 validation images, and 5,000 test images, each with five human captions.
- Evaluation: Human evaluation presents a generated caption with its target image and a distractor, asking workers which image better matches the caption.The machine evaluation reports retrieval accuracy, while human evaluation follows the same target-versus-distractor discrimination setup.
- Discriminability results: Discriminability loss raises retrieval-based discriminability, and similar Acc and Acc-new values indicate that captioners do not overfit their training-time retrieval model.Acc measures correct target selection against a distractor; Acc-new uses a separately trained retrieval model.
- Model and loss selection: λ = 1 with ATTN+CIDER yields the best discriminability/descriptiveness tradeoff, whereas higher λ improves human and machine discriminability but reduces other metrics and perceived fluency.A relatively broad λ range of 0.5–1.2 produces similar automatic-metric improvements.
- Test-set comparisons: Human captions reach 82% discrimination accuracy, exceeding every automatic caption model, while ATTN+CIDER+DISC(10) achieves the most discriminative automatic result without major degradation under other metrics.The ATTN+CIDER+DISC(1) model remains the best tradeoff among the evaluated models.
- SPICE analysis: Adding discriminability loss improves SPICE Color, Attribute, and Cardinality scores, while small λ can also produce the best Object score and highest overall SPICE.The Cardinality improvement may reflect better distinctions between “one” or “two” and “group of” or “many.”
6. Conclusions
The discriminability loss improves caption quality across multiple properties and metrics, producing captions that better identify images while also receiving higher standard scores. The approach can yield more specific descriptions, though stronger loss weighting can make captions overly repetitive.
- The discriminability loss improves caption quality across a variety of properties and metrics.
- Higher discriminability-loss weights produce more discriminative captions for both humans and machines.
- With λ = 1, captions add specific scene details such as bicycles near a harbor, a sunset, or a store with umbrellas.
- Standard metrics including BLEU, METEOR, ROUGE, CIDEr, and SPICE also score the resulting captions higher.
- Future work targets more sophisticated visual semantic embeddings and further improvements in discriminability.
A. Result on standard split and MSCOCO test server
The evaluation covers automatic scores on the standard split and the MSCOCO test server, comparing models with CIDEr optimization and discriminability loss. On the test server, the proposed model slightly exceeds the ATTN+CIDER baseline on all reported metrics.
- The evaluation reports automatic scores on the standard split and on the MSCOCO test server.
- The standard-split analysis includes automatic metrics and SPICE subclass scores, with subclass values scaled by 100.
B. Improved diversity with discriminability objective
The discriminability objective increases caption specificity and diversity. Larger loss weights produce longer and more diverse captions, while Figure 4 contrasts generic baseline captions with more specific outputs from the proposed model.
- Larger λ values produce longer and more diverse captions.
- Pure CIDEr optimization harms the diversity of output captions.
- ATTN+CIDER can assign the same caption to image pairs, whereas ATTN+CIDER+DISC(1) generates more specific captions.
- Table 8 reports distinct caption number and average sentence length on the validation set for different methods.
C. Comparison to [41, 43, 44]
The paper contrasts its discriminability objective with prior losses by emphasizing different contrastive pairings and a different target. Unlike methods focused on diversity or surface matching, it evaluates and improves discriminability while retaining standard-metric gains.
- Loss comparisons: Prior losses compare different matched and mismatched pairings, including human captions and generated captions.
- Loss comparisons: The proposed loss contrasts generated captions with mismatched generated image-caption pairs in both image-retrieval and caption-retrieval directions.
- Target comparison: Methods [43] [44] target diverse natural captions, whereas this work targets discriminability accuracy with a single caption.
- Target comparison: The paper reports improvements in both standard metrics and discriminability, while [43] [44] sacrifice standard metrics to improve diversity.
- SPICE comparison: The proposed objective improves color, attribute, and count SPICE aspects, differing from the GAN discriminator’s favored caption properties.
- Method comparison: Unlike [41], the method explicitly incorporates a discriminative task and allows CIDEr optimization, which appears to improve over MLE.
D. More results
Figures 5–8 present examples in which the proposed objective improves caption accuracy and discriminability-related details, while Figure 9 shows failure cases. Captions compare ATTN+MLE and ATTN+CIDER baselines with the proposed method under different λ values, using distractor images from validation/test sets.
- D. More results: The examples target improvements in color, attribute, and cardinality accuracy, corresponding to SPICE sub-score gains discussed in the paper.Figures 5, 6, and 7 illustrate these three aspects respectively.
- D. More results: Some DISC captions add potentially discriminative details but also contain grammatical or factual problems, such as repeated or incorrect wording.The examples include “a blue and blue airplane” and repeated “bottles” phrasing, illustrating the trade-off shown in the examples.
- D. More results: The proposed captions are compared against ATTN+MLE and ATTN+CIDER baselines using ATTN+CIDER+DISC(x) with different λ values.The comparisons use target images and corresponding distractors selected from validation/test sets; the pairs were included in AMT experiments.
- D. More results: Figure 8 provides additional examples, and Figure 9 presents failure cases of the proposed methods.Green highlights indicate caption elements judged to aid discriminability, while red highlights indicate incorrect or jarringly non-fluent portions.
- D. More results: For motorcycle and skiing scenes, DISC captions more explicitly identify counts or scene details than the baseline captions.Examples include “two people” on a motorcycle and “two people” standing on skis in the snow.