Source-linked AI summary
Contrastive Learning for Image Captioning
Bo Dai, Dahua Lin
TL;DR
Image captioning often overlooks the distinctiveness that helps descriptions identify images by their unique aspects. The paper proposes Contrastive Learning, which uses positive and mismatched pairs relative to a reference model to encourage distinctiveness while maintaining caption quality. Across MSCOCO and InstaPIC-1.1M, CL improves target models by significant margins and generalizes across model structures.
Problem
Image-captioning systems often produce safe, training-caption-like descriptions, while the distinctiveness of natural descriptions remains overlooked despite its relation to identifying unique image aspects.
Method
Contrastive Learning uses a reference model with positive and mismatched image-caption pairs, increasing positive-pair probabilities and decreasing negative-pair probabilities relative to the reference.
Results
Across MSCOCO and InstaPIC-1.1M, CL improves target models by significant margins, achieves state-of-the-art results across multiple metrics, and extends to models with different structures.
Takeaways & Limitations
CL provides a generic way to encourage caption distinctiveness while maintaining overall caption quality and building on strong reference models.
Takeaways & Limitations
The formulation can face numerical problems because both target and reference caption probabilities may be very small, around 1e-8.
Abstract
from arXiv · showhide
Image captioning, a popular topic in computer vision, has achieved substantial progress in recent years. However, the distinctiveness of natural descriptions is often overlooked in previous work. It is closely related to the quality of captions, as distinctive captions are more likely to describe images with their unique aspects. In this work, we propose a new learning method, Contrastive Learning (CL), for image captioning. Specifically, via two constraints formulated on top of a reference model, the proposed method can encourage distinctiveness, while maintaining the overall quality of the generated captions. We tested our method on two challenging datasets, where it improves the baseline model by significant margins. We also showed in our studies that the proposed method is generic and can be used for models with various structures.
1 Introduction
The paper identifies nondistinctive image captions as a quality problem and proposes Contrastive Learning (CL) to encourage distinctiveness while preserving caption quality.
- Machine-generated captions often favor safe, training-caption-like wording and can resemble captions for different images.
- Distinctiveness helps descriptions identify an image by emphasizing aspects that distinguish it from similar images.A self-retrieval study links lack of distinctiveness to poorer description quality.
- MLE maximizes probabilities of matched image-caption pairs but does not explicitly account for differences among captions for different images.The resulting captions empirically resemble training captions word by word without being distinctive.
- Contrastive Learning uses a reference model, positive pairs, and mismatched negative pairs to preserve positive-caption probabilities while lowering negative-pair probabilities.The positive constraint protects overall performance, while the negative constraint encourages distinctiveness.
- CL is presented as generic across model formulations and can build on state-of-the-art reference models.
2 Related Work
Related work spans detection-based, encoder-decoder, attention, MLE, reinforcement, introspective, and adversarial approaches; CL is framed as broadly applicable and distinct in its reference-based design.
- Modern image captioning commonly uses encoder-decoder models, with CNN image features and LSTM-based word generation.
- Attention extends encoder-decoder models by dynamically combining features from image regions or other conditions during generation.
- Because captioning models provide p(c|I), they can serve as either target or reference models in CL regardless of structure.
- MLE maximizes conditional likelihood of training samples, but high resemblance in generated captions has been observed.
- RL optimizes rewards from sampled-caption evaluation scores, while prior MLE and RL approaches did not consider distinctiveness.
- Compared with IL and GAN, CL uses an independent fixed reference and dynamically sampled negative captions, providing more stable and diverse guidance.The paper also describes CL as more generic than the cited GAN approach.
3 Background
The background introduces Noise Contrastive Estimation (NCE), which learns a target distribution through classification of observed samples against reference-distribution samples.
- NCE estimates a target distribution by using observed samples X together with samples Y drawn from a reference distribution.
- NCE assigns binary labels to the union U of target and reference samples, with X labeled 1 and Y labeled 0.
- The class-label posteriors are expressed from the target density pm(u; θ), reference density pn(u), and sample-count ratio ν.
- The posterior can be written as h(u; θ) = rν(G(u; θ)), where G is the logarithm of the density ratio and rν is a logistic transformation.
- NCE maximizes joint conditional log-probabilities of the labels and thereby estimates G(·; θ), from which the target distribution can be derived.
4 Contrastive Learning for Image Captioning
Contrastive Learning (CL) enhances caption distinctiveness by learning a target model relative to a reference model, while preserving performance on true image-caption pairs. Self-retrieval evidence links distinctiveness with captioning quality, and the method is designed to generalize across model structures.
- Motivation: Distinctiveness is difficult to specify with rules, so CL improves it relative to a strong baseline while maintaining overall caption quality.The method uses a reference model rather than explicitly enumerating all desirable distinctive properties.
- Empirical Study: Self Retrieval: Self-retrieval evaluates distinctiveness by ranking images for each generated caption and computing the fraction whose source image appears within the top-k positions.The study samples 5,000 MSCOCO test images and uses conditional caption probabilities to rank candidate images.
- Empirical Study: Self Retrieval: Self-retrieval recalls positively correlate with Rouge and Cider performance, supporting distinctiveness as a significant aspect of caption quality.The comparison includes MLE-trained Neuraltalk2, MLE-trained AdaptiveAttention, and CL-trained AdaptiveAttention.
- Contrastive Learning: CL trains on matched and mismatched image-caption pairs, increasing target probabilities for positive pairs and decreasing them for negative pairs relative to a reference model.The positive constraint preserves performance relative to the reference, while the negative constraint encourages distinctiveness.
- Contrastive Learning: The initial difference-based formulation treats easy, hard, and mistaken samples equally and can encounter numerical problems because conditional probabilities are very small.The paper replaces the difference with a log-ratio and applies a logistic function to saturate easy-sample influence.
- Discussion: CL is generic because it requires conditional caption probabilities, allowing target and reference models based on LSTMs, MRFs, and memory-networks.The reference model can also be periodically replaced by the latest target model.
5 Experiment
Experiments on MSCOCO and InstaPIC-1.1M evaluate Contrastive Learning with two captioning architectures and multiple reference-model settings. CL improves captioning results, supports distinctive captions, and benefits from combining positive and negative constraints.
- 5.1 Settings: CL is evaluated on MSCOCO and the more challenging InstaPIC-1.1M using Neuraltalk2 and AdaptiveAttention.Both architectures are encoder-and-decoder models; AdaptiveAttention includes an adaptive attention component, whereas Neuraltalk2 does not.
- 5.2 Overall Results: CL improves AdaptiveAttention across all metrics on MSCOCO and achieves a 14% Cider improvement on InstaPIC-1.1M.On MSCOCO, Cider rises from 1.003 to 1.029, while the InstaPIC-1.1M result is reported as state-of-the-art.
- 5.2 Overall Results: A single CL-trained AdaptiveAttention model achieves the best single-model Cider result on MSCOCO C40, reaching 1.029.The MLE version requires five models for a 4.5% C40 boost, whereas CL gives a 2.5% improvement with one model.
- 5.3 Compare Learning Methods: Full CL outperforms CL(P), CL(N), IL, and GAN because combining positive and negative constraints gives the largest improvements on all metrics.The negative-only variant downgrades the base model, while the positive-only variant provides less improvement than full CL.
- 5.3 Compare Model Choices: CL improves both AdaptiveAttention and Neuraltalk2, while using AdaptiveAttention as Neuraltalk2’s reference raises Cider from 0.905 to 0.956.With their own references, AdaptiveAttention rises from 1.042 to 1.142 and Neuraltalk2 from 0.882 to 0.905.
- 5.3 Reference Replacement: When the reference is sufficiently strong, replacing it repeatedly is unnecessary because most overall-performance improvement is achieved in the first run.The second run produces only incremental relative improvements compared with the first run.
6 Conclusion
Contrastive Learning uses a state-of-the-art reference model to preserve target-model optimality while encouraging distinctive captions. It improves results across two challenging datasets and generalizes across model structures.
- Contrastive Learning uses a state-of-the-art model as a reference to maintain target-model optimality while encouraging captions to learn from distinctiveness.
- On MSCOCO and InstaPIC-1.1M, the proposed method improves the target model by significant margins and achieves state-of-the-art results across multiple metrics.
- Comparative studies show that the proposed method extends to models with different structures, demonstrating generalization ability.