Source-linked AI summary
Text-Only Training for Image Captioning using Noise-Injected CLIP
David Nukrai, Ron Mokady, Amir Globerson
TL;DR
The paper asks whether image captioning can be learned without captioned images, using only CLIP and additional text. It trains a decoder on frozen CLIP text embeddings, adds noise to bridge the text-image embedding gap, and reports strong zero-shot captioning across standard, cross-domain, and style-guided settings. The approach also supports style transfer from unpaired text examples, while remaining behind fully supervised methods and limited to the capabilities of large English-pretrained models.
Problem
Image captioning usually requires paired captioned images, and adapting models to new styles without such paired data is difficult.
Method
CapDec trains a decoder to reconstruct text from frozen CLIP text embeddings and injects Gaussian noise to reduce the text-image embedding gap.
Results
CapDec works across standard, cross-domain, and style-guided captioning, outperforming methods with the same supervision.
Takeaways & Limitations
Text-only training can adapt CLIP to image captioning and use unpaired examples to generate captions in new styles.
Takeaways & Limitations
CapDec remains outperformed by fully supervised baselines and relies on CLIP and a language model pretrained on large English corpora, making extension to other languages challenging.
Abstract
from arXiv · showhide
We consider the task of image-captioning using only the CLIP model and additional text data at training time, and no additional captioned images. Our approach relies on the fact that CLIP is trained to make visual and textual embeddings similar. Therefore, we only need to learn how to translate CLIP textual embeddings back into text, and we can learn how to do this by learning a decoder for the frozen CLIP text encoder using only text. We argue that this intuition is "almost correct" because of a gap between the embedding spaces, and propose to rectify this via noise injection during training. We demonstrate the effectiveness of our approach by showing SOTA zero-shot image captioning across four benchmarks, including style transfer. Code, data, and models are available on GitHub.
1 Introduction
The paper addresses image captioning without captioned images by adapting CLIP with text-only data, including unpaired examples from target styles. It uses noise injection to address the gap between CLIP’s text and image embeddings and reports strong performance across captioning settings.
- Image captioning typically requires large captioned-image datasets, while adapting pretrained vision-language models to new styles remains unclear.
- CapDec trains with CLIP and text data alone, using unpaired textual examples from a target style to enable style transfer.
- Noise injection addresses the text-image embedding gap by making nearby embeddings decode to the same caption.
- Across standard, cross-domain, and style-guided captioning, CapDec outperforms methods using the same supervision.
2 Related Work
Prior captioning methods commonly rely on paired image-text data or extensive pretraining. CLIP-based methods reduce training requirements, but existing zero-shot techniques can underperform.
- The paper’s CapDec overview depicts a gap between corresponding text and visual embeddings and decodes image embeddings at inference.
- Most prior vision-language captioning approaches require extensive training and large paired datasets that are difficult to collect.
- CLIP has been used to reduce training time, improve captions, and support zero-shot captioning.
3 Method
CapDec learns a caption decoder from text embeddings reconstructed into their source texts, keeping the encoders frozen. Noise-injected training makes the decoder robust to the domain gap between text and image embeddings, after which image embeddings are decoded directly.
- CapDec trains a textual decoder to reconstruct each text from its frozen CLIP textual embedding, then applies it directly to image embeddings at inference.
- The method addresses inference failure by modeling the text-image domain gap rather than learning a paired cross-modal mapping.
- Gaussian noise with standard deviation ϵ is added to text embeddings so nearby embeddings decode to the same caption.
- ϵ is estimated from the mean ℓ∞ embedding-difference norm among five captions for each of 15 MS-COCO images.
- The decoder is trained with autoregressive cross-entropy over all tokens while the CLIP text encoder remains frozen.
4 Results
CapDec achieves strong captioning performance across standard, cross-domain, and style-guided settings using text-only supervision. Its noise level requires tuning, with the selected variance determined from text rather than benchmark results.
- Image Captioning: CapDec outperforms unsupervised MAGIC and ZeroCap on standard captioning, while fully supervised methods score higher using paired image-text data.The comparison uses MS-COCO and Flickr30k under the same evaluation datasets.
- Cross-Domain Captioning: CapDec outperforms MAGIC in cross-domain captioning while using the same supervision.The model is trained on one dataset and evaluated on another.
- Style-Guided Captioning: CapDec outperforms other baselines for style-guided captioning, including with considerably less pretraining supervision than Zhao et al. (2020).Text-only and text-image pretraining variants both improve results when additional training data is available.
- The Effect of Noise Level: Too little or too much noise is suboptimal for MS-COCO performance, while the chosen variance is ϵ2=0.016.The selected value was based only on text-derived embedding statistics, not on the plotted benchmark results.
5 Noise Injection Analysis
The analysis tests whether noise helps through generic regularization or specifically addresses the modality gap. Results support the latter interpretation, although simple mean-offset correction is insufficient overall.
- Noise-Injection Analysis: Noise does not improve text reconstruction or noisy ClipCap, suggesting its benefit is not merely generic augmentation.The tests cover text auto-encoding and a paired image-text model trained with noisy image embeddings.
- Noise-Injection Analysis: Mean-shift modality correction outperforms CapDec below ϵ2 < 0.01 but underperforms overall, indicating that the estimated gap is incomplete.The authors leave more complex or fully supervised modality-gap modeling for future research.
6 Conclusion
The paper proposes adapting generic vision-language models such as CLIP to image captioning with text-only data, while addressing CLIP’s modality gap through noise injection. It presents this paradigm as a basis for future text-only vision-language tasks.
- Text-only training adapts CLIP to image captioning without requiring additional captioned images.
- Noise injection provides a simple technique for overcoming CLIP’s inherent vision-language domain gap.
- The paper identifies visual question answering and visual scene graph generation as future applications of text-only training.
7 Ethics Statement
The paper highlights inherited bias as a problem in image captioning and argues that text-only data may make some mitigation strategies more feasible. In particular, modifying textual descriptions can be easier than collecting additional balanced image-text pairs.
- Image captioning models inherit biases from their training data.
- Balancing text-only datasets may be more feasible than collecting balanced image-text pairs.
- Replacing gendered terms and synonyms in captions is presented as a simple example of text-based bias mitigation.
8 Limitations
CapDec performs better than baselines trained with text alone but remains behind fully supervised methods. Its reliance on models pretrained on large English corpora also limits straightforward extension to other languages.
- CapDec outperforms baselines that use only text during training but is still outperformed by fully supervised baselines.
- The authors believe text-only training could approach supervised quality because CLIP captures rich semantics in its latent space.
- Extending CapDec to other languages is challenging because both CLIP and its language model were pretrained on large English corpora.
A.1 Implementation Details
The implementation uses a CLIP RN-50x4 image encoder with a GPT-2 large language model and a transformer mapping network. Experiments follow established dataset splits and captioning metrics, with scores reproduced or obtained from prior work.
- Optimization uses AdamW with weight decay, a 2e−5 learning rate, and 5000 warm-up steps.
- MS-COCO and Flickr30k evaluations follow the Karpathy split, while FlickrStyle10K uses a random 6/7 training and 1/7 test split.
- Evaluation reports BLEU, METEOR, and ROUGE-L scores, with qualitative scores reproduced or obtained from prior studies after validating matching splits.