PaperlayerView exact revision on arXiv

Source-linked AI summary

DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training

Wei Li, Linchao Zhu, Longyin Wen, Yi Yang

arXiv:2303.03032v1cs.CVcs.AIcs.CL

TL;DR

Zero-shot captioning needs to generate image descriptions without depending on human-annotated paired data, while CLIP’s modality gap prevents direct use of visual embeddings for decoding. DeCap trains a lightweight decoder only on text and uses a training-free projection into CLIP’s text space; it outperforms competing zero-shot and unpaired captioning methods on the reported benchmarks.

  • Problem

    Zero-shot captioning seeks image descriptions without human-annotated paired data, but the modality gap prevents directly using CLIP visual embeddings as decoder prefixes.

  • Method

    DeCap trains a lightweight visual-aware decoder on text-only data and projects visual embeddings into CLIP’s text embedding space with a training-free support-memory mechanism.

  • Results

    DeCap outperforms other zero-shot captioning methods on MSCOCO and NoCaps and other unpaired captioning methods on MSCOCO and Flickr30k.

  • Takeaways & Limitations

    Text-only decoder training combined with modality-gap reduction supports zero-shot captioning across image and video captioning scenarios.

Abstract

from arXiv · show

Large-scale pre-trained multi-modal models (e.g., CLIP) demonstrate strong zero-shot transfer capability in many discriminative tasks. Their adaptation to zero-shot image-conditioned text generation tasks has drawn increasing interest. Prior arts approach to zero-shot captioning by either utilizing the existing large language models (e.g., GPT-2) or pre-training the encoder-decoder network in an end-to-end manner. In this work, we propose a simple framework, named DeCap, for zero-shot captioning. We introduce a lightweight visual-aware language decoder. This decoder is both data-efficient and computation-efficient: 1) it only requires the text data for training, easing the burden on the collection of paired data. 2) it does not require end-to-end training. When trained with text-only data, the decoder takes the text embedding extracted from the off-the-shelf CLIP encoder as a prefix embedding. The challenge is that the decoder is trained on the text corpus but at the inference stage, it needs to generate captions based on visual inputs. The modality gap issue is widely observed in multi-modal contrastive models that prevents us from directly taking the visual embedding as the prefix embedding. We propose a training-free mechanism to reduce the modality gap. We project the visual embedding into the CLIP text embedding space, while the projected embedding retains the information of the visual input. Taking the projected embedding as the prefix embedding, the decoder generates high-quality descriptions that match the visual input. The experiments show that DeCap outperforms other zero-shot captioning methods and unpaired captioning methods on the typical image captioning benchmarks, i.e., MSCOCO and NoCaps.

1 INTRODUCTION

DeCap addresses zero-shot captioning without relying on human-annotated image-text pairs. It combines a text-only-trained decoder with a training-free projection that bridges CLIP’s visual and text embedding spaces.

  • 1 INTRODUCTION: The framework targets poor generalization from small, visually limited human-annotated datasets to images in the wild.DeCap is presented as a zero-shot alternative requiring text-only data for training.
  • 1 INTRODUCTION: Prior web-scale captioning systems still rely on human-annotated paired-data fine-tuning, while their captioning pretraining can require about 1M steps on 512 TPU v3 chips.These limitations motivate a more flexible and efficient text-only training approach.
  • 1 INTRODUCTION: DeCap uses a lightweight visual-aware language decoder trained only on text data, reducing reliance on human-annotated paired data.The decoder is trained to exploit CLIP’s multimodal embedding space while avoiding end-to-end paired-data training.
  • 1 INTRODUCTION: A training-free projection mechanism maps visual embeddings into CLIP’s text embedding space using a support memory, reducing the modality gap.The projected embedding is then supplied to the decoder as its prefix embedding.
  • 1 INTRODUCTION: DeCap outperforms other zero-shot captioning methods on MSCOCO and NoCaps, and surpasses unpaired captioning methods on MSCOCO and Flickr30k.It also achieves state-of-the-art zero-shot results on MSR-VTT and ActivityNet-Captions.

2 RELATED WORK

DeCap reframes CLIP-based captioning as text-decoder training over CLIP’s aligned multimodal latent space. This avoids additional pairwise training while addressing limitations of retrieval, direct visual decoding, and prior reconstruction methods.

  • CLIP in Captioning: DeCap trains a text decoder with text-only data to invert CLIP’s text encoder and uses the aligned multimodal latent space for captioning.This allows CLIP to support caption generation without additional pairwise training.
  • Zero-shot Captioning: Unlike CLIP-guided language-model methods, DeCap avoids requiring a CLIP text-encoder forward pass for every generated word.The related methods combine CLIP with a pretrained language model such as GPT-2.
  • Text Reconstruction: Prior unsupervised captioning methods often require complex pseudo-training or adversarial alignment because they lack a well-aligned multimodal latent space.One cited approach instead uses a knowledge graph to correlate visual and textual representations.

3 METHOD

DeCap trains a text-only decoder to generate descriptions from CLIP text embeddings, then projects image embeddings into that space for inference. The method compares retrieval, direct visual decoding, and nearest-neighbor decoding to examine the effects of the decoder and projection strategy.

  • 3.1 TEXT-ONLY DECODER PRE-TRAINING: DeCap trains a decoder with prefix language modeling to reconstruct sentences conditioned on fixed CLIP text embeddings, using only text data.The decoder is trained from scratch, and the text embedding serves as the caption prefix.
  • 3.2.1 PROJECTION-BASED DECODING (PD): The method projects each CLIP image embedding into the text embedding space by softmax-weighting support-memory text embeddings according to scaled cosine similarity.The resulting projected vector is normalized and used as the decoder’s prefix embedding without additional training.
  • 3.2.1 PROJECTION-BASED DECODING (PD): The projection-based method requires no additional training and permits support-memory text to be selected for the target domain.The training corpus and support-memory corpus can differ.
  • 3.2.2 DISCUSSION: Directly using the image embedding as the decoder prefix performs poorly in most scenarios, supporting a modality gap between CLIP image and text embeddings.This strategy is called Visual Decoding (VD).
  • 3.2.2 DISCUSSION: Nearest-neighbor decoding generally outperforms CLIP retrieval across experiments, suggesting that the decoder can generate more descriptive sentences.Nearest-neighbor decoding uses the most similar support-memory text embedding as the prefix.

4 EXPERIMENTS

DeCap is evaluated across zero-shot, unpaired, cross-domain, and video captioning settings using text-only training and CLIP-based decoding. It achieves strong benchmark results, including state-of-the-art zero-shot performance and improvements over competing unpaired and cross-domain methods.

  • 4.1 ZERO-SHOT IMAGE CAPTIONING: DeCap attains a new state-of-the-art on all reported zero-shot captioning metrics on MSCOCO and NoCaps.On MSCOCO, CC3M-text and SS1M improve CIDEr over ZeroCap by 27.5% and 36%, respectively.
  • 4.2.1 IN-DOMAIN CAPTIONING: On MSCOCO and Flickr30K, DeCap outperforms recent unpaired approaches and is competitive with supervised BUTD on Flickr30K.The comparison supports the role of both the decoder and projection mechanism in the reported performance.
  • 4.2.2 CROSS-DOMAIN CAPTIONING: DeCap significantly outperforms other text-only methods in cross-domain captioning, while DeCap-TT raises CIDEr from 44.4% to 63.1% without additional training.DeCap-TT uses target-domain captions only to construct the support memory.
  • 4.3 VIDEO CAPTIONING: DeCap trained on image captions outperforms recent zero-shot video-captioning approaches on standard metrics, while video-caption training further improves performance.The method uses random sampling and temporal mean pooling for video captioning.

5 CONCLUSION

DeCap combines a lightweight visual-aware language decoder with a training-free projection mechanism to reduce CLIP’s modality gap for zero-shot captioning, achieving state-of-the-art results across several benchmarks.

  • DeCap combines a lightweight visual-aware language decoder with a training-free projection mechanism that reduces the CLIP modality gap.The decoder is data-efficient and computation-efficient, while the projection maps visual embeddings into the text embedding space.
  • DeCap significantly outperforms existing zero-shot methods on MSCOCO, MSR-VTT, and ActivityNet-Captions.
  • The framework may extend to other zero-shot text-generation tasks such as visual dialog.

A MORE IMPLEMENTATION DETAILS

The implementation uses a frozen CLIP feature extractor, a lightweight Transformer decoder, and a learned projection from CLIP’s embedding size to the decoder’s hidden-state size.

  • The implementation uses a frozen pre-trained ViT-B/32 CLIP model as the cross-modal feature extractor.
  • The decoder is a lightweight 4-layer Transformer with 4 attention heads and hidden-state size 768.
  • A jointly trained linear layer projects CLIP embeddings from 512 to 768 dimensions.
  • Table 5 summarizes training-data sizes and hyperparameters for different datasets.

B DISCUSSION ABOUT THE RECONSTRUCTION

The reconstruction discussion compares retrieval-based baselines with DeCap’s decoder and evaluates results on the MSCOCO Karpathy-test split using reconstructed corpora as support memory.

  • CLIPRe retrieves the most relevant text embedding and uses its original sentence directly as the caption.
  • Nearest-neighbor Decoding retrieves a text embedding and generates a sentence conditioned on that embedding with the decoder.
  • Table 6 reports MSCOCO Karpathy-test results for CLIPRe-recons and DeCap-recons using reconstructed corpora as support memory.

C PRETRAINING-FINETUNING.

DeCap benefits from text-only pretraining and fine-tuning, while broader text coverage improves out-of-domain performance and support-memory changes enable adaptation without training.

  • 48.7% CIDEr on the out-of-domain case improves over 25.8% when training uses CC3M instead of MSCOCO.
  • 58.2% overall CIDEr is obtained by fine-tuning a text-only pretrained DeCap model on MSCOCO.
  • CC3M improves out-of-domain performance because it covers more diverse classes than MSCOCO.
  • Changing the support memory without fine-tuning achieves comparable performance and enables adaptation to new domains using text data.

D THE INFERENCE SPEED

DeCap substantially accelerates caption generation relative to ZeroCap by avoiding gradient updates and repeated text-encoder forward passes.

  • 113x faster than ZeroCap, DeCap achieves substantially lower inference time for captioning.The comparison uses 100 images with batch size 1 on a single Nvidia RTX2080Ti GPU, without beam search.

E AN EFFICIENT STRATEGY TO REDUCE THE NUMBER OF SUPPORT

The paper examines reducing support-memory size through similarity-based filtering and studies how prompt engineering affects decoders trained on different text corpora.

  • Similarity-based filtering reduces redundant support embeddings while preserving DeCap performance.The filtering strategy selects features according to cosine similarity because sampled sentences often have repetitive semantics.
  • PROMPT ENGINEERING: Prompt engineering benefits the BookCorpus-trained decoder but usually degrades the CC3M-trained decoder.CC3M text is caption-related, so adding redundant prompts can disrupt its original text structure; BookCorpus is less caption-oriented.

G VISUALIZATION OF THE EMBEDDINGS

The visualizations examine semantic clustering, the modality gap between CLIP image and text embeddings, projection effects, and style differences in generated captions.

  • CLIP text embeddings form semantic clusters, while image and text embeddings exhibit a clear modality gap before projection.The visualizations use t-SNE embeddings sampled from MSCOCO support memory and image-text pairs.
  • The projection method effectively reduces the modality gap between CLIP image and text embeddings.
  • DeCap-MSCOCO and DeCap-CC3M generate captions with visibly different styles on the MSCOCO Karpathy-test split.
Loading 2303.03032v1…