PaperlayerView exact revision on arXiv

Source-linked AI summary

ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

Yoad Tewel, Yoav Shalev, Idan Schwartz, Lior Wolf

arXiv:2111.14447v2cs.CVcs.AIcs.CL

TL;DR

Existing image-text matching models score image-sentence compatibility but do not generate captions from images. ZeroCap combines such a model with a large language model and optimizes the combination only at inference time, yielding less restrictive captions and enabling image arithmetic, image comparison, and textual visual analogies.

  • Problem

    Image-text matching models support zero-shot matching and related tasks but lack a way to generate descriptive text from an image.

  • Method

    ZeroCap guides GPT-2 with CLIP during inference by modifying the language model’s context cache, without additional training or tuning.

  • Results

    The method produces captions that are less restrictive than supervised outputs, semantically match images, incorporate real-world information, and support visual-semantic arithmetic.

  • Takeaways & Limitations

    ZeroCap describes differences between images and combines textual or visual concepts to answer image analogy questions with generated text.

  • Takeaways & Limitations

    The web-trained language and image-text models exhibit cultural bias, showing stronger knowledge of Western media than other cultures.

Abstract

from arXiv · show

Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences. While such models can provide a powerful score for matching and subsequent zero-shot tasks, they are not capable of generating caption given an image. In this work, we repurpose such models to generate a descriptive text given an image at inference time, without any further training or tuning steps. This is done by combining the visual-semantic model with a large language model, benefiting from the knowledge in both web-scale models. The resulting captions are much less restrictive than those obtained by supervised captioning methods. Moreover, as a zero-shot learning method, it is extremely flexible and we demonstrate its ability to perform image arithmetic in which the inputs can be either images or text, and the output is a sentence. This enables novel high-level vision capabilities such as comparing two images or solving visual analogy tests. Our code is available at: https://github.com/YoadTew/zero-shot-image-to-text.

1. Introduction

ZeroCap repurposes CLIP and a language model for zero-shot image captioning without additional training, producing flexible text and enabling visual-semantic arithmetic and image comparison.

  • Motivation and contribution: CLIP enabled many fixed-prompt zero-shot vision tasks, but ZeroCap extends this paradigm to generate textual descriptions from images.The method combines CLIP with GPT-2 rather than using CLIP only to select among predefined sentences.
  • Motivation and contribution: ZeroCap requires no additional training and uses CLIP trained on automatically collected web-scale image-text pairs rather than curated captioning datasets.This differs from supervised captioning methods trained on sources such as MS-COCO or Visual Genome.
  • Caption characteristics: Its captions are less scripted and more diverse than supervised outputs, match images in CLIP’s visual-semantic space, and incorporate knowledge from CLIP and GPT-2.The authors note that this freedom lowers conventional word-to-word metric alignment while improving semantic matching.
  • Visual-semantic capabilities: ZeroCap performs semantic arithmetic with images, generating short sentences from image-based analogies rather than retrieving only a word.The operation parallels NLP concept arithmetic such as king − man + woman ≈ queen.
  • Visual-semantic capabilities: The method can describe the difference between two scenes, providing a novel capability for semantic image comparison.This is presented as a corollary of the broader zero-shot visual-semantic approach.

2. Related work

Prior captioning systems combine increasingly capable vision and language models but still rely on curated captions or supervised stages, motivating CLIP-guided language generation.

  • Captioning foundations: Deep captioning progressed from RNNs and attention to scene graphs, graph networks, and Transformers modeling image elements and language interactions.These developments improved modeling of salient objects, spatial relations, semantic relations, and long-range interactions.
  • Web-scale supervision: Large-scale web image-text data support vision-language pretraining, yet prior captioning systems retained human-annotated datasets such as MS-COCO or Visual Genome in final training.The related work identifies curated data as a persistent component of existing captioning pipelines.
  • Open-vocabulary captioning: Curated captions cannot cover a sufficiently broad range of plausible images and objects, producing dataset biases and motivating external data, taggers, and joint embedding methods.The paper contrasts these approaches with directing an existing language model using an image-text alignment score.
  • Guided language modeling: This work explores the reverse of text-driven image generation: generating text from an image by guiding a large-scale language model with CLIP.The related direction includes CLIP-supported image manipulation and guided language modeling for controlling language-model outputs.

3. Method

ZeroCap guides GPT-2 during inference by modifying its context cache so generated tokens align with an image while remaining close to the language model’s original distribution.

  • Problem setup: Visual captioning is framed as generating each sentence token conditioned on the image and previously generated tokens.The target is the conditional distribution p(x_i|[x_t]_{t<i}, I).
  • Context-cache optimization: The method uses a transformer language model to infer text and adjusts its context cache during inference instead of changing language-model weights.The cache stores key and value outputs from previous tokens and is modified at each generation step.
  • Extensions: The flexible embedding-space formulation supports arithmetic over visual cues and mixes concepts from text and images.These operations extend the system beyond ordinary image-to-text generation.
  • Context-cache optimization: CLIP guidance aligns candidate tokens with the image, while cross-entropy regularization keeps the modified output distribution close to the original language-model distribution.The two objectives respectively promote image relevance and preservation of language attributes.
  • CLIP loss: CLIP potentials are computed for the top 512 token candidates, using cosine distance and a temperature τ_c = 0.01 to shape the target distribution.The remaining token candidates receive zero potential for efficiency.
  • Inference procedure: At each generation step, the context-cache optimization is performed with five gradient-descent steps, and the gradients are normalized before each step.The implementation uses learning rate α = 0.3.
  • Decoding: Beam search reduces myopic token choices caused by byte-level subword tokenization, such as failing to select the prefix needed to form “zebra.”The broader search can recover descriptions whose relevant concept spans multiple subwords.

4. Visual-Semantic Arithmetic

The method uses CLIP embedding arithmetic to expose semantic directions and steer generated text, supporting interactions among images, text, and conceptual relations.

  • Subtraction: Subtracting CLIP representations can express relationships between images by generating text that verbalizes the resulting semantic direction.The paper describes subtraction as a way to transcribe relationships between encoded images.
  • Summation and Mixed Inputs: Summing vectors guides captions toward conceptual directions, extending arithmetic beyond visual inputs through CLIP’s textual encoder.Expressions can contain both image- and text-embeddings.
  • Arithmetic in CLIP Space: CLIP encodes images or text, arithmetic combines their embeddings, and the resulting vector replaces the image encoding to steer sentence generation.For two images, the method computes Eimage(I1) + Eimage(I2) before generating text.
  • Conceptual Directions: The approach generates external-world knowledge by moving along conceptual directions, such as deriving a presidential direction from Obama minus America’s flag.Adding that direction to another country’s flag produces a generated answer such as “Angela Merkel.”

5. Experiments

Experiments show that zero-shot captions trade reference-based metric performance for greater diversity and image-relatedness, while also supporting OCR, external knowledge, and visual arithmetic.

  • Image Captioning: The method lags supervised captioning methods on human-reference metrics because those methods are trained on COCO annotations similar to the evaluation references.The supervised metrics include BLEU, METEOR, CIDEr, SPICE, and CLIPScoreRef.
  • Image Captioning: 8681 versus 2464 vocabulary items and 100% on %Novel show substantially greater caption diversity than previous approaches.None of the generated sentences appear in COCO’s training set.
  • Image Captioning: 87% versus 77% on CLIPScore indicates stronger reference-free image-caption relatedness for the proposed method than for the supervised method.Supervised methods outperform the proposed method on CLIPScoreRef, although the gap is narrower than on other supervised metrics.
  • Image Captioning: Qualitative examples show richer captions, image reasoning, object grounding, OCR, and recognition of real-world knowledge.The method can generate text from images containing written content by changing the prefix prompt.
  • Visual-Semantic Arithmetic Study: Subtraction reveals semantic directions, while addition guides generated captions through visual cues and supports visual analogy puzzles.Examples include deriving relations from image pairs and adding concepts such as a police officer’s hat or a hammer.
  • Visual Relations Benchmark: The visual-relations benchmark contains 320 relations and evaluates single-word answers with BLEU-1, Recall@5, and CLIP-score.The benchmark constructs directions by subtracting visual pairs and adds each direction to another visual element.
  • Visual Relations Benchmark: 30% at R@5 in most relations demonstrates meaningful but limited success on the challenging visual-relations benchmark.The benchmark lacks multiple references, marks variants such as “US” versus “USA” as misses, and includes truncated outputs.
  • Visual Relations Benchmark: ClipCap fails to retrieve the correct response on the visual-relations benchmark despite using the same large-scale models and CLIP embedding arithmetic.CLIP-VL and other supervised methods cannot be tested because the benchmark uses spatial-grid-incompatible representations.

6. Discussion and Limitations

The paper presents generative visual-semantic zero-shot capabilities beyond CLIP’s classification, while noting resource and data biases that constrain the approach.

  • The work introduces generative visual-semantic zero-shot capability, extending CLIP beyond multiclass classification.
  • Image arithmetic can remove one concept and add another, producing outputs such as presidents paired with different national flags.
  • The visual-relations benchmark compares the proposed method with ClipCap using BLEU-1, Recall@5, and CLIP-score.
  • GPT-2 is substantially less capable than GPT-3 and other recent language models, reflecting unequal access to model-training resources.
  • Combining zero-shot and supervised training may improve captioning metrics, but supervised methods can require substantial computational resources.UNITER is cited as using 3645 hours of a V100 GPU.
  • Training on large collected corpora introduces cultural bias; the model is more knowledgeable about Western media and fails to form relations with Xi Jinping.

7. Conclusions

The paper combines a language model with a visual-semantic matching model without optimizing model weights to produce flexible zero-shot captions. This flexibility supports semantic arithmetic, image comparison, concept combination, and textual answers to visual analogies.

  • The method combines a language model with a visual-semantic matching model to support zero-shot captioning and web-scale knowledge.
  • Instead of optimizing model weights, it modifies language-model token key-value pairs across layers and attention heads during inference.
  • The captions are less restrictive than supervised outputs, match images semantically, and exhibit real-world information despite lower word-to-word metrics.
  • Visual-semantic arithmetic describes differences between images and combines concepts from multiple images.
  • Image analogy questions produce a text string for A is to B as C is to X, with A, B, and C optionally textual or visual.

C. Ablation Study

The ablation study examines direct CLIP-based token optimization and the trade-off between language fluency and image similarity. Human evaluations favor the proposed method over the ablations and ClipCap.

  • Directly optimizing the language model with CLIP scores yields captions that are not competitive with the proposed method.
  • Despite a higher CLIP score, the proposed method has better language fluency, showing that higher CLIP similarity does not necessarily produce better wording.
  • 70.5% of annotators selected the proposed caption, compared with 8.9% for A1 and 20.6% for A2.
  • Increasing the regularizer coefficient improves language fluency while decreasing CLIP similarity; λ = 0.2 is identified as a good trade-off.
  • On web images, the proposed method scores 3.79 versus 3.17 for ClipCap on human-likeness and 3.98 versus 3.21 on image grounding.

D. Additional Qualitative Results

The method produces original captions that differ substantially from baseline captions in both vocabulary and structure, evaluated on 200 randomly selected MS-COCO test images.

  • 200 randomly selected MS-COCO test images were captioned by the method and compared with ClipCap, CLIP-VL, and VinVL.The comparison is presented in Figure 15.
  • The generated captions are completely different from baseline captions in both vocabulary and pattern.

E. Limitations

The method inherits caption-quality and bias issues from the noisy web-scale data used to train CLIP and GPT-2.

  • Noisy web-scale training data can introduce source-related or irrelevant entities, such as Flickr or photographer names, into generated captions.Negative-prior regularization on capitalized subwords makes captions more generic but reduces world-knowledge capabilities.
  • The method can produce irrelevant captions when generation fails to remain focused.Length control uses hyperparameters fe = 1.04 and te = 3 to adjust end-token probability, with examples shown in Figure 12.
  • Web-scale data can also introduce biases from biased sources such as news.The paper discusses these biases alongside caption-quality issues in its limitations analysis.

F. Visual Relations Benchmark Study

The visual relations benchmark tests real-world knowledge together with visual-relationship representation, while the paper documents several recurring error sources and qualitative analyses.

  • F. Visual Relations Benchmark Study: The benchmark combines real-world knowledge with the ability to represent visual relationships.
  • F. Visual Relations Benchmark Study: Benchmark errors include unpopular knowledge, synonym choices, closely related concepts, and incorrect inferred relations.Examples include selecting Sydney over Canberra, “Canadian” over “Canada,” Sinai over Egypt, and incorrect university relations.
  • F. Visual Relations Benchmark Study: Using the same pair-subtraction relation across multiple image triplets means an incorrect relation can produce many benchmark errors.
  • F. Visual Relations Benchmark Study: Figure 10 compares direct CLIP token selection with token selection using a combined CLIP and language-model score.
  • F. Visual Relations Benchmark Study: Figure 11 compares caption generation with and without the entity-control mechanism, while Figure 12 compares length-controlled and uncontrolled generation.
  • F. Visual Relations Benchmark Study: Figures 13–15 cover bias cases, visual-relations error analysis, and qualitative caption comparisons with ClipCap, CLIP-VL, and VinVL.
Loading 2111.14447v2…