PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models Can See: Plugging Visual Controls in Text Generation

Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, Nigel Collier

arXiv:2205.02655v2cs.CVcs.CL

TL;DR

MAGIC addresses the open question of guiding language-model generation with images. It combines GPT-2 and CLIP in a training-free decoding framework that adds image relevance to token selection. The authors report improved results on image captioning and visually grounded story generation, alongside faster decoding than prior zero-shot captioning methods.

  • Problem

    It remains unclear how to guide language-model generation with information beyond text, while image-text models remain under-explored for image-grounded text generation.

    Source: §1 · pp. 1–2

  • Method

    MAGIC combines GPT-2 and CLIP in a training-free decoding framework that adds a CLIP-based magic score to guide next-token selection toward an image.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    MAGIC outperforms previous state-of-the-art methods in image captioning and visually grounded story generation, and decodes around 27 times faster than the prior zero-shot image-captioning state of the art.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The experiments demonstrate that MAGIC can perform zero-shot image captioning and visually grounded story generation given an image and text prompt.

    Source: §1 · p. 2

    Applying MAGIC Search to existing multimodal generative models that take both image and text input is outside this paper’s scope.

    Source: §6 · p. 10

Abstract

from arXiv · show

Generative language models (LMs) such as GPT-2/3 can be prompted to generate text with remarkable quality. While they are designed for text-prompted generation, it remains an open question how the generation process could be guided by modalities beyond text such as images. In this work, we propose a training-free framework, called MAGIC (iMAge-Guided text generatIon with CLIP), for plugging in visual controls in the generation process and enabling LMs to perform multimodal tasks (e.g., image captioning) in a zero-shot manner. MAGIC is a simple yet efficient plug-and-play framework, which directly combines an off-the-shelf LM (i.e., GPT-2) and an image-text matching model (i.e., CLIP) for image-grounded text generation. During decoding, MAGIC influences the generation of the LM by introducing a CLIP-induced score, called magic score, which regularizes the generated result to be semantically related to a given image while being coherent to the previously generated context. Notably, the proposed decoding scheme does not involve any gradient update operation, therefore being computationally efficient. On the challenging task of zero-shot image captioning, MAGIC outperforms the state-of-the-art method by notable margins with a nearly 27 times decoding speedup. MAGIC is a flexible framework and is theoretically compatible with any text generation tasks that incorporate image grounding. In the experiments, we showcase that it is also capable of performing visually grounded story generation given both an image and a text prompt.

1 Introduction

The paper asks how image-text models can guide language-model generation, then introduces MAGIC, a training-free decoding framework for image-grounded text. The authors report strong zero-shot captioning results and visually grounded story generation.

1 Introduction

  • The work addresses the open question of how to guide text-prompted language-model generation with information beyond text, such as images.Image-grounded text generation remains under-explored for pre-trained image-text embedding models.Source: §1 · pp. 1–2
  • Earlier approaches used annotated image-text pairs or detector-generated pseudo-pairs, while ZeroCap combined frozen CLIP and GPT-2 but relied on gradient-based context-cache optimization.The authors identify that optimization as slowing inference.Source: §1 · p. 2
  • MAGIC combines GPT-2 and CLIP in a decoding scheme that adds a CLIP-induced magic score to guide next-token selection toward a given image.The framework requires no additional training or parameters and uses control knobs to select outputs.Source: §1 · p. 2
  • MAGIC achieves state-of-the-art performance across evaluation metrics on MS-COCO and Flickr30k, with inference around 27 times faster than the prior zero-shot image-captioning state of the art.The paper also reports strong results on visually grounded story generation.Source: §1 · p. 2

2 Background

The background reviews image-captioning approaches and plug-and-play generation, emphasizing the data requirements of captioning methods and the optimization cost of prior unsupervised approaches.

2.1 Image Captioning

  • Image-captioning methods are grouped into supervised approaches using manually labeled image-text datasets and weakly or unsupervised approaches that use pseudo-pairs or no paired data.The task aims to produce meaningful, syntactically correct descriptions of images and remains unsolved.Source: §2.1 · p. 2
  • Supervised captioning methods commonly train image encoders and sentence generators on ground-truth captions, while later work models visual-textual interactions or uses additional control annotations.Vision-language pre-training methods also rely on paired image-text data for downstream training or fine-tuning.Source: §2.1 · p. 3
  • Weakly supervised captioning uses detector-derived pseudo-captions, but noisy alignment and fixed detector labels limit its coverage of out-of-domain concepts and scenes.The approaches may require constraints to mitigate irrelevant words in pseudo-captions.Source: §2.1 · p. 3
  • ZeroCap combines pre-trained CLIP and GPT-2 for zero-shot captioning but optimizes the context cache at every prediction step, increasing computational overhead as the language model grows.MAGIC instead uses explicit control knobs and reports decoding around 27 times faster than ZeroCap.Source: §2.1 · p. 3

2.2 Plug and Play Generative Models

  • Plug-and-play generation uses pre-trained models without retraining, but many prior methods shift hidden or context representations through iterative gradient optimization.MAGIC changes the decoding strategy, uses task-corpus adaptation of GPT-2, and keeps the adapted GPT-2 and CLIP fixed during decoding.Source: §2.2 · pp. 3–4

3 Methodology

The methodology adapts a language model to the target text domain, then uses MAGIC Search to choose tokens that balance model confidence, image relevance, and avoidance of degeneration.

3.1 Unsupervised Language Modelling

  • The language model is adapted on the end-task text corpus using a maximum-likelihood objective and a contrastive objective intended to calibrate representations and improve perplexity.The contrastive objective uses a predefined margin and cosine similarity between token representations.Source: §3.1 · p. 4

3.2 MAGIC Search

  • MAGIC Search selects each next token from the model’s top-k predictions while incorporating model confidence and a degeneration penalty based on similarity to prior token representations.The candidate token representation is computed from the prefix concatenated with that token.Source: §3.2 · p. 4
  • The magic score uses pre-trained CLIP image-text similarity over candidate continuations to encourage text that is semantically related to the image.The visual-control strength is set by β; when β = 0, MAGIC Search reduces to vanilla contrastive search.Source: §3.2 · pp. 4–5
  • MAGIC Search plugs visual controls into decoding without extra supervised training or gradient updates on additional features, making it more computationally efficient than prior approaches.The paper reports an approximately 27-times speedup in its experiments.Source: §3.2 · p. 5

4 Zero-Shot Image Captioning

MAGIC is evaluated for zero-shot image captioning on MS-COCO and Flickr30k, where it leads most reported metrics and decodes nearly 27 times faster than ZeroCap. It also leads in cross-domain evaluation, while qualitative examples show more fluent, image-grounded captions than the compared methods.

4 Zero-Shot Image Captioning

  • The evaluation uses MS-COCO and Flickr30k, compares image-aware and image-unaware zero-shot baselines, and measures caption quality and inference speed.GPT-2 is fine-tuned on each benchmark’s training text corpus; the comparisons include CLIPRe and ZeroCap.Source: §4 · p. 5

4.1 Results

  • MAGIC achieves the best performance on 11 of 13 metrics across MS-COCO and Flickr30k, beats ZeroCap on 12 of 13, and decodes nearly 27× faster.The reported speedup is attributed to decoding without gradient updates.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Cross-Domain Experiment

  • Although all methods drop in cross-domain evaluation, MAGIC performs best among the compared methods.The experiment applies a language model fine-tuned on source-domain text to a target-domain test set.Source: §4.2 · pp. 6–7, Table 2 · p. 6

4.3 Qualitative Evaluation

  • MAGIC produces fluent captions grounded in image details, while CLIPRe can be irrelevant and ZeroCap can hallucinate objects or produce ungrammatical text.Examples include correctly identifying a street sign and a yellow boat lined up on a beach.Source: §4.3 · p. 7, Figure 1 · p. 7

5 Story Generation

MAGIC applies visual guidance to story generation by retrieving an image related to the title and using it alongside the text prompt. Automatic, human, and qualitative evaluations report stories that are more diverse, coherent, fluent, and image-related than the compared methods.

5 Story Generation

  • The task uses ROCStories titles as prompts and compares MAGIC with greedy, beam, sampling, and contrastive decoding methods.The language model is fine-tuned on ROCStories, and decoding hyperparameters are selected using validation-set MAUVE.Source: §5 · p. 7

5.1 Automatic Evaluation

  • For ROCStories generation, MAGIC uses the title to retrieve an image and generates a story conditioned on both; it performs best on most automatic metrics.The image index contains over 3.3 million ConceptualCaptions images; automatic measures include diversity, coherence, MAUVE, and CLIPScore.Source: §5 · pp. 7–8, §5.1 · p. 8, Table 3 · p. 8

5.2 Human Evaluation

  • Human evaluation finds MAGIC better on fluency and informativeness, with significant gains in coherence and story-image relevance.The coherence and relevance gains are significant by Sign Test (p-value < 0.05); the evaluation uses 200 test instances rated by five graders.Source: §5.2 · pp. 8–9, Table 3 · p. 8

5.3 Qualitative Evaluation

  • Compared with contrastive search, MAGIC’s stories include image-related details and stay on topic in the illustrated examples.Examples describe ice-cream varieties and a beach volleyball game, while contrastive search is described as off-topic in the latter example.Source: §5.3 · p. 9, Figure 2 · p. 9

6 Conclusion and Future Work

MAGIC is a training-free decoding scheme for image-grounded captioning and story generation, with experiments reporting gains in automatic and human evaluations. The conclusion notes that adapting MAGIC to other multimodal generative models or modalities remains future work.

6 Conclusion and Future Work

  • MAGIC is evaluated on image captioning and visually grounded story generation, outperforming prior state-of-the-art methods in automatic and human evaluations.The framework is presented as training-free and preserves decoding speed.Source: §6 · p. 9
  • MAGIC Search could fit existing multimodal generative models that take image and text as input, but this extension is outside the paper's scope.The authors leave this direction to future work.Source: §6 · p. 10
  • The authors describe MAGIC as theoretically extendable beyond images when a similarity metric can measure control-to-text relevance, and propose exploring audio and video.Source: §6 · p. 10

A Related Work

  • Pre-trained language models have advanced natural language understanding and text generation tasks.The related-work discussion traces this progress to GPT-2 and BERT and large-scale free text.Source: A Related Work · p. 17
  • Contrastive learning is used in image and video vision, language-image pretraining, and video-text alignment.Source: A Related Work · p. 17
  • In NLP, contrastive learning has been applied to token- and sentence-level representations and to NER, document summarisation, knowledge probing, and open-ended generation [13].Source: A Related Work · p. 17

B Ablation Study on Hyperparameters of MAGIC Search

The ablations vary one MAGIC Search hyperparameter at a time on MS-COCO. They report how search width, context similarity, and visual-control strength affect performance, with selected settings of k=45, α=0.1, and β=2.0.

Effect of β

Effect of k

Effect of α

B Ablation Study on Hyperparameters of MAGIC Search

C Detailed Results of Zero-Shot Image Captioning

Table 4 reports detailed numerical results for stochastic baselines in zero-shot image captioning, separating in-domain from cross-domain experiments and summarizing three runs.

C Detailed Results of Zero-Shot Image Captioning

D More Visual Examples of Zero-Shot Image Captioning

Figure 4 provides additional visual examples for zero-shot image captioning, comparing MAGIC with two zero-shot baselines and showing reference captions.

D More Visual Examples of Zero-Shot Image Captioning

E Detailed Results of Story Generation

The section reports detailed story-generation results for Top-k, Nucleus, and Typical stochastic baselines, averaging three runs and reporting their standard deviations.

E Detailed Results of Story Generation

  • Results for Top-k, Nucleus, and Typical are reported across three random-seed runs, with each method’s mean and standard deviation.The supplied passage describes the reporting procedure but does not provide numerical values.Source: E Detailed Results of Story Generation · p. 19

F Human Evaluation Guidelines

The human-evaluation guidelines assess coherence, fluency, informativeness, and story-image relevance. The section also presents story-generation examples and numerical results for stochastic methods.

F Human Evaluation Guidelines

F.1 Coherence

  • Coherence is judged by consistency with the story title, from fully supported or inferable content to off-topic or contradictory content.The guidelines define scores from 1 to 5, with intermediate levels reflecting increasing semantic inconsistency.Source: §F.1 · pp. 19–20

F Human Evaluation Guidelines

  • Table 5 reports complete stochastic-method story-generation results, with averages and standard deviations across three runs.Source: Table 5 · p. 20

F.2 Fluency

  • Fluency scores range from human-like, grammatical, easily understood text to nonsensical, unreadable text.Scores 2 and 4 are used when judges hesitate between adjacent odd-numbered scores.Source: §F.2 · p. 20

F.3 Informativeness

  • Informativeness scores reflect novelty, diversity, and interestingness, ranging from informative and enjoyable text to dull, repetitive text without new information.Scores 2 and 4 are used when judges hesitate between adjacent odd-numbered scores.Source: §F.3 · p. 20

F.4 Story-Image Relevance

  • Story-image relevance scores range from diverse concepts visibly present in the image to text completely irrelevant to it.The middle score indicates partial image relevance; scores 2 and 4 resolve hesitation between adjacent odd-numbered scores.Source: §F.4 · p. 21

G More Examples of Story Generation

  • Figure 5 presents additional story-generation examples from contrastive search and MAGIC Search, grounded on an image retrieved using the story title.The passage identifies the retrieved image as shown on the left-hand side of Table 5.Source: G More Examples of Story Generation · p. 21, Figure 5 · p. 21

Related papers

Loading 2205.02655v2…