PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding Language Models to Images for Multimodal Inputs and Outputs

Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried

arXiv:2301.13823v4cs.CLcs.AIcs.CVcs.LG

TL;DR

Text-only language models lack visual grounding for tasks involving visual reasoning and image production. The paper introduces FROMAGe, which freezes pretrained language and visual models while learning linear mappings and retrieval mechanisms for interleaved multimodal inputs and outputs. It reports strong zero-shot performance across grounded tasks and interactive multimodal dialogue, while identifying retrieval-candidate coverage as a practical limitation.

  • Problem

    Text-only language models are limited on visual reasoning and grounding tasks and cannot produce images.

  • Method

    FROMAGe grounds frozen language and visual models by learning linear mappings and a retrieval token with captioning and retrieval objectives.

  • Results

    The model shows strong zero-shot performance on contextual image retrieval and visual dialogue, with improved discourse-context sensitivity for image retrieval.

  • Takeaways & Limitations

    The approach supports coherent interleaved image-text outputs and multimodal interaction using pretrained language-model capabilities.

  • Takeaways & Limitations

    Retrieved-image relevance is sometimes limited by the candidate retrieval set, especially for prompts unlikely to occur in natural images.

Abstract

from arXiv · show

We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images. Our method leverages the abilities of language models learnt from large scale text-only pretraining, such as in-context learning and free-form text generation. We keep the language model frozen, and finetune input and output linear layers to enable cross-modality interactions. This allows our model to process arbitrarily interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images. We achieve strong zero-shot performance on grounded tasks such as contextual image retrieval and multimodal dialogue, and showcase compelling interactive abilities. Our approach works with any off-the-shelf language model and paves the way towards an effective, general solution for leveraging pretrained language models in visually grounded settings.

1. Introduction

FROMAGe grounds a frozen language model for interleaved multimodal inputs and outputs, using image captioning and retrieval to support visual reasoning, dialogue, and contextual image retrieval.

  • Text-only language models lack rich visual grounding, limiting visual reasoning and preventing image production.
  • FROMAGe combines a frozen language model and visual encoder with learned mappings for image captioning and image-text retrieval.Captioning maps visual embeddings into the language model input space, while retrieval aligns [RET] representations with paired-image embeddings.
  • The model learns multimodal capabilities from image-caption pairs while keeping most parameters frozen.The approach is presented as an efficient way to leverage pretrained language-model capabilities.
  • FROMAGe performs text-to-image retrieval with greater sensitivity to long and complex free-form text than existing models.
  • The paper demonstrates contextual image retrieval, zero-shot visual dialogue, and improved discourse-context sensitivity for image retrieval.
  • The method supports long, coherent multimodal sequences, with code and pretrained models released publicly.

2. Related Work

The paper builds on large language models, vision-language systems, and parameter-efficient adaptation methods, extending prior work toward multimodal inputs and interleaved image-text outputs.

  • Large language models derive strong abilities from scaling, in-context learning, and long-form generation, but are typically Transformer-based text models.
  • Vision-language research includes models for text-to-image generation and visual language modeling, including approaches based on image tokens and scaled pretraining.
  • Parameter-efficient adaptation methods modify small parameter subsets while keeping pretrained language-model parameters frozen.
  • Prior work motivates FROMAGe’s use of pretrained representations and efficient adaptation while leaving room for interleaved multimodal outputs.

3. Method

FROMAGe integrates frozen language and visual models through trainable linear mappings, using captioning and contrastive retrieval objectives to process and produce interleaved image-text sequences.

  • Architecture: FROMAGe keeps the language model and visual model frozen, learning linear translation layers between image and text embedding spaces.This design aims to leverage pretrained language-model capabilities while adapting the system to multimodal inputs and outputs.
  • Architecture: The language model processes tokenized text autoregressively while the visual backbone produces frozen image embeddings.
  • Architecture: An image-to-text mapping converts visual embeddings into a sequence of vectors matching the language model’s text-embedding dimensionality.
  • Architecture: A learned [RET] token enables the autoregressive model to generate retrieval requests within interleaved text, while mappings project text and image representations into a shared retrieval space.
  • Training: Training jointly optimizes image captioning and image-text retrieval using a weighted combination of captioning and retrieval losses.Captioning conditions text generation on a visual prefix; retrieval uses paired examples and in-batch negatives.
  • Training: Only the linear mappings and [RET] embedding receive gradient updates, while the pretrained language and visual models remain frozen.The implementation reports that 97.0% of model parameters are frozen.

4. Experiments

Experiments evaluate FROMAGe on contextual image retrieval, visual dialogue, and multimodal generation, emphasizing its ability to use interleaved image-text context and in-context learning. FROMAGe improves retrieval with longer descriptions and multimodal context, achieves competitive zero-shot visual dialogue, and generates interleaved image-text outputs.

  • Evaluation scope: The experiments cover contextual image retrieval, image-and-text generation, and zero-shot visual dialogue across multimodal input configurations.The retrieval evaluation uses Visual Storytelling sequences, while visual dialogue tests both text answers and image retrieval from conversations.
  • Multimodal generation: FROMAGe supports multimodal dialogue, in-context few-shot task learning, and semantically meaningful interleaved image-text outputs.The examples emphasize interaction settings unavailable or not meaningfully supported by prior work cited in the passage.
  • Contextual image retrieval: FROMAGe evaluates retrieval of the final Visual Storytelling image from its description, preceding descriptions, or preceding descriptions plus images.These settings isolate standard image-text retrieval, temporally dependent language, and interleaved multimodal context.
  • Contextual image retrieval: With five captions, CLIP R@1 decreases by 50.4% from the single-caption setting, whereas FROMAGe improves from 11.3 to 11.9 R@1.The comparison indicates greater robustness to longer, temporally dependent descriptions for FROMAGe than for CLIP.
  • Contextual image retrieval: With image-and-text context, FROMAGe R@1 increases 37.9% from 13.2 to 18.2 and improves 107% over CLIP with five captions, from 8.8 to 18.2.Similar improvements occur on R@5 and R@10.
  • Visual dialogue: In zero-shot visual dialogue, FROMAGe outperforms ESPER, CLIP, and ViLBERT on R@1, improving 20.5% relative to ESPER.It also reports MRR 22.0 and NDCG 16.5, while remaining below ESPER on those metrics.

5. Analysis

FROMAGe’s ablations show that preserving the frozen LLM, adding a dedicated retrieval token, and supplying richer multimodal context are central to performance. More context also improves retrieval and generated-story quality, with human evaluations favoring full image-text context.

  • 5.1. Ablation Experiments: Freezing the LLM preserves in-context learning and few-shot generalization; finetuning reduces VIST R@1 from 12.8 to 6.2 and VisDial text-retrieval R@1 from 14.6 to 1.0.The degradation occurs on both VIST and VisDial retrieval tasks.
  • 5.1. Ablation Experiments: Adding the dedicated [RET] token improves VIST R@1 by 38.1% relative to training without it.The token represents image embeddings for text-to-image retrieval, with similar improvements reported across other tasks.
  • 5.2. The Effect of Context: Adding one image and text example raises VIST R@1 by 30.1% relative to one caption, from 11.3 to 14.7.Increasing captions alone from one to five raises R@1 by 5.3%, from 11.3 to 11.9.
  • 5.2. The Effect of Context: With five captions and four images, VIST R@1 reaches 18.2, a 61.1% relative improvement over the single-caption case.VisDial retrieval also improves with additional dialogue rounds, and FROMAGe exceeds CLIP in all tested settings.
  • 5.3. In-context Learning and Text Generation: Providing multimodal in-context examples produces more story-like VIST outputs than a single image, which generally produces caption-like descriptions.The model uses the full context of five images and four stories to synthesize text for the fifth image.
  • 5.3. In-context Learning and Text Generation: Human evaluations rate full multimodal context as more coherent than only the last image or text description, while comparing relevance to the image.The evaluation sampled 100 examples with five independent ratings each.

6. Future Work

Future work targets novel image generation and more reliable retrieval-token use. The current retrieval setup can fail when relevant images are absent from the candidate set or when prompts are unusual.

  • 6. Future Work: FROMAGe’s relevant-image output is sometimes limited by its candidate retrieval set, especially for fantastical prompts unlikely to occur in natural images.The authors identify novel image generation as a direction for overcoming this boundary.
  • 6. Future Work: During inference, FROMAGe does not always generate [RET] and is biased toward regular text tokens after text-only pretraining.The authors describe improving novel image generation and retrieval-token behavior as open directions.

7. Conclusion

The paper presents FROMAGe as an efficient way to visually ground frozen pretrained language models through limited finetuning. It reports coherent interleaved image-text outputs and strong zero-shot performance across image-text tasks.

  • 7. Conclusion: FROMAGe visually grounds pretrained frozen language models by finetuning several linear layers.The approach is designed to consume and produce arbitrarily interleaved images and text.
  • 7. Conclusion: The model produces coherent interleaved image-text outputs and shows strong zero-shot performance across tasks involving image-text inputs and outputs.The paper also qualitatively demonstrates multimodal dialogue abilities.
  • 7. Conclusion: Scaling to larger language models and datasets, and generating novel images from scratch, are identified as future directions.

A. Qualitative Examples

Qualitative examples show FROMAGe’s sensitivity to prompt changes, use of pretrained world knowledge, and multimodal dialogue capabilities. Compared with CM3 on selected VIST examples, FROMAGe produces more relevant retrieved images.

  • Sensitivity to prompts: FROMAGe retrieves different images for Winograd-style sentences that differ by a single word, demonstrating sensitivity to small prompt changes.
  • World knowledge: A frozen LLM pretrained on web-scale text supplies world knowledge that FROMAGe can reference despite finetuning only linear layers on image-caption data.The examples include visual implications of 0 degrees Celsius and other world knowledge.
  • Multimodal dialogue: In multimodal dialogue, FROMAGe reasons about user images, returns semantically appropriate images, and can provide coherent text-only responses.The examples also include factual responses about the water cycle and temperature.
  • Comparison with CM3: FROMAGe uses substantially fewer computational resources than CM3: one GPU for one day versus 384 GPUs for 24 days.The comparison evaluates qualitative image relevance on image-and-text VIST story inputs.
  • Comparison with CM3: On selected VIST examples, CM3 often produces uninterpretable or irrelevant outputs, whereas FROMAGe retrieves images relevant to the story input.The comparison focuses on semantic relevance because FROMAGe retrieves images while CM3 generates them.

B.1. Details on Freezing Ablation

The ablations examine freezing, multitask training, example concatenation, and model scaling. Results favor a frozen language model, joint objectives, image-text concatenation for interleaved inputs, and larger backbones.

  • Loss-function ablation: Joint captioning and retrieval training leaves individual-task performance largely unchanged, with captioning scores slightly improving in FROMAGe.The ablation is evaluated on MS-COCO because single-task ablations cannot process VIST or VisDial.
  • Image-text concatenation: Image-text concatenation improves VIST R@1 from 11.6 to 15.6 when five captions and four images are provided.The strategy encourages attention to multiple images within an interleaved sequence; VisDial performance is similar for the ablated model.
  • Retrieval concatenation: Concatenating retrieval examples decreases VIST R@1 from 15.6 to 14.4.The authors hypothesize that retrieving disparate images conflicts with related, coreferential image outputs in these tasks.
  • Scaling properties: Performance generally improves with increasing model size on zero-shot contextual image retrieval for Visual Storytelling and Visual Dialog.Experiments span OPT models from 125M to 6.7B parameters.

B.6. Text Generation Results

The model is competitive on zero-shot VQAv2 while retaining interleaved image retrieval and requiring substantially less training compute than a cited prior method.

  • Zero-shot VQA: 28.51 VQAv2 score is higher than Frozen’s 25.53 and comparable to MAGMA’s 28.35.The evaluation is zero-shot and uses the same normalization techniques as the VQAv2 repository.
  • Multimodal output: FROMAGe produces images interleaved with generated text, whereas Frozen and MAGMA cannot produce such interleaved images.The passage presents this capability as enabling a wider range of tasks.
  • Training efficiency: FROMAGe requires 1 GPU day of training, compared with MAGMA’s 32 GPUs for 1.25 days.The comparison is reported alongside the zero-shot VQAv2 results.

C. Human Evaluation Procedure

Human raters compare stories generated from last-image, preceding-text, or full multimodal context. Full multimodal context yields more coherent, story-like outputs, with a relevance trade-off against image-only conditioning.

  • Evaluation procedure: The evaluation samples 500 examples and compares three conditioning settings: last image only, preceding text only, and preceding images plus text.Raters evaluate generated story quality under each input-context condition.
  • Coherence: Full multimodal context produces text rated as more coherent than both single-modality settings.The comparison is based on head-to-head human ratings summarized in Figure 12.
  • Image relevance: Full multimodal context is rated more image-relevant than text-only conditioning but less relevant than image-only conditioning.Image-only outputs are more factual and caption-like, while multimodal outputs are more story-like and coherent overall.
  • Qualitative behavior: FROMAGe conditions on images and text to produce coherent stories aligned with a corresponding image.The authors connect this behavior to in-context learning and illustrate it with qualitative examples.

D. Current Limitations and Broader Impacts

The paper identifies inherited language-model risks and retrieval-data constraints. It also notes that fixed-corpus retrieval limits novelty while enabling explicit output control and filtering strategies.

  • Language-model risks: FROMAGe inherits repetitive outputs, instruction-following failures, disinformation risks, and toxic or socially biased content from its frozen language-model backbone.The authors suggest future use of human-feedback or instruction-finetuned backbones to improve quality and reduce harmful-content risks.
  • Retrieval scope: Retrieving from a fixed Conceptual Captions image set cannot produce truly novel images outside that corpus.This constraint is contrasted with image-generation models that synthesize images from scratch.
  • Evaluation context: Human evaluations compare outputs conditioned on different contexts, including last-image-only, preceding-text-only, and full multimodal inputs.The interface asks raters to judge coherence and relevance to the last image.
  • Output control: Fixed-corpus retrieval allows explicit control over outputs and possible filtering of inappropriate images.The authors present this as a benefit relative to unconstrained image synthesis, while retaining concerns about dataset bias.
Loading 2301.13823v4…