PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Images with Multimodal Language Models

Jing Yu Koh, Daniel Fried, Ruslan Salakhutdinov

arXiv:2305.17216v3cs.CLcs.CVcs.LG

TL;DR

The paper addresses how to extend text-only language models to multimodal systems that process interleaved image-and-text inputs and produce images and text. GILL maps frozen LLM representations into frozen visual models using lightweight trainable modules, enabling retrieval, generation, and multimodal dialogue. It outperforms Stable Diffusion on longer-form, context-dependent image-generation tasks, while inheriting limitations such as hallucination and repetition.

  • Problem

    Existing multimodal language models have limited support for generating novel images from arbitrarily interleaved image-and-text inputs.

  • Method

    GILL maps frozen text-only LLM representations into pretrained visual models using lightweight trainable mapping and decision modules.

  • Results

    GILL generates text, retrieves images, and generates novel images, outperforming Stable Diffusion on longer-form language and contextual image-generation tasks.

  • Takeaways & Limitations

    A frozen LLM can be efficiently connected to frozen visual models to support interleaved multimodal dialogue and context-dependent image outputs.

  • Takeaways & Limitations

    GILL inherits LLM limitations including hallucinations, repetition, and occasionally incoherent dialogue from OPT-6.7B.

Abstract

from arXiv · show

We propose a method to fuse frozen text-only large language models (LLMs) with pre-trained image encoder and decoder models, by mapping between their embedding spaces. Our model demonstrates a wide suite of multimodal capabilities: image retrieval, novel image generation, and multimodal dialogue. Ours is the first approach capable of conditioning on arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs. To achieve strong performance on image generation, we propose an efficient mapping network to ground the LLM to an off-the-shelf text-to-image generation model. This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs. Our approach outperforms baseline generation models on tasks with longer and more complex language. In addition to novel image generation, our model is also capable of image retrieval from a prespecified dataset, and decides whether to retrieve or generate at inference time. This is done with a learnt decision module which conditions on the hidden representations of the LLM. Our model exhibits a wider range of capabilities compared to prior multimodal language models. It can process image-and-text inputs, and produce retrieved images, generated images, and generated text -- outperforming non-LLM based generation models across several text-to-image tasks that measure context dependence.

1 Introduction

GILL extends frozen text-only LLMs with multimodal input and output capabilities by mapping their representations to visual models. It generates text and images, retrieves images, and handles interleaved multimodal dialogue while improving performance on longer-form language.

  • GILL processes arbitrarily interleaved image-and-text inputs to generate text, retrieve images, and generate novel images.The model can interleave these outputs into coherent multimodal dialogue.
  • The approach maps a frozen text-only LLM’s output embedding space to a frozen image-generation model despite their different text encoders.GILLMapper is an efficient mapping module trained to learn this correspondence.
  • GILL is trained by finetuning a small number of parameters on image-caption pairs rather than interleaved image-text data.The image-generation model need not run during training.
  • GILL outperforms Stable Diffusion on tasks requiring longer-form language, including dialogue and discourse.The comparison concerns language context dependence in image generation.
  • GILL is the first model described here as outputting retrieved images, novel images, and text interleaved for coherent multimodal dialogue.This combines retrieval, generation, and text output in one multimodal system.

2 Related Work

Related work connects text-only language models to multimodal processing and advances text-to-image generation through increasingly capable architectures. GILL builds on these directions while avoiding image-generator execution during training and supporting both retrieval and generation.

  • Prior multimodal language models process image and text inputs to generate text, using visual encoders, adapters, larger models, improved data, or instruction tuning.These approaches generally extend text-only LLMs with visual processing components.
  • GILL differs from retrieval-only FROMAGe by supporting both image retrieval and novel image generation.This allows GILL to produce images beyond the limits of a fixed candidate retrieval set.
  • Large language models contribute few-shot in-context learning and long-text processing capabilities to multimodal adaptation methods.GILL leverages recent Transformer-based LLMs and open-source LLM weights.
  • Text-to-image generation has progressed through conditional GANs, multi-stage and attention-based models, sequence modeling, and diffusion models.Scaling text encoders has also improved generated-image relevance.
  • Unlike computationally intensive end-to-end methods, GILL does not require running the image-generation model during training.Its approach maps a language model to a pretrained visual generation model.

3 Method

GILL adapts a mostly frozen text LLM to process interleaved image-text inputs and produce text, retrieved images, or generated images. Small mapping and decision modules connect the LLM to visual encoders and a text-to-image decoder while supporting retrieval-versus-generation choices.

  • Learning to Process Images: GILL learns mappings that let a frozen LLM process arbitrarily interleaved image-and-text sequences.A visual backbone produces image embeddings, and Wcap maps them into sequences of LLM input vectors trained on image-caption pairs.
  • Learning to Produce Images: Special [IMG] tokens let the LLM learn when to emit image outputs, whose hidden states drive subsequent retrieval or generation.The LLM remains frozen while Eimg is trained to predict the first [IMG] token; subsequent image tokens are generated automatically.
  • Learning to Produce Images: GILLMapper converts [IMG] hidden states and learnt query embeddings into representations consumed by a frozen text-to-image decoder.The lightweight transformer is trained by matching its outputs to embeddings from the frozen generator’s text encoder, allowing the decoder to be omitted during training.
  • Image Retrieval: Image retrieval maps [IMG1] and pooled visual representations into a shared space, then selects the candidate with highest cosine similarity.The text and image mappings are trained with InfoNCE, and retrieval occurs from a prespecified candidate image set.
  • Deciding to Generate or Retrieve: A separately trained decision model chooses retrieval or generation using prompts whose retrieved and generated images receive human preference labels.PartiPrompts examples are labeled ‘gen’ or ‘ret’ from five annotators, retaining roughly 900 examples with agreement of at least 4/5.
  • Data and Implementation Details: The final multitask objective combines captioning, image-token prediction, generation, and retrieval losses.Training uses Conceptual Captions with 3.3M image-text pairs and an OPT-6.7B LLM backbone.

4 Experiments

GILL is evaluated on contextual image generation, multimodal inputs, retrieval, and qualitative multimodal outputs. It increasingly outperforms Stable Diffusion as text and image context becomes longer and more complex.

  • Experimental scope: GILL generates text, retrieves images, and generates novel images from interleaved image-and-text inputs.The experiments primarily evaluate novel image generation while also benchmarking retrieval and broader multimodal capabilities.
  • Evaluation Metrics: The evaluation uses CLIP Similarity, where higher is better, and LPIPS, where lower indicates greater perceptual similarity.These metrics assess relevance and similarity between generated and real images.
  • Qualitative Results: Qualitative results show GILL processing contextual multimodal cues to retrieve and generate appropriate image and text outputs.The model also conditions on image inputs to produce more visually and semantically relevant images.
  • Contextual Image Generation: On VIST, GILL improves over Stable Diffusion as inputs expand from one caption to five captions and then to five captions plus four images.With full multimodal context, CLIP Similarity rises to 0.641 and LPIPS falls to 0.693.
  • Contextual Image Generation: On VisDial, GILL increasingly improves with dialogue context and significantly outperforms Stable Diffusion after all 10 rounds of question-and-answer dialogue.CLIP Similarity improves from 0.622 to 0.645, while LPIPS improves from 0.723 to 0.714.

5 Analysis

Analysis examines retrieval loss, image-token count, context sensitivity, and the GILLMapper architecture. Results indicate that multimodal context and the lightweight transformer mapping improve generalization, while retrieval loss is not required for generation quality.

  • Contextual Image Retrieval: GILL performs comparably or better than prior approaches on contextual image retrieval without deterioration from the image-generation objective.This evaluation uses VIST with image-text inputs.
  • The Effect of Context: With two captions and one image as context, GILL significantly outperforms its result with five text-only captions on VIST.Performance generally improves as input context increases, highlighting the value of multimodal context.
  • Generation-Only Objective: Removing retrieval loss yields CLIP similarity 0.636 and LPIPS 0.694, comparable to the original model’s 0.641 and 0.693.The ablated model generates images and text but cannot retrieve images.
  • GILLMapper Module: GILLMapper, a lightweight transformer, maps frozen LLM embeddings into the input space of a text-to-image generation model.It is substantially better than linear, MLP, and bidirectional-transformer baselines on FID for CC3M and CLIP Similarity on VIST.
  • Number of [IMG] Tokens: Generation generally improves as the number of [IMG] tokens increases, plateauing around r = 4.Lower token counts provide shorter and less expressive inputs to GILLMapper.

6 Conclusion

GILL maps frozen text-only LLMs to frozen visual models so it can process interleaved image-text inputs and produce text, retrieved images, and generated images. The approach improves context-dependent generation and remains modular for stronger future backbones.

  • Conclusion: The method maps embeddings between a frozen pretrained LLM and a frozen pretrained image-generation model.This mapping enables multimodal inputs and generated text, retrieved images, and generated images.
  • Conclusion: GILL processes arbitrarily interleaved image-and-text inputs and produces generated text, retrieved images, and generated images.These capabilities are presented as a unified multimodal system.
  • Conclusion: Mapping to a strong visual model boosts image generation on tasks requiring stronger language-context dependence.The approach is modular and can benefit from stronger language, image-generation, or visual-processing models.

A Limitations

GILL inherits language-model limitations, including hallucinations, repetition, and occasional incoherent dialogue. Its visual processing is also constrained by representing each image with only k = 4 visual vectors and by a relatively small finetuning dataset.

  • Language-model limitations: GILL inherits typical LLM limitations, including hallucinations, repeated content, and neural text degeneration.The OPT-6.7B backbone does not always generate coherent dialogue text.
  • Language-model limitations: Scaling the approach to larger or improved language models is identified as a potential way to address hallucinations, degeneration, and dialogue weaknesses.The paper also suggests instruction finetuning, human feedback, and dialogue-specific training as future directions.
  • Visual limitations: GILL represents each input image with only k = 4 visual vectors, which may omit relevant visual information for downstream tasks.These vectors come from a frozen visual encoder whose information is constrained by its pretraining task.
  • Visual limitations: The model may process images insufficiently accurately or in enough detail to produce accurate or high-quality results.The paper also observes that some prompts yield irrelevant generated images, attributing this partly to finetuning on the relatively small CC3M dataset.
  • Future scaling: GILL’s modular design could benefit from stronger visual, language, and text-to-image generation models, but scaling explorations remain future work.

B Broader Impact

GILL’s multimodal inputs and outputs suggest applications for interactive assistants that provide visual as well as textual responses. However, the system raises risks involving disinformation, generated images of real people, dataset bias, and deployment in high-risk domains.

  • AI Assistants: GILL could support assistants that answer queries with image and text outputs, including iterative image refinement and questions benefiting from visual content.The paper presents these as possible applications rather than current deployment claims.
  • Disinformation and Harms: Interleaved text and realistic generated images could make disinformation more convincing, while the system also inherits risks from text-to-image models such as false images of real people.
  • Bias and Safety: Pretrained components rely on large, noisy, Internet-scraped datasets that may contain undesired biases and malignant stereotypes.The model’s modularity allows its components to be replaced or examined separately.
  • Intended Uses: GILL is a research prototype and is not intended for practical deployment, especially in high-risk or sensitive domains, without further analysis.At 6.7B parameters, it also lacks many abilities of larger language models.

C Deciding to Generate or Retrieve

GILL selects between retrieving and generating an image using a learned decision module. The final linear classifier performs comparably to a heuristic while requiring less hyperparameter tuning, and qualitative samples indicate generally good decisions across many prompts.

  • Evaluation setup: PartiPrompts labels each prompt as “ret” or “gen” according to whether retrieval or generation is judged more appropriate.
  • Baselines: 0.267, 0.389, and 0.451 are the F1 scores for always retrieving, always generating, and randomly deciding, respectively.
  • Heuristic: The heuristic baseline achieves an F1 of 0.261–0.559 depending on its cosine-similarity threshold.At a threshold of 0.5, it achieves an F1 of 0.261.
  • Linear classifier: The linear classifier uses LLM outputs for [IMG] tokens and maximum cosine similarity to predict retrieval versus generation.Its F1 ranges from 0.393–0.552 across probability thresholds, with 0.547 at threshold 0.5.
  • Final decision module: The final model uses the linear classifier because it requires less hyperparameter tuning and performs comparably to the heuristic on quantitative metrics.Qualitative samples show it generally decides correctly whether to retrieve or generate.

D Qualitative Results

Qualitative evaluations indicate that GILL handles complex prompts and dialogue-conditioned image generation effectively. The reported improvements are attributed to the LLM’s stronger text representations and the GILLMapper network.

  • Complex prompts: GILL processes complex text prompts more effectively than Stable Diffusion for many PartiPrompts examples.
  • Dialogue-conditioned generation: On VisDial dialogue inputs, GILL generates more relevant outputs when evaluated against groundtruth images.
  • Interpretation: The paper attributes these improved qualitative results to stronger LLM text representations and the effectiveness of the GILLMapper network.

E.1 Comparison to Prior Multimodal LMs

GILL extends multimodal language modeling with retrieval and generation capabilities, showing stronger performance than Stable Diffusion as dialogue context grows, while remaining weaker on standard MS-COCO caption generation and compared with larger SOTA systems.

  • 0.3178 zero-shot val accuracy on VQAv2, slightly above FROMAGe, Frozen, and MAGMA at similar model sizes and compute.The cited baselines score 0.2851, 0.2553, and 0.2835, respectively.
  • Comparison to Prior Multimodal LMs: GILL’s scores remain below SOTA systems that use substantially more compute and data, including Flamingo at 23,040 TPU days and BLIP-2 at 144 GPU days versus 4 GPU days for GILL.The paper identifies scaling data and parameter counts as a future direction.
  • Increasing Context on VisDial: GILL continues improving with longer Visual Dialogue context, outperforming Stable Diffusion when 7 or more dialogue rounds are provided.Stable Diffusion performance plateaus after 6 rounds, whereas GILL benefits from additional context despite sharing the same image-generation backbone.
  • Image Generation: 12.2 versus 9.22 zero-shot FID on MS-COCO, where GILL performs worse than Stable Diffusion.The passage attributes this result to the task’s limited benefit from the LLM backbone and Stable Diffusion’s caption-focused training.
Loading 2305.17216v3…