Source-linked AI summary
Retrieval-Augmented Multimodal Language Modeling
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih
TL;DR
Existing multimodal models store knowledge in parameters, creating a need for more scalable access to external knowledge. The paper introduces RA-CM3, which retrieves multimodal documents with CLIP and generates with a CM3 Transformer trained on LAION. RA-CM3 outperforms multimodal baselines on image and caption generation while enabling faithful image generation and multimodal in-context learning.
Problem
Existing multimodal models store learned knowledge in parameters, motivating retrieval from external memory for more scalable and modular knowledge access.
Method
RA-CM3 combines a pretrained CLIP-based multimodal retriever with a CM3 Transformer generator trained on 150M LAION text-image pairs.
Results
RA-CM3 outperforms baseline and existing multimodal models on image and caption generation, including 12 FID and 17 CIDEr improvements over CM3 on MS-COCO.
Takeaways & Limitations
RA-CM3 supports faithful entity-knowledge image generation and multimodal in-context learning for both text and image generation.
Takeaways & Limitations
The retriever is fixed and only the generator is trained; co-training or fine-tuning the retriever remains future work.
Abstract
from arXiv · showhide
Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in text-to-image and image-to-text generation. However, these models store all learned knowledge (e.g., the appearance of the Eiffel Tower) in the model parameters, requiring increasingly larger models and training data to capture more knowledge. To integrate knowledge in a more scalable and modular way, we propose a retrieval-augmented multimodal model, which enables a base multimodal model (generator) to refer to relevant text and images fetched by a retriever from external memory (e.g., documents on the web). Specifically, for the retriever, we use a pretrained CLIP, and for the generator, we train a CM3 Transformer on the LAION dataset. Our resulting model, named Retrieval-Augmented CM3 (RA-CM3), is the first multimodal model that can retrieve and generate both text and images. We show that RA-CM3 significantly outperforms baseline multimodal models such as DALL-E and CM3 on both image and caption generation tasks (12 FID and 17 CIDEr improvements on MS-COCO), while requiring much less compute for training (<30% of DALL-E). Moreover, we show that RA-CM3 exhibits novel capabilities, such as faithful image generation and multimodal in-context learning (e.g., image generation from demonstrations).
1. Introduction
Existing multimodal models encode knowledge in parameters, motivating external memory for scalable, updateable, and potentially more faithful generation. RA-CM3 retrieves multimodal documents and improves image and caption generation while adding faithful generation and multimodal in-context learning.
- Motivation: Multimodal models typically store knowledge implicitly in parameters, requiring 10–80B parameters and 1–10B images to cover broad knowledge.External memory can accommodate knowledge growth and updates, especially for entity-rich tasks.
- Approach: Retrieval-augmented multimodal modeling extends text-based retrieval methods to models that retrieve and generate both text and images.The approach addresses the open challenge of designing multimodal retrievers and generators.
- Approach: RA-CM3 uses a multimodal retriever and a CM3-based generator trained on 150M text-image pairs from LAION.The retrieved multimodal documents provide external information to the generator.
- Results: 12 FID and 17 CIDEr improvements on MS-COCO show RA-CM3 outperforming baseline CM3 on image and caption generation.It also outperforms DALL-E and Flamingo with fewer than 30% of their parameters and training compute.
- Capabilities: RA-CM3 performs faithful entity-knowledge image generation and multimodal in-context learning, including controlled image generation from demonstrations and few-shot image classification.It is described as the first model to perform in-context learning for both text and image generation.
- Contribution: The work proposes a general, modular retrieval-augmentation framework for multimodal models.The framework supports further development of multimodal retrievers and generators.
2. Related work
The paper places retrieval-augmented multimodal modeling within related work on multimodal generation and retrieval-augmented language models.
- Related work: Related works are discussed in detail in §B.
3. Approach
RA-CM3 generalizes retrieval-augmented language modeling to multimodal documents by combining a dense mixed-modal retriever with a CM3 generator. Retrieved documents are prepended as in-context examples, while relevance, modality, diversity, and joint training shape the pipeline.
- Framework: RA-CM3 retrieves relevant multimodal documents from external memory and uses them to generate continuations for multimodal input documents.Its documents may contain arbitrary sequences of text and images.
- Multimodal retrieval: The retriever uses dense retrieval with mixed-modal query and memory encoders, implemented using a pretrained CLIP extension.Maximum Inner Product Search ranks memory candidates before final documents are sampled.
- Retrieval strategy: Retrieved multimodal documents preserve both text and images because their combined context improves generator performance.The paper reports that multimodal retrieval is more informative than retrieving only one modality.
- Retrieval strategy: The retrieval strategy promotes diversity by avoiding redundant candidates and applying Query Dropout to retrieved queries.Candidates overly similar to the query or already retrieved documents are skipped, and Query Dropout drops some query tokens.
- Multimodal generator: The CM3 generator receives retrieved documents prepended to the main input, treating them as in-context examples.The generator is trained with a joint main-sequence and retrieval loss, with α ≥0 weighting the retrieval term.
- Training: Training retrieves using either the text or image part of a full input document and applies teacher forcing to the concatenated sequence.The query excludes the target because retrieving from the full input could make token prediction too easy and would differ from typical inference.
- Training: The retriever remains fixed while only the generator is trained, leaving co-training or retriever fine-tuning as future work.
4. Experiments
The experiments evaluate RA-CM3 on MS-COCO image-to-text and text-to-image generation, alongside training-efficiency and qualitative capability analyses. RA-CM3 achieves stronger generation results than retrieval-free and other baseline models while using the same controlled training setup as vanilla CM3.
- Training setup: RA-CM3 is trained on 150M LAION text-image pairs, with the same architecture, data, and compute as the vanilla CM3 baseline.Its external memory also uses the same training data, controlling the total accessible information.
- Main results: 16 FID outperforms CM3’s 29 and DALL-E’s 28 on MS-COCO caption-to-image generation without finetuning.Lower FID indicates better image-generation quality.
- Qualitative capabilities: RA-CM3 generates faithful images for entity-rich captions involving rare entities such as Ming Dynasty vases and named towers.The model accesses retrieved images in context, whereas vanilla CM3 and Stable Diffusion tend to struggle on these captions.
- Training efficiency: RA-CM3 achieves better FID with less training compute than existing autoregressive Transformer models.Figure 2 compares MS-COCO FID against normalized A100 GPU hours and places RA-CM3 below the baseline trend.
- Main results: 89 CIDEr outperforms CM3’s 72 on MS-COCO image-to-caption generation without finetuning.RA-CM3 also outperforms Parti and Flamingo despite using approximately 3B parameters and 2-shot in-context examples.
- Scope: The experiments report a unified model that performs both image and text generation and supports knowledge-intensive generation and further model analyses.The paper also analyzes scaling laws and key design choices.
5. Qualitative results
RA-CM3 demonstrates knowledge-intensive image generation and multimodal in-context learning, including rare entity compositions, image infilling, controlled generation, and few-shot classification.
- Multimodal in-context learning: RA-CM3 shows multimodal in-context learning for image generation and image-to-text classification.The classification setup uses non-semantic labels and one or more demonstration pairs.
- Knowledge-intensive generation: RA-CM3 generates faithful images for captions involving rare entities and compositions, whereas non-retrieval baselines often struggle.Examples include entity-rich captions and unusual combinations such as a French flag on the moon.
- Knowledge-intensive generation: RA-CM3 recovers both legs and skis during image infilling, while vanilla CM3 tends to infill only legs.The task requires world knowledge about skiing.
- Controlled generation: Manually supplied in-context images can edit image content, such as changing a person’s black jacket to red.The examples replace retrieved context with user-specified demonstrations.
- Controlled generation: RA-CM3 follows visual styles specified by demonstration images, enabling image-based control beyond textual captions.Demonstrations such as a triangular wooden house and orange autumn leaves guide the generated image’s visual characteristics.
- Multimodal in-context learning: 0.78 accuracy at k = 1 and 0.90 accuracy at k = 8 show that RA-CM3 performs well with one demonstration and improves with more.The reported k values are 1, 2, 4, and 8, and accuracy improves over baseline CM3 across them.
6. Conclusion
The paper presents RA-CM3 as a retrieval-augmented multimodal model for generating text and images, with lower training compute and capabilities including knowledge-intensive generation and multimodal in-context learning.
- Conclusion: RA-CM3 retrieves and refers to external memory while generating both images and text.The paper frames this as a general retrieval-augmented multimodal model.
- Conclusion: The framework is intended to be general and modular, motivating future work on multimodal retrievers, generators, modalities, and prompting.The authors identify these as research avenues opened by the work.
A. Ethics and societal impact
The authors discuss ethical risks from web-scale multimodal training data and identify possible societal benefits from lower compute, long-tail knowledge, and provenance.
- Risks and safeguards: RA-CM3 may still produce problematic text or images despite extensive filtering of web-collected training data.The authors discourage use in high-risk or sensitive domains and for generating images of people.
- Potential benefits: The authors identify potential benefits including energy savings, better coverage of long-tail knowledge, and improved interpretability through provenance.These benefits are presented as potential societal impacts of retrieval augmentation.
B. Related work
Multimodal models store knowledge in parameters, while retrieval augmentation adds external memory to generation. RA-CM3 extends this idea to multimodal retrieval and generation of both text and images.
- Related limitations: Existing models store knowledge inside model parameters, requiring substantial parameters and training data.Examples cited include roughly 10B parameters and 1B images.
- RA-CM3: RA-CM3 is presented as the first retrieval-augmented multimodal model that retrieves and generates both text and images.It augments generation with relevant examples from external memory.
- Retrieval augmentation: Retrieval-augmented language models retrieve relevant documents from external memory to inform generation.Prior work primarily uses text passages as the external memory.
C.1. Intrinsic evaluation of CLIP-based retriever
The CLIP-based retriever performs reasonably well for multimodal retrieval, while RA-CM3 maintains consistent perplexity improvements over vanilla CM3 across the evaluated model scales.
- Retriever evaluation: CLIP-based retrieval performs reasonably well across text-to-image, image-to-text, text-to-mixture, and image-to-mixture retrieval.The mixed-modal variants use an encoder based on CLIP.
- Scaling evaluation: RA-CM3 provides consistent perplexity improvements over vanilla CM3 across model scales from 125M to 2.7B parameters.Models were trained with the same amount of compute and evaluated on MS-COCO validation perplexity.
- Scaling evaluation: No diminishing returns were observed in the studied 125M–2.7B parameter range.This observation is based on perplexity evaluations using equal compute.
C.3. Analysis of RA-CM3 designs
RA-CM3 performance depends on relevant, multimodal, diverse retrieval and joint training of the main and retrieved documents. These design choices improve retrieval-augmented generation relative to simpler alternatives.
- Retrieval relevance: Relevant documents are crucial during both training and inference, as replacing them with random documents causes significant performance drops.The strongest baseline uses retrieved documents at both training and inference time.
- Retrieval modality: Retrieving multimodal documents performs better than retrieving only images or only text.The mixed-modal retriever can return documents containing both images and text.
- Retrieval diversity: Selecting the top K documents without diversity can perform slightly worse than no retrieval augmentation, whereas redundancy-avoidance improves performance.The analysis identifies diversity in retrieved documents as important.
- Generator training: Jointly optimizing token prediction for the main and retrieved documents improves performance, with α = 0.1 working well.Setting α too large, such as α = 1, hurts training.
D.1. Fair comparison of the retrieval-augmented model and non-retrieval-augmented model
Controlled experiments compare RA-CM3 and vanilla CM3 under the same architecture, data, and compute. RA-CM3 substantially outperforms the non-retrieval baseline while retaining the same user-facing task definition and enabling additional generation benefits.
- Fair comparison: RA-CM3 substantially outperforms vanilla CM3 in image and text generation under matched architecture, training data, and compute.The external retrieval memory also consists of the same training data.
- Fair comparison: A 1.3B-parameter RA-CM3 can outperform a 2.7B-parameter vanilla CM3.This comparison is reported from the scaling analysis.
- Usage: The retriever automatically fetches relevant images or text from the user’s input and adds them to the generator context without requiring additional user input.Users may also specify in-context examples themselves.
- Capabilities: Retrieval augmentation opens benefits including improved explainability, faithfulness, and controllability in generation.These benefits are stated within the same task definition as non-retrieval-augmented models.
- Retrieval count: Using K = 2 retrieved documents worked best in the MS-COCO caption-to-image generation analysis.The paper relates this result to captions involving one or two objects.