PaperlayerView exact revision on arXiv

Source-linked AI summary

MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, William W. Cohen

arXiv:2210.02928v2cs.CLcs.AIcs.CV

TL;DR

Existing retrieval-augmented models use external memory but are limited to textual knowledge, while some visual information is unavailable in text. MuRAG introduces a multimodal retrieval-augmented transformer trained with contrastive and generative objectives, and it achieves state-of-the-art results on WebQA and MultimodalQA, outperforming existing models by 10-20% accuracy across distractor and full-wiki settings.

  • Problem

    Prior retrieval-augmented models are restricted to text or structured data, despite multimodal knowledge containing information not covered by text.

  • Method

    MuRAG uses an external memory of images, text, and image-text pairs, with a joint contrastive and generative training objective for multimodal retrieval and generation.

  • Results

    10-20% accuracy: MuRAG outperforms existing models on WebQA and MultimodalQA under both distractor and full-wiki settings.

  • Takeaways & Limitations

    MuRAG demonstrates the effectiveness of integrating multimodal knowledge into retrieval-augmented language generation for open multimodal question answering.

  • Takeaways & Limitations

    The approach requires large computational resources, suffers from pre-training and fine-tuning format misalignment, and uses an expensive visual representation that challenges scaling to large Top-K values.

Abstract

from arXiv · show

While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs. Recently, retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated world knowledge into language generation by leveraging an external non-parametric index and have demonstrated impressive performance with constrained model sizes. However, these methods are restricted to retrieving only textual knowledge, neglecting the ubiquitous amount of knowledge in other modalities like images -- much of which contains information not covered by any text. To address this limitation, we propose the first Multimodal Retrieval-Augmented Transformer (MuRAG), which accesses an external non-parametric multimodal memory to augment language generation. MuRAG is pre-trained with a mixture of large-scale image-text and text-only corpora using a joint contrastive and generative loss. We perform experiments on two different datasets that require retrieving and reasoning over both images and text to answer a given query: WebQA, and MultimodalQA. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20\% absolute on both datasets and under both distractor and full-wiki settings.

1 Introduction

MuRAG addresses the limits of text-only retrieval by augmenting language generation with an external multimodal memory. It is trained and fine-tuned to retrieve images and text and achieves strong open multimodal question-answering results.

  • Motivation: Retrieval-augmented models reduce reliance on parameterized world knowledge but previously could not use multimodal knowledge unavailable in text.These architectures access external memory, with benefits including decreased model size, improved attribution, and adaptation to new information without retraining.
  • Motivation: Visual information-seeking queries require retrieving and reasoning over images because some answers do not exist in text form.The example asks what can be found on White House balconies at Christmas; an image supports the answer “wreaths and garlands.”
  • Approach: MuRAG combines T5 and ViT encoders to represent image-text pairs, images, and text, then retrieves multimodal knowledge from an external memory.The same backbone embeds memory items and queries, and the retrieved items augment generation.
  • Approach: MuRAG is pre-trained on image-text and text-only datasets with joint contrastive and generative losses.The retrieve-and-predict format teaches the model to distinguish relevant memory entries and use multimodal knowledge during generation.
  • Approach: MuRAG is fine-tuned with a similar loss and a two-stage pipeline that scales from small in-batch memory to a statically indexed global memory.The downstream memory contains captioned images and text snippets.
  • Results: 10-20% accuracy: MuRAG outperforms sophisticated baselines on WebQA and MultimodalQA in both distractor and full-wiki settings.The distractor setting uses 40+ candidates, while the full-wiki setting uses 1M candidates.

2 Related Work

Related work spans retrieval-augmented language models, multimodal transformers, and multimodal question answering. MuRAG is positioned as the first retrieval-augmented model using both visual and textual knowledge for open-world multimodal QA.

  • Retrieval Augmented Models: Retrieval-augmented models combine parameterized sequence models with non-parametric memory to infuse language models with world knowledge.Examples include KNN-LM, RETRO, REALM, RAG, and FiD, which use text corpora or Wikipedia passages.
  • Retrieval Augmented Models: MuRAG differs from prior retrieval-augmented models by using knowledge in multiple modalities rather than text-only knowledge.The paper identifies visual and textual knowledge data as the modalities it can use.
  • Multimodal Transformers: Multimodal transformers learn cross-modal representations for tasks including image-text retrieval, image captioning, and visual question answering.They typically combine unimodal visual and textual backbones within a joint transformer.
  • Multimodal Question Answering: WebQA and MultimodalQA evaluate open queries requiring retrieval and reasoning over a large-scale web multimodal corpus.These datasets extend multimodal question answering toward open-world settings.
  • Multimodal Question Answering: MuRAG obtains state-of-the-art results on WebQA and MultimodalQA.The model uses these datasets to study open-world multimodal question answering.

3 Model

MuRAG combines a multimodal backbone encoder with external-memory retrieval and a decoder, allowing retrieved images and text to augment generation. It is trained in stages with joint contrastive and generative objectives over in-batch and indexed global memories.

  • 3.1 Backbone Encoder: MuRAG builds on a ViT–T5 backbone that encodes image-text pairs, image-only inputs, and text-only inputs into multimodal representations.The encoder uses visual and textual embeddings, supports empty modality components, and produces a fused representation for retrieval.
  • 3.2 MuRAG: The retriever applies maximum inner product search between the encoded query and memory candidates to select the Top-K nearest multimodal items.Queries may have any modality, and the memory contains image-text pairs.
  • 3.2 MuRAG: Retrieved memory items are concatenated with the query, encoded jointly, and passed through an attention-based decoder to generate textual outputs token by token.The reader uses raw image patches and associated text as retrieval-augmented input.
  • 3.3 Pre-training: During pre-training, image-plus-prompt queries retrieve textual captions or passages and are optimized with a joint generative and contrastive loss.Retrieved augmentation is omitted for LAION and CC examples when the target caption would otherwise appear directly in memory.
  • 3.3 Pre-training: Pre-training uses LAION, Conceptual-Caption, VQA, and PAQ, while downstream fine-tuning answers text questions using captioned images or text snippets from multimodal memory.The pre-training corpus combines image-text and text-only sources; fine-tuning uses a two-stage in-batch then fixed-global-retrieval pipeline.
  • 3.4 Fine-tuning: Fixed-retrieval training indexes encoded cross-modal pairs, continues optimizing generation over global Top-K retrievals, and keeps stored encodings unchanged during this phase.Inference also uses the fixed encodings.

4 Experiments

MuRAG is evaluated on multimodal question-answering datasets requiring retrieval and reasoning over images and text, with comparisons across retrieval settings, baselines, and ablations. It achieves strong results overall, while visual understanding and retrieval adaptation remain important limitations.

  • Datasets: MuRAG is evaluated on WebQA and MultimodalQA, datasets requiring retrieval of multimodal evidence from large corpora.WebQA questions require 1–2 images or text snippets, while the MultimodalQA subset focuses on text-only and image-only questions requiring one source.
  • Datasets: The experiments compare distractor retrieval against full-wiki retrieval, including searches over 1.1M WebQA text and visual sources.Distractor settings use associated hard negatives and positives, whereas full-wiki settings search the larger Wikipedia corpus.
  • MultimodalQA results: MuRAG improves over AutoRouting by 10+% EM for text questions and 20+% for image questions under both evaluation settings.The model is also reported to handle the full-wiki corpus more effectively than the routing-based baseline.
  • Ablation study: Pre-training corpus ablations show that no pre-training yields an overall score of 23.5, while adding CC, PAQ, and VQA after LAION produces steady improvements.Among individual datasets, LAION yields the highest score.
  • Ablation study: Using only in-batch memory with global-memory generation lowers performance by roughly 2% on both datasets, while fixed retrieval drops more severely because retrieval objectives differ.Pre-training retrieves image and text to text, whereas fine-tuning retrieves text to image and text, motivating downstream retriever adaptation.

5 Examples

The examples show that MuRAG can answer correctly when retrieval is grounded in the oracle image-text pair, and sometimes despite retrieving the wrong pair by using textual clues.

  • MuRAG makes the correct prediction when grounded on the oracle image-text pair.
  • In another example, MuRAG answers correctly despite retrieving the wrong image-text pair.The reported answer is “the angel is holding a dead body”.
  • The authors conjecture that textual clues, rather than the image itself, support this correct prediction.They identify this shortcut learning as concerning and call for better learning algorithms.

6 Conclusion

MuRAG is presented as a visually grounded language generator that retrieves multimodal knowledge from a large-scale corpus and substantially outperforms existing baselines. However, image-reasoning queries remain harder than text-only queries.

  • MuRAG retrieves multimodal knowledge from a large-scale corpus as a visually grounded language generator.
  • MuRAG outperforms existing baselines by a large margin in the reported experiments.
  • Performance on knowledge-seeking queries requiring image reasoning remains significantly lower than on queries requiring only text.The authors describe this gap as leaving ample room for further improvement.

Limitations

The approach depends on large in-batch negative sets during pretraining and faces format misalignment between pretraining and fine-tuning, while visual representations make large Top-K scaling expensive.

  • Pretraining uses negatives from other examples in the same batch because hard negatives are not mined.Competitive retrieval ability therefore requires a sufficiently large batch size and substantial computation resources.
  • The pretraining format, image -> text, differs from the fine-tuning format, text -> image+text.The authors state that this misalignment limits performance.
  • Each image contributes 16x16=196 visual tokens in the reader, making large Top-K values challenging under quadratic attention complexity.

Ethical Statement

MuRAG uses LAION, a large web-crawled visual-language corpus, but residual harmful-content risks and dataset biases remain possible despite automatic filtering.

  • LAION is a publicly available visual-language corpus crawled from the web.
  • Automatic filtering greatly reduces harmful content but cannot fully remove all potential risks from the large dataset.
  • The authors anticipate racial and gender biases in MuRAG and observed gender misrecognition causing 5% of errors.

A Pre-training

MuRAG’s scheduled pre-training begins with LAION before continuing on a fixed-ratio mixture of three datasets, producing strong generation, comprehension, VQA, and retrieval results.

  • Training schedule: Direct mixture training was unstable, so MuRAG first uses LAION for 1M steps before fixed-ratio training on the remaining datasets.The schedule was selected to obtain a stable solution.
  • Evaluation: The mixture-pre-trained model achieves higher than 1.2 CiDEr on the CC12M+3M validation set.
  • Evaluation: The model achieves higher than 55% EM on PAQ, a text-only reading-comprehension task similar to NQ.
  • Evaluation: MuRAG achieves higher than 72% VQA accuracy on the validation set.
  • Evaluation: Multimodal-memory retrieval reaches 85% RECALL@1 from a 4K memory of captions and passages.

B Model Configuration

The paper specifies ViT and T5 encoder-decoder configurations, alongside figures describing mixture-pretraining metrics and retrieval accuracy across datasets.

  • ViT configuration: The ViT configuration uses 16 × 16 patches, 1024 hidden units, 24 layers, and 224 × 224 images.
  • T5 configuration: The T5 encoder-decoder configuration uses 768-dimensional hidden states, 12 encoder layers, 12 decoder layers, and a retrieval weight of 0.5.
  • Reported metrics: Figure 9 reports CiDEr, EM, and VQA accuracy for CC, PAQ, and VQA datasets during mixture pre-training.
  • Reported metrics: Figure 10 reports mixture-pretraining retrieval accuracy over the CC, PAQ, and VQA datasets.
Loading 2210.02928v2…