Source-linked AI summary
REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, Alireza Fathi
TL;DR
Existing visual-language models store world knowledge in large parameter sets and require retraining when that knowledge changes. REVEAL instead encodes diverse multimodal knowledge into a memory, jointly trains retrieval and generation, and achieves state-of-the-art results on knowledge-intensive VQA and image captioning.
Problem
Large visual-language models require massive scale and retraining when world knowledge is updated, while retrieval training lacks direct supervision for identifying useful knowledge entries.
Method
REVEAL encodes diverse multimodal knowledge into a large-scale memory and jointly pre-trains its memory, retriever, and generator end-to-end.
Results
REVEAL achieves state-of-the-art results across knowledge-intensive visual question answering and image captioning, including 59.1% accuracy on OKVQA.
Takeaways & Limitations
Using diverse knowledge sources with end-to-end retrieval training enables REVEAL to retrieve meaningful entries and perform strongly without relying on a large language model.
Abstract
from arXiv · showhide
In this paper, we propose an end-to-end Retrieval-Augmented Visual Language Model (REVEAL) that learns to encode world knowledge into a large-scale memory, and to retrieve from it to answer knowledge-intensive queries. REVEAL consists of four key components: the memory, the encoder, the retriever and the generator. The large-scale memory encodes various sources of multimodal world knowledge (e.g. image-text pairs, question answering pairs, knowledge graph triplets, etc) via a unified encoder. The retriever finds the most relevant knowledge entries in the memory, and the generator fuses the retrieved knowledge with the input query to produce the output. A key novelty in our approach is that the memory, encoder, retriever and generator are all pre-trained end-to-end on a massive amount of data. Furthermore, our approach can use a diverse set of multimodal knowledge sources, which is shown to result in significant gains. We show that REVEAL achieves state-of-the-art results on visual question answering and image captioning.
1. Introduction
REVEAL addresses the scale and update limitations of models that memorize world knowledge in their weights by using a multimodal memory and end-to-end retrieval training. It combines diverse knowledge sources and reports state-of-the-art results on knowledge-intensive visual-language tasks.
- Large-scale visual-language models require massive parameters, data, and computation, and must be retrained when world knowledge changes.
- REVEAL decouples knowledge memorization from reasoning by transforming world knowledge into a key-value memory that supports knowledge-intensive queries.
- The model integrates Wikipedia passages and images, WikiData triplets, web image-text pairs, and visual question-answering data as external knowledge sources.
- Unlike single-modality retrieval approaches, REVEAL stores multimodal knowledge in a unified memory and retrieves it using multimodal query encodings.
- REVEAL trains its multimodal memory, retriever, and generator end-to-end on massive data from diverse knowledge sources.
- An attentive fusion layer provides retrieval supervision by considering multiple retrieved entries jointly and injecting retrieval scores into attention.
- 59.1% accuracy on OKVQA establishes a new state of the art while using an order of magnitude fewer parameters than previous methods.
2. Related Work and Background
Knowledge-based visual question answering requires retrieving external knowledge, while retrieval-augmented models must often learn relevance without explicit retrieval supervision. Prior work uses structured or unstructured knowledge, image-derived tags, distant supervision, auxiliary tasks, or implicit knowledge from large language models.
- Knowledge-based VQA benchmarks require external knowledge beyond what can be directly observed in input images.
- Some VQA datasets formulate questions that can be answered by retrieving relevant triplets from structured knowledge graphs.
- Existing approaches retrieve explicit knowledge from structured graphs or unstructured text, often using image tags, distant supervision, or auxiliary tasks.
- Other methods use image captions to convert visual inputs into text and augment them with knowledge from pretrained large language models.
- Most retrieval-augmented tasks lack clear indications of which knowledge entries are relevant for generating answers, complicating retrieval-model training.
3. Method
REVEAL encodes multimodal knowledge into a unified key-value memory, retrieves top-K entries for a multimodal query, and fuses them with the query to generate text. Its design supports end-to-end updating of the memory, retriever, and generator while compressing values for scalable retrieval.
- Core architecture: The workflow encodes the query and knowledge items, indexes unified key/value pairs, retrieves top-K entries, and fuses them through attentive knowledge fusion.The fusion layer injects retrieval scores as a prior during attention, enabling joint training across the pipeline.
- Core architecture: REVEAL models knowledge-intensive generation by retrieving a set of memory entries M conditioned on query x and generating output y from both x and M.The retrieval set is treated as a latent variable, with the overall likelihood marginalized over possible memory subsets.
- Memory construction: Knowledge items from diverse corpora are stored as key/value memory entries, with keys used for indexing and values preserving token-level knowledge information.The unified corpus can contain image-text pairs, text-only or image-only items, and knowledge-graph triplets; memory embeddings are recomputed during training.
- Memory construction: Perceiver-based value compression maps each knowledge item to c compressed tokens, using c = 32 in the experiments, making retrieval of large K feasible.Compression addresses the storage and quadratic-attention costs of retaining full token sequences for hundreds of millions of items.
- Retrieval: The retriever scores entries within a unified multimodal memory using corpus gating and query-entry relevance, then returns the highest-probability top-K entries.The retrieval probability combines a corpus-level gate with a temperature-scaled relevance score; 10% of retrieved items are re-encoded to support encoder updates.
- Generation: The generator concatenates the query with compressed retrieved values and uses retrieval probabilities as an attention mask before decoding the textual output.The attentive fusion module guides attention toward important retrieved items and provides a gradient path to the retriever.
4. Generative Pre-Training
REVEAL is pre-trained on a massive image-text corpus while indexing diverse multimodal knowledge into a memory for retrieval-augmented generation. Its pipeline warm-starts retrieval, encodes multiple knowledge sources, and supports distributed memory search.
- Pre-training data: REVEAL pre-trains on roughly 1.3 billion filtered image-caption pairs from a 3-billion-pair Web-Image-Text corpus.Captions shorter than 50 characters are removed before pre-training.
- Pre-training objective: The pre-training objective conditions on an image and text prefix to autoregressively generate the remaining text sequence.The objective maximizes the probability of tokens after a randomly sampled prefix.
- Retriever initialization: A pseudo-ground-truth retrieval dataset warm-starts the retriever and avoids irrelevant retrievals that would provide weak training signals.Wikipedia image-caption pairs are paired with surrounding passages as initial retrieval targets.
- Knowledge sources: The memory combines Wikipedia-Image-Text, Conceptual, VQA-v2, and WikiData as complementary multimodal knowledge sources.These sources include image-text pairs, question-answer passages, and structured knowledge-graph information.
- Model initialization: The model warm-starts multimodal components from pre-trained ViT and T5 checkpoints before end-to-end generative pre-training.The query and key heads and attentive fusion layers are initialized from T5 components.
- Distributed retrieval: REVEAL distributes memory embeddings across training devices for approximate top-k maximum-inner-product retrieval.TPU-KNN performs local searches after synchronizing each query across devices.
5. Experimental Results
REVEAL achieves strong results on knowledge-based VQA and image captioning, while ablations show benefits from diverse knowledge sources, attentive retrieval training, and memory updates. The model is evaluated across benchmark tasks and component studies.
- Knowledge-based VQA: REVEAL achieves 59.1% OKVQA accuracy, exceeding KAT by +6.0% and ReVIVE by +2.5%.These comparisons use prior state-of-the-art methods with T5-Large generators.
- Knowledge-based VQA: 52.2% A-OKVQA accuracy is +3.6% higher than the previous best, GPV-2.The result is reported for the A-OKVQA benchmark.
- Image captioning: REVEAL outperforms strong baselines on both MSCOCO and NoCaps image captioning benchmarks under CIDEr evaluation.The 1.4B-parameter REVEAL-Large also outperforms 2.1B-parameter CoCa and 80B-parameter Flamingo.
- Knowledge sources: WIT is the most informative isolated source, while removing CC12M, VQA-v2, or WikiData lowers complete-system accuracy by 1.3%, 0.6%, or 1.1%, respectively.The ablation compares Only-One-Left and Leave-One-Out settings on OKVQA.
- Retrieval training: The attentive fusion layer is evaluated against frozen ALIGN retrieval and end-to-end alternatives including Attention Distill, EMDR2, and Perplexity Distill.Retrieval quality is measured by whether the correct passage appears among the top 10 or 100 results, alongside fine-tuned OKVQA accuracy.
- Knowledge sources: Combining pairs of knowledge sources consistently improves performance on OKVQA.The pairwise study evaluates all pairs among the four knowledge sources.
- Knowledge modification: With only 10% of knowledge available during fine-tuning, restoring the removed knowledge during inference yields 51.8 accuracy, +6.7 higher than leaving it removed.The result supports updating memory contents without retraining model parameters.
6. Conclusion
REVEAL jointly trains a retriever and generator to use diverse multimodal knowledge sources for knowledge-intensive visual-language tasks. Pre-training over four knowledge corpora produces state-of-the-art results in visual question answering and image captioning.
- Conclusion: REVEAL jointly trains a retriever and generator to return multiple entries from diverse multimodal knowledge sources.The model is pre-trained on a massive image-text corpus with four diverse knowledge corpora.
- Conclusion: The paper reports state-of-the-art results on knowledge-intensive visual question answering and image captioning tasks.Future work considers attribution and broader multimodal applications.