Source-linked AI summary
Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
TL;DR
MLLMs can struggle with specific or compositional questions that require knowledge beyond their image, prompt, and learned parameters. Wiki-LLaVA retrieves documents and passages hierarchically and supplies them as additional context. The paper reports higher accuracy with external knowledge, especially on InfoSeek, while incorrect retrieval can slightly degrade scores.
Problem
MLLMs may struggle with specific or compositional questions requiring external knowledge that is difficult to encode because long-tail information is scarce in training data.
Source: §1 · p. 1, §3.1 · p. 3Method
Wiki-LLaVA augments an MLLM with a hierarchical pipeline that retrieves relevant documents and passages from an external knowledge base and adds them to the model’s input context.
Source: §3 · p. 3, §3.1 · p. 4Results
External knowledge significantly increases accuracy, especially on InfoSeek, while incorrect entity retrieval can slightly degrade scores by supplying uninformative passages.
Source: §4.4 · p. 6Takeaways & Limitations
The experiments demonstrate the approach’s effectiveness and its capability to maintain MLLM proficiency across different tasks.
Source: §5 · p. 8The approach remains constrained by document retrieval and selection: better embedding spaces and more efficient ways to distinguish appropriate retrieved items are needed.
Source: §4.5 · p. 7
Abstract
from arXiv · showhide
Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality. As research is being carried out to design novel architectures and vision-and-language adapters, in this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. Relevant passages, using this approach, are retrieved from the external knowledge source and employed as additional context for the LLM, augmenting the effectiveness and precision of generated dialogues. We conduct extensive experiments on datasets tailored for visual question answering with external data and demonstrate the appropriateness of our approach.
1. Introduction
Multimodal LLMs extend language models to vision-and-language tasks, but can struggle with specific or compositional questions that depend on long-tail knowledge. Wiki-LLaVA adds hierarchical retrieval from external documents to provide relevant information as context for answering.
- MLLMs can struggle with highly specific queries, compositional reasoning, and knowledge that is scarce in their training data.Benchmarks such as InfoSeek and Encyclopedic-VQA evaluate questions requiring external data, but the cited works had not developed architectures specifically for that purpose.Source: §1 · pp. 1–2
- Wiki-LLaVA retrieves relevant passages from an external document knowledge base and feeds them to an MLLM without changing its structure.The paper presents this as the first MLLM augmented with a retrieval module, to the authors’ knowledge.Source: §1 · p. 2
- The authors report that external retrieval improves answering capabilities and that experiments support their design choices.They frame the work as an initial step toward retrieval-augmented MLLMs.Source: §1 · p. 2
2. Related Work
Related work spans multimodal language models, retrieval-augmented text generation, and knowledge-based visual question answering. Wiki-LLaVA applies hierarchical retrieval over multimodal documents to augment an MLLM.
- Early MLLMs combined pretrained language models with vision-and-language models, while later systems integrated image features into LLMs using cross-attention or Q-Former blocks.FROMAGe, Kosmos-1, and MiniGPT-4 continued this line of work, and LLaVA-family models introduced multimodal instruction tuning.Source: §2 · p. 2
- Textual retrieval-augmented generation expands model inputs with passages from external sources, including documents retrieved from large corpora such as Wikipedia.Prior work also adapted retrieval for generation or incorporated latent knowledge retrievers into pretraining.Source: §2 · p. 2
- Prior vision-and-language retrieval work included memory-based world knowledge [13] and retrieval for tasks such as image captioning [35] [36].Wiki-LLaVA instead applies retrieval augmentation to MLLMs using a hierarchical strategy over multimodal documents.Source: §2 · p. 2
- Visual knowledge-question answering benchmarks such as Encyclopedic-VQA and InfoSeek test questions about specific entities that often require external information.A common approach retrieves an entity from the query image, then uses its name to access passages in a knowledge base.Source: §2 · p. 2
3. Proposed Method
Wiki-LLaVA augments an existing MLLM with retrieved external knowledge while largely preserving its design. Its method retrieves documents and passages in two stages, adds selected text to the model’s input, and also investigates fine-tuning with mixed training examples.
3. Proposed Method
- Wiki-LLaVA combines a visual encoder, an external knowledge base, and hierarchical retrieval to supply relevant documents and passages as additional MLLM context.The visual encoder provides visual context and serves as a query to retrieve from the knowledge base.Source: §3 · p. 3
3.1. Knowledge-based Augmentation
- CLIP matches the input image to document titles, then Contriever ranks chunks within retrieved documents by question similarity; the selected raw passages join the MLLM prompt.Retrieving k documents and n passages per document yields k · n passages; the prompt also includes image tokens, the system prompt, and the user question.Source: §3.1 · pp. 3–4
3.2. Training
- Fine-tuning uses question-answer pairs requiring external knowledge mixed in the same mini-batch with pairs that do not require it.The mixing approach is intended to address the potential loss of capability on tasks not requiring external knowledge.Source: §3.2 · p. 4
4. Experiments
The experiments evaluate Wiki-LLaVA on knowledge-based VQA datasets, testing retrieval, fine-tuning, and preservation of general MLLM performance. Results show gains from retrieved context, with accuracy depending on entity retrieval and the selected fine-tuning data.
4.1. Datasets
- Encyclopedic-VQA has about 221k question-answer pairs across 16.7k entities and a 2M-article knowledge base; InfoSeek has 1.3M triplets across about 11k entities.Experiments evaluate Encyclopedic-VQA on its test set after filtering out two-hop questions, and InfoSeek on its validation set; InfoSeek's test answers and entities were unavailable at submission.Source: §4.1 · pp. 4–5
4.2. Implementation Details
- The retrieval pipeline embeds Wikipedia documents in 600-character chunks and uses the CLIP ViT-L/14@336 encoder for both image input and initial visual retrieval.Fine-tuning uses low-rank adapters, and entity retrieval uses approximate kNN search with Faiss and an HNSW index.Source: §4.2 · p. 5
4.3. Evaluation Protocol
- Evaluation compares answers with and without a Wikipedia knowledge base, reporting accuracy on the Encyclopedic-VQA test split and InfoSeek validation split.The external knowledge bases contain 2M and 100k Wikipedia pages, respectively.Source: §4.3 · p. 5
4.4. Experimental Results
- 36.9% R@1 and 66.1% R@10 on InfoSeek contrast with 3.3% R@1 and 9.9% R@10 on Encyclopedic-VQA, whose knowledge base is larger.The results use CLIP image queries to retrieve Wikipedia titles; the knowledge bases contain 100k and 2M items, respectively.Source: §4.4 · p. 5, Table 1 · p. 5
- Oracle entities improve accuracy by 13.8% on Encyclopedic-VQA and 22.6% on InfoSeek over the best CLIP-based configurations.On InfoSeek, using n = 2 and n = 3 chunks also improves accuracy by 2.1 and 3.4 points, respectively; passages from different entities do not always help.Source: §4.4 · p. 6, Table 2 · p. 6
- Visual instruction-tuning data improves InfoSeek fine-tuning by 1.9 accuracy points over using dataset triplets alone.On Encyclopedic-VQA, instruction tuning does not improve performance, but does not degrade the original results.Source: §4.4 · p. 6, Table 3 · p. 6
- LLaVA-Instruct samples better preserve original LLaVA-1.5 performance across evaluation benchmarks, with the largest deterioration on MME and slight gains on MMMU and POPE.The comparison covers MME, MMMU, MMBench, and POPE.Source: §4.4 · p. 7, Table 4 · p. 7
4.5. Limitations and Future Works
- The authors identify better document-embedding spaces and more efficient selection of relevant documents as needed research directions; MLLMs may need new architectures to judge retrieved-item appropriateness.The paper also notes retrieval-augmented approaches' adaptability across domains and loose coupling between pretrained information and retrievable data.Source: §4.5 · p. 7
5. Conclusion
Wiki-LLaVA augments an existing MLLM with external knowledge through hierarchical document retrieval, incorporating selected passages into model context; experiments report effectiveness and retained proficiency across tasks.
- Extensive experiments demonstrate the solution’s effectiveness and its capability to maintain MLLM proficiency across different tasks.Source: §5 · p. 8
- Wiki-LLaVA augments an existing MLLM with external knowledge to improve its effectiveness on questions and dialogues.Source: §5 · pp. 7–8
- Its hierarchical retrieval architecture selects document parts for inclusion in the MLLM’s input context.Source: §5 · p. 8