PaperlayerView exact revision on arXiv

Source-linked AI summary

EchoSight: Advancing Visual-Language Models with Wiki Knowledge

Yibin Yan, Weidi Xie

arXiv:2407.12735v4cs.CV

TL;DR

Knowledge-based visual question answering requires external knowledge that generative models may integrate only weakly. EchoSight uses visual-only retrieval followed by multimodal reranking, and achieves state-of-the-art accuracy on Encyclopedic VQA and InfoSeek.

  • Problem

    Knowledge-based VQA requires external knowledge, while generative models can struggle to integrate it and lack efficient retrieval mechanisms.

    Source: §1 · p. 1, §4.2 · p. 9

  • Method

    EchoSight retrieves knowledge-base candidates using visual-only information, then reranks them using the image and textual query before generating an answer.

    Source: §1 · pp. 1–2

  • Results

    EchoSight achieves state-of-the-art performance, with accuracy of 41.8% on E-VQA and 31.3% on InfoSeek.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results highlight the importance of efficient retrieval and multimodal information integration for improving LLM performance on knowledge-based VQA.

    Source: §5 · p. 9

    Performance depends on knowledge-base quality and comprehensiveness; domain-specific knowledge absent from those databases may yield sub-optimal results.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Knowledge-based Visual Question Answering (KVQA) tasks require answering questions about images using extensive background knowledge. Despite significant advancements, generative models often struggle with these tasks due to the limited integration of external knowledge. In this paper, we introduce EchoSight, a novel multimodal Retrieval-Augmented Generation (RAG) framework that enables large language models (LLMs) to answer visual questions requiring fine-grained encyclopedic knowledge. To strive for high-performing retrieval, EchoSight first searches wiki articles by using visual-only information, subsequently, these candidate articles are further reranked according to their relevance to the combined text-image query. This approach significantly improves the integration of multimodal knowledge, leading to enhanced retrieval outcomes and more accurate VQA responses. Our experimental results on the Encyclopedic VQA and InfoSeek datasets demonstrate that EchoSight establishes new state-of-the-art results in knowledge-based VQA, achieving an accuracy of 41.8% on Encyclopedic VQA and 31.3% on InfoSeek.

1 Introduction

Knowledge-based VQA requires external context that may not be present in images or adequately encoded by visual-language models. EchoSight addresses this with visual-only retrieval followed by multimodal reranking, and reports state-of-the-art performance on two benchmarks.

1 Introduction

  • Knowledge-based VQA requires external context, unlike standard VQA questions answerable from visual content and commonsense.Questions may require historical facts, detailed object properties, or situational context not evident in the image.Source: §1 · p. 1
  • Visual-language models struggle with long-tail encyclopedic knowledge and with connecting visual attributes to entity knowledge.An image of a church alone, for example, provides little information about its construction date.Source: §1 · p. 1
  • EchoSight first retrieves candidates using visual-only information, then reranks them using both the reference image and textual query.The two-stage search narrows the knowledge search visually before refining candidate relevance to the multimodal query.Source: §1 · pp. 1–2
  • EchoSight reports state-of-the-art performance on Encyclopedic VQA and InfoSeek using a retrieval-and-reranking scheme.It first searches using visual information, then reranks candidates using the image and textual query.Source: §1 · p. 2

2 Method

EchoSight answers free-form questions about reference images by retrieving knowledge from a large Wikipedia article-and-image collection. It uses visual-only search followed by multimodal section reranking, then supplies the selected section and question to an LLM.

2.1 Problem Formulation

  • The target is free-form visual question answering grounded in an external, million-scale collection of Wikipedia entity articles and their associated images.The paper represents the collection as paired articles and images.Source: §2.1 · p. 2

2 Method

  • EchoSight retrieves knowledge-base entries from an image, reranks their sections using image-and-question relevance, and gives the selected section and question to an LLM for answering.The pipeline comprises a knowledge base, retriever, reranker, and answer generator.Source: §2.1 · p. 2, §2.2 · p. 2, Figure 2 · p. 3

2.2 Retrieval and Reranking

  • Visual-only search embeds database images and uses cosine similarity to retain the top k matching images and their corresponding Wikipedia entries.The retained candidates are a small subset of the full knowledge base.Source: §2.2 · p. 3
  • The reranker divides candidate articles into titled sections, encodes them alongside 32 Q-Former query tokens, and scores section relevance to the image-question query.The section score uses the maximum pairwise similarity between a query-token embedding and the section’s [CLS] embedding.Source: §2.2 · p. 3
  • The final reranking score combines visual similarity with the section reranking score using a weighted sum, with visual similarity drawn from the best-matched image in the section’s article entry.The selected section is the highest-ranked entry section.Source: §2.2 · pp. 3–4, Figure 2 · p. 3
  • Reranker training uses contrastive hard negatives chosen for visual similarity but contextual difference from the query.The contrastive objective includes positive and negative samples; its temperature controls softmax smoothness.Source: §2.2 · p. 4

2.3 Answer Generation with LLMs

  • An off-the-shelf LLM receives the selected Wikipedia section and target question to generate the answer.The retrieved section supplies contextual knowledge for questions demanding precise and detailed knowledge.Source: §2.3 · p. 4

3 Experiments

Experiments evaluate retrieval and answer generation on E-VQA and InfoSeek. Reranking improves retrieval recall, and EchoSight with reranking outperforms prior VQA approaches.

3.1 Datasets

  • E-VQA uses single-hop questions with a 2M-article Wikipedia base, while InfoSeek evaluation uses its validation split and an E-VQA-filtered 100K-article base.E-VQA contains 221K unique question-answer pairs and 1M samples; InfoSeek comprises 1.3M questions.Source: §3.1 · p. 4

3.2 Metrics

  • Retrieval uses exact-URL Recall@K; question answering uses BEM on E-VQA and VQA accuracy plus Relaxed Accuracy on InfoSeek.Source: §3.2 · pp. 4–5

3.3 Implementation Details

  • A frozen Eva-CLIP retriever feeds a BLIP-2 reranker trained on E-VQA evidence sections with hard negatives, then Mistral answers E-VQA and LLaMA answers InfoSeek.The reranker is transferred to InfoSeek in a zero-shot manner.Source: §3.3 · p. 5

3.4 Results

  • Recall@1 rises from 13.3% to 36.5% on E-VQA and 45.6% to 53.2% on InfoSeek with reranking; EchoSight also significantly outperforms prior VQA work.Source: §3.4 · pp. 5–6, Table 3 · p. 6

3.5 Ablation Study

  • EvaCLIP-8B achieves Recall@20 48.8% versus CLIP-ViT-Large 32.2%; reranking lifts Recall@1 to 23.8% for CLIP-ViT-Large and 36.5% for EvaCLIP-8B.Source: §3.5 · p. 6, Table 4 · p. 6
  • Reranking improves results from Top-5 to Top-500, while hard negatives drawn from retrieved candidates train it on more demanding examples.Final VQA reporting uses a reranking scope of 20 candidates to balance efficiency and quality.Source: §3.5 · p. 7, Table 5 · p. 7
  • With oracle retrieval, LLMs can almost answer; stronger tested LLMs score better, although EchoSight’s performance remains stable across generators.Source: §3.5 · p. 7, Table 8 · p. 7, Table 9 · p. 7

3.6 Qualitative Results

  • Figure 3 compares EchoSight with GPT-4V on landmarks and natural species and includes failure cases alongside ground truth.Source: §3.6 · p. 8, Figure 3 · p. 8

4 Related Work

Visual question answering includes questions answerable from images alone and knowledge-based questions requiring information beyond the image. Prior work explores external knowledge and retrieval-augmented vision-language systems, while EchoSight uses visual-only retrieval followed by multimodal reranking.

4.1 Visual Question Answering

  • Knowledge-based VQA requires information beyond the image, unlike standard VQA, which focuses on objects, attributes, and perceptual details visible in the image.Standard VQA datasets include VQAv1, VQAv2 (Goyal et al., 2017), and VizWiz (Gurari et al., 2018).Source: §4.1 · p. 8
  • Text-only knowledge bases such as GS112K and Wiki21M limit use of visual context, while E-VQA and InfoSeek use Wikipedia as a multimodal knowledge base.E-VQA spans topics such as animals, plants, and landmarks; InfoSeek focuses on information-seeking questions about visual entities.Source: §4.1 · p. 8

4.2 Vison Language Models for VQA

  • VLMs perform well on standard VQA but struggle with knowledge-based questions due to hallucination, internal biases, and limited external-knowledge retrieval.The passage cites hallucination and bias (Li et al., 2023c) and retrieval limitations (Caffagni et al., 2024).Source: §4.2 · p. 9
  • RAG-based VLM systems including KAT (Gui et al., 2021), REVIVE (Lin et al., 2022), and REVEAL (Hu et al., 2023b) show promise on commonsense questions but struggle with encyclopedic tasks.The passage attributes this limitation to restricted ability to fetch and incorporate precise information from large encyclopedic knowledge bases (Mensink et al., 2023).Source: §4.2 · p. 9
  • EchoSight combines visual-only retrieval with multimodal reranking to improve alignment between retrieved textual knowledge and visual content on E-VQA and InfoSeek.The second stage uses both the image and textual query to rerank candidates.Source: §4.2 · p. 9

5 Conclusion

EchoSight uses visual-only retrieval followed by multimodal reranking to improve knowledge retrieval for visual question answering. It reports accuracies of 41.8% on E-VQA and 31.3% on InfoSeek.

5 Conclusion

  • EchoSight improves retrieval through visual-only search followed by multimodal reranking, and the paper reports ablation studies of its components.The reranking stage incorporates both visual and textual information.Source: §5 · p. 9
  • 41.8% on E-VQA and 31.3% on InfoSeek are EchoSight's reported accuracies.The paper describes these as significant performance improvement while comparing with existing state-of-the-art approaches.Source: §5 · p. 9
  • The paper highlights efficient retrieval and multimodal information integration as important to LLM performance on knowledge-based VQA.Source: §5 · p. 9

Limitations

EchoSight's performance depends on its knowledge base and its retrieval process adds computational overhead, limiting suitability for some settings.

Limitations

  • Knowledge-base quality and comprehensiveness constrain performance; specialized queries may be sub-optimal when their domain knowledge is not covered.Source: Limitations · p. 9
  • Multimodal reranking adds computational overhead, making EchoSight less suitable for real-time applications and potentially affecting response time.Source: Limitations · p. 9

A Dataset Details

The dataset section says that dataset details and statistics are summarized in Table 10.

A Dataset Details

A.1 E-VQA

The E-VQA evaluation focuses only on single-hop questions, specifically the Templated, Automatic, and Multi Answer types listed in the table.

A.1 E-VQA

  • The E-VQA evaluation is restricted to single-hop questions.Source: §A.1 · p. 13
  • The included question types are Templated, Automatic, and Multi Answer.Source: §A.1 · p. 13
  • The table covers these three question types within the single-hop evaluation scope.Source: §A.1 · p. 13

A.2 InfoSeek

For InfoSeek, examples with entities missing from the knowledge base are removed; most examples remain in both training and validation.

A.2 InfoSeek

  • 916,385 of 934,048 training examples are kept (98.1%) after examples with missing knowledge-base entities are removed.Source: §A.2 · p. 13
  • 71,335 of 73,620 validation examples are kept (96.9%) after examples with missing knowledge-base entities are removed.Source: §A.2 · p. 13
  • The authors state that these results remain consistent with the dataset’s original setting while accounting for knowledge-base limitations.Source: §A.2 · p. 13

B Vision backbones

DINOv2, evaluated as the visual-only retriever, improved performance on Encyclopedic VQA but substantially reduced it on InfoSeek; the authors retain Eva-CLIP for overall consistency and performance.

B Vision backbones

  • DINOv2 yielded a notable performance improvement on Encyclopedic VQA but a significant drop on InfoSeek, prompting continued use of Eva-CLIP.Tables 11 and 12 present the comparisons.Source: B Vision backbones · p. 13, Table 11 · p. 13, Table 12 · p. 13
  • DINOv2 was benchmarked as the visual-only retriever, unlike CLIP’s visual-language training approach, because DINO uses self-supervised visual-focused training.Source: B Vision backbones · p. 13
  • The authors chose Eva-CLIP to maintain EchoSight’s consistency and overall performance across the evaluated tasks.Source: B Vision backbones · p. 13

C Prompt Template

The appendix gives distinct prompt formats for E-VQA and InfoSeek, with a one-shot example used for InfoSeek to help match its exact-match evaluation format.

C.1 E-VQA

  • The appendix introduces the prompt used for testing E-VQA.Source: §C.1 · p. 13
  • The E-VQA prompt presents a context and question, followed by an answer field.Source: §C.1 · p. 13

C.2 InfoSeek

  • Because InfoSeek uses strict exact-match metrics, its prompt is designed to make generated answers comparable with ground truth, using a one-shot example to keep the format correct.Source: §C.2 · p. 13
  • The InfoSeek system instruction tells the model to answer the user’s question and nothing else.Source: §C.2 · p. 13
  • The one-shot example supplies the context that the southern side of the Alps is next to Lake Como.Source: §C.2 · p. 13
  • In the example, the question asks which body of water the mountain is in or next to, and the short answer is Lake Como.Source: §C.2 · p. 14
  • The final InfoSeek prompt provides a context and question, requests no explanation, and ends with a short-answer field.Source: §C.2 · p. 14

Related papers

Loading 2407.12735v4…