PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering

Yuyang Hong, Jiaqi Gu, Qi Yang, Lubin Fan, Yue Wu, Ying Wang, Kun Ding, Shiming Xiang, Jieping Ye

arXiv:2510.14605v2cs.CVcs.AI

TL;DR

KB-VQA requires models to combine visual understanding with external knowledge, but existing RAG methods struggle with precise multimodal queries and relevant retrieval. Wiki-PRF uses processing, multimodal retrieval, and filtering, with an RL-trained VLM for tool use and reasoning. It achieves state-of-the-art results on E-VQA and InfoSeek, scoring 36.0 and 42.8.

  • Problem

    KB-VQA still challenges RAG systems because complex visual content can produce imprecise multimodal queries and irrelevant retrieved information.

  • Method

    Wiki-PRF combines tool-based visual processing, multimodal retrieval, and question-based filtering, while VLM-PRF is trained with reinforcement learning.

  • Results

    Wiki-PRF achieves state-of-the-art performance on E-VQA (36.0) and InfoSeek (42.8).

  • Takeaways & Limitations

    The framework supports precise answers by refining visual retrieval inputs and filtering retrieved knowledge before generation.

  • Takeaways & Limitations

    The study is limited to three retrieval tools, while processing, retrieval, and filtering consume more time than answering.

Abstract

from arXiv · show

Knowledge-based visual question answering (KB-VQA) requires visual language models (VLMs) to integrate visual understanding with external knowledge retrieval. Although retrieval-augmented generation (RAG) achieves significant advances in this task by combining knowledge-base querying, it still struggles with the quality of multimodal queries and the relevance of retrieved results. To overcome these challenges, we propose a novel three-stage method, termed Wiki-PRF, including Processing, Retrieval and Filtering stages. The processing stage dynamically invokes visual tools to extract precise multimodal information for retrieval. The retrieval stage integrates visual and text features to achieve multimodal knowledge retrieval. The filtering stage performs relevance filtering and concentration on retrieval results. To this end, we introduce a visual language model trained with answer accuracy and format consistency as reward signals via a reinforcement learning manner. This enhances the model's reasoning, tool invocation for accurate queries, and filtering of irrelevant content. Experiments on benchmark datasets (E-VQA and InfoSeek) show significant improvements~(36.0 and 42.8) in answer quality, achieving state-of-the-art performance. Code is available at https://github.com/cqu-student/Wiki-PRF

1 Introduction

Wiki-PRF addresses KB-VQA retrieval failures with a three-stage multimodal RAG framework and an RL-trained VLM for tool use and filtering. It achieves state-of-the-art results on E-VQA and InfoSeek.

  • Existing KB-VQA retrieval methods struggle with complex scenes because full-image features can overemphasize prominent but irrelevant objects.A small target such as a statue may be overshadowed by a larger bell tower during retrieval.
  • The three-stage framework processes images, retrieves multimodal knowledge, and filters retrieved context before answer generation.Processing invokes visual tools to extract question-relevant information; retrieval combines visual and textual features; filtering removes redundancy and concentrates relevant knowledge.
  • VLM-PRF uses reinforcement learning to improve reasoning, flexible tool invocation, and retrieval-content quality with limited training data.The approach uses answer accuracy as a reward signal and addresses the lack of intermediate reasoning traces in supervised training data.
  • Wiki-PRF achieves state-of-the-art performance on E-VQA (36.0) and InfoSeek (42.8).

2 Related Work

Prior work applies VLMs and reinforcement learning to visual-language reasoning and specialized perception tasks. Wiki-PRF is positioned within this broader movement toward RL-enhanced VLM capabilities.

  • Knowledge-based VQA methods use unimodal or multimodal external knowledge sources while requiring models to combine visual understanding with question interpretation.Unimodal approaches commonly use text-only datasets as external knowledge sources.
  • Reinforcement learning has been widely applied to VLMs to improve reasoning and specialized capabilities such as visual perception and mathematical reasoning.Prior approaches use task-specific rewards, including verifiable signals such as intersection-over-union for detection.

3 Method

Wiki-PRF uses a three-stage processing, multimodal retrieval, and filtering framework to convert visual-question inputs into task-oriented knowledge for answer generation. VLM-PRF selects visual tools, ranks multimodal retrieval results, filters question-relevant content, and is trained with reinforcement-learning rewards.

  • Overview: Wiki-PRF processes images and questions, retrieves multimodal knowledge, filters the results, and uses the retained knowledge to generate answers.The framework separates processing, retrieval, and filtering before final VLM answer generation.
  • Processing Stage: VLM-PRF invokes captioning, grounding, and flipping tools to produce more precise and comprehensive retrieval queries from the reference image.Captioning supplies semantic information, grounding extracts relevant regions, and flipping mitigates orientation effects.
  • Multimodal Retrieval Stage: The retrieval stage embeds tool-generated queries, searches the knowledge base for top-k images and documents, and selects sections using multimodal similarity.Captioning compares query features with sections, while grounding uses question features to fuse modal information.
  • Filtering Stage: VLM-PRF filters retrieved documents and sections into compact, task-oriented knowledge before the VLM generates the final answer.The filtering input combines directly retrieved information with tool-based search results.
  • Training via Reinforcement Learning: GRPO-based reinforcement learning improves tool selection and information filtering using answer and format rewards for structured model outputs.Format rewards encourage reasoning and tool-call structure, while answer rewards supervise response content.

4 Experiments

Experiments evaluate Wiki-PRF on KB-VQA benchmarks using retrieval and QA metrics, comparisons with baselines, stage analyses, ablations, and efficiency measurements. The method achieves strong benchmark performance while improving retrieval, filtering, tool use, and accuracy with limited training data, although processing and filtering add inference cost.

  • VQA Results: 36.0 on E-VQA and 42.8 on InfoSeek establish Wiki-PRF-7B as state of the art, outperforming previous methods.The method also reaches 39.2% on E-VQA with InternVL3-8B and 77.8 on OK-VQA.
  • Processing and Retrieval: 53.44% Top-1 article recall with tools and 54.89% after reinforcement-learning supervision improve over 45.56% direct image retrieval.RL-trained models also use more diverse tool combinations, with captioning invoked most frequently.
  • Filtering Stage: 65.8% VQA accuracy in the oracle setting shows that the fine-tuned model can locate necessary information effectively from ground-truth articles.This evaluates the filtering stage while holding retrieval information fixed.
  • Reinforcement Learning: Reinforcement learning significantly outperforms supervised fine-tuning for filtering, which the authors attribute to better learning of information-filtering principles and generalization.The comparison uses 2,000 InfoSeek instances with other configurations held identical.
  • Modules and Tools: Combining processing and filtering improves the baseline by 2.54% and 2.02%, while captioning and grounding tools add 1.94% and 0.98%; all tools peak at 39.48%.The ablation uses 10K InfoSeek validation samples.
  • Additional Analyses: Accuracy generally rises with more training samples and retrieved articles, but larger knowledge bases degrade performance through added retrieval noise; K=5 balances accuracy and inference time.Processing, retrieval, and filtering consume more time than answering because of tool invocation and long-text processing.

5 Conclusion

Wiki-PRF introduces a three-stage Process-Retrieval-Filtering framework and VLM-PRF, trained with reinforcement learning for multimodal retrieval-augmented generation. The method achieves state-of-the-art performance on E-VQA and InfoSeek, while currently using three retrieval tools.

  • Wiki-PRF combines processing, retrieval, and filtering to support knowledge-based visual question answering.VLM-PRF invokes tools to process raw information and filters retrieved knowledge during the filtering stage.
  • Wiki-PRF is presented as the first reinforcement learning method for multimodal retrieval-augmented generation.
  • Wiki-PRF achieves state-of-the-art results on the E-VQA and InfoSeek benchmarks.
  • The study is limited to three retrieval tools, with expanded tool integration left for future work.

NeurIPS Paper Checklist

The checklist records that the paper discusses limitations, provides experimental and reproducibility details, and uses public E-VQA and InfoSeek datasets. It also states that the work has no theoretical results and describes tool-based prompting for retrieval and filtering.

  • The paper states that its approach’s limitations are discussed at the end of the paper.
  • The checklist marks theoretical-results assumptions and proofs as not applicable because the paper does not include theoretical results.
  • The paper states that its methods provide sufficient detail to reproduce the experiments.
  • The paper states that E-VQA and InfoSeek are public datasets and that code will be open sourced when appropriate.
  • The retrieval prompt supports captioning, grounding, and flipping tools, while the filtering prompt requires reasoning before selecting useful information.

C.1 Training Loss

During reinforcement-learning training on E-VQA, answer and format rewards increase while task-oriented knowledge tokens decrease. Tool invocation and long-text processing contribute substantially to inference time.

  • Answer and format rewards rise consistently during VLM-PRF-7B training on E-VQA.The passage attributes the trend to learning tool invocation and relevant-information filtering.
  • Task-oriented knowledge tokens decrease progressively as training steps increase.This suggests improved selection and retention of relevant knowledge during learning.
  • Processing, retrieval, and filtering consume more inference time than answering.The additional duration primarily comes from tool invocation and long-text processing.

C.3 Weights of Rewards

The reward-weight ablation on InfoSeek varies the answer-reward weight against the combined format-reward weights. Equal weighting produces the best performance and is used in the experiments.

  • Equal weighting of answer reward and format reward achieves the best InfoSeek performance.The ablation decreases α : (β + γ) from 3 : 1 to 1 : 3 while fixing β and γ at 1.0.
  • The experiments therefore use an equal ratio between α and (β + γ).

C.4 The Number of Selected Sections

The ablation studies examine how the number of retrieved articles and sections affects Wiki-PRF accuracy on InfoSeek and E-VQA. Performance generally improves with more selected sections, but redundant sections can introduce noise.

  • C.4 The Number of Selected Sections: More selected sections generally improve Wiki-PRF performance on InfoSeek and E-VQA.The ablations evaluate top-1 and top-3 retrieved articles or sections during training.
  • C.4 The Number of Selected Sections: When only one article is considered, its overall relevance is the primary determinant of accuracy.
  • C.4 The Number of Selected Sections: Redundant retrieved sections introduce noise and may reduce performance.

D Qualitative Results

Qualitative comparisons and case examples show Wiki-PRF answering varied visual questions, including number-related questions and questions involving distant targets. The examples also illustrate individual and combined tool calls that support information retrieval.

  • D Qualitative Results: Wiki-PRF accurately answers number-related questions and questions whose target subject is far away.The comparison includes plants, buildings, and animals against Vanilla RAG and Wiki-PRF without reinforcement-learning fine-tuning.
  • D Qualitative Results: Tool visualizations show that Wiki-PRF can invoke tools individually or together in different orders.
  • D Qualitative Results: Tool calls extend information retrieval to obtain information that contributes to an answer.

E Broader Impacts of Wiki-PRF

The broader-impacts discussion emphasizes that Wiki-PRF assists VLMs through knowledge retrieval and that restricting the knowledge-base scope can mitigate potentially harmful retrieved information. The supplied examples also show answer outputs for visual questions, including numerical and historical facts.

  • E Broader Impacts of Wiki-PRF: Restricting the knowledge-base scope can mitigate potentially harmful information encountered during retrieval.
  • E Broader Impacts of Wiki-PRF: Wiki-PRF produces numerical answers for mountain sea level and bird weight questions.The examples report 1228 meters for the mountain and approximately 129 grams for the bird, alongside corresponding ground-truth values.
  • E Broader Impacts of Wiki-PRF: Wiki-PRF identifies the Judith Bridge as the immediately prior item in a bridge series, matching the ground truth.
Loading 2510.14605v2…