Source-linked AI summary
Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning
Qiyou Liu, Yong Zhang, Jianjie Luo, Zhenguo Yang, Yi Yu
TL;DR
KB-VQA reasoning is hindered when heterogeneous multimodal evidence is concatenated into long, unstructured prompts. SCoRe progressively acquires, selects, and compresses context into relational triplets, and experiments on OK-VQA and A-OKVQA show consistent improvements over state-of-the-art methods.
Problem
Long, unstructured prompts can contain irrelevant context and lack explicit relational structure, making heterogeneous evidence difficult to use for KB-VQA reasoning.
Method
SCoRe uses Context Acquisition, Context Selection, and Context Compression with RLD to convert multimodal evidence into compact entity-relation triplets.
Results
SCoRe consistently outperforms state-of-the-art zero-shot and few-shot methods on the OK-VQA and A-OKVQA benchmarks.
Takeaways & Limitations
The results indicate that explicitly structuring context into relational representations improves KB-VQA reasoning beyond simply increasing or filtering textual context.
Abstract
from arXiv · showhide
Knowledge-based Visual Question Answering aims to answer questions about an image by integrating external knowledge with visual and textual information. Recent approaches often rely on in-context learning to prompt Large Language Models (LLMs) with multimodal context in a zero-shot or few-shot manner. However, we observe that directly concatenating heterogeneous visual descriptions and retrieved knowledge into long, unstructured prompts often degrades reasoning performance, due to both excessive irrelevant context and the lack of explicit relational structure. In this paper, we propose an LLM-based Structured Context Reasoning (SCoRe) framework that infers both explicit and implicit relationships for prediction. SCoRe consists of three stages: Context Acquisition, which generates diverse visual notes and retrieves explicit knowledge via an efficient two-stage multimodal retrieval strategy; Context Selection, which filters relevant visual, explicit, and implicit knowledge using LLM-guided selection; and Context Compression, which performs Relational Logic Distillation (RLD) to transform raw text into explicit entity-relation triplets. These relational triplets serve as a concise and structured prompt for final answer prediction. Extensive experiments on the OK-VQA and A-OKVQA benchmarks demonstrate that SCoRe consistently outperforms state-of-the-art methods.
I. INTRODUCTION
KB-VQA requires organizing heterogeneous visual, textual, and external knowledge for multistep reasoning. SCoRe addresses limitations of flat prompting by progressively selecting and compressing multimodal context into relational representations, with consistent gains on OK-VQA and A-OKVQA.
- KB-VQA integrates external knowledge with visual and textual information, often requiring implicit concepts and multistep reasoning.
- Flat prompts concatenate image descriptions, retrieved knowledge, and instructions, introducing redundant information and obscuring entity relationships.
- SCoRe progressively acquires, selects, and compresses multimodal context into compact relational representations for LLM-based reasoning.
- SCoRe consistently improves over state-of-the-art zero-shot and few-shot methods on OK-VQA and A-OKVQA, indicating that context structure matters beyond context quantity.
- A two-stage progressive multimodal retrieval strategy reduces the retrieval search space while preserving high recall of relevant knowledge.
- RLD transforms evidence into structured, reasoning-oriented context by modeling entity relations.
II. RELATED WORK
The paper situates SCoRe within knowledge-based VQA, multimodal relation extraction, and LLM-based reasoning. Its context-acquisition component combines diverse visual notes with two-stage multimodal retrieval to construct a smaller, structured evidence pool.
- Knowledge-based VQA: Knowledge-based VQA integrates external knowledge with visual and textual information to support implicit-concept and multistep reasoning.
- LLM-based reasoning: Prior LLM-based approaches concatenate image descriptions, retrieved knowledge, and instructions into a single prompt, leaving entity relationships implicit.
- Relation Extraction: SCoRe uses Relational Logic Distillation to decompose heterogeneous evidence into structured relational triplets for final answer prediction.
- Context Acquisition: The Context Acquisition stage represents images with object attributes, question-guided captions, and OCR text as separate visual notes.
- Context Acquisition: SCoRe retrieves explicit knowledge through title-level coarse retrieval followed by paragraph-level fine-grained retrieval using multimodal query representations.
- Context Acquisition: The candidate passage set is much smaller than the full passage set, compressing retrieval scope while preserving the strategy’s intended fine-grained selection process.
B. Context Selection
Context Selection reduces redundancy while retaining question-relevant visual and explicit knowledge, and it also retrieves implicit knowledge from the LLM using image captions and the question.
- Context Selection: LLMs select relevant visual knowledge from visual notes and explicit knowledge from the top-k2 retrieved passages.
- Context Selection: Visual selection uses captions, detected objects, OCR strings, and the question as its input context.
- Context Selection: Explicit-knowledge selection combines image captions, the question, and the retrieved top-k2 passages.
- Context Selection: Implicit knowledge is retrieved from the LLM using image captions and the question to maintain alignment.
C. Context Compression
Context Compression uses Relational Logic Distillation to convert heterogeneous textual evidence into refined explicit and implicit entity-relation structures for reasoning.
- C. Context Compression: Relational Logic Distillation decomposes textual evidence through entity extraction, relation extraction, and relation refinement.The procedure bridges unstructured evidence and logical reasoning by guiding the LLM toward explicit relational structures.
- C. Context Compression: The framework initializes visual entities from detected objects and separately extracts explicit and implicit knowledge entities with an LLM.The resulting entity lists combine visual entities with entities extracted from explicit or implicit knowledge.
- C. Context Compression: Relations are represented as triplets (ep, r, eq), where r specifies the relation between distinct entity pairs.The framework extracts both explicit and implicit relationships between entities.
- C. Context Compression: Relation refinement filters initial relational triples by retaining key entities identified from the image caption and question.The final relation set is produced after removing irrelevant relationships.
- C. Context Compression: The bidirectional relation symbol ∗↔ indicates that two entities can reach each other through one or more relations.This notation captures multi-step connectivity between entities.
D. Prediction
The QA model predicts answers from the structured relation set and question, focusing reasoning on entity dependencies instead of organizing raw context.
- D. Prediction: The QA model conditions answer prediction on the structured relation set Ri and question Qi.Replacing unstructured context with relational representations directs reasoning toward entity dependencies.
A. Experiment Settings
Experiments evaluate SCoRe on OK-VQA and A-OKVQA using established benchmark protocols, specified models, retrieval components, and zero- or few-shot settings.
- A. Experiment Settings: Experiments use the OK-VQA test set and A-OKVQA validation set.These are the two benchmark datasets used for evaluation.
- A. Experiment Settings: DeepSeek-V3 processes context, while Mistral-7B and Gemma-7B serve as QA models.Mistral-7B and Gemma-7B use greedy decoding; DeepSeek-V3 is accessed through an API.
- A. Experiment Settings: Table I compares methods on the OK-VQA and A-OKVQA datasets, highlighting best and second-best scores.The table distinguishes the strongest scores using bold and underline formatting.
- A. Experiment Settings: The study compares SCoRe with LLM-based zero-shot and few-shot baselines under official evaluation protocols.The reported VQA score is min(1, T(a)/3), and A-OKVQA uses direct-answer evaluation.
- A. Experiment Settings: Zero-shot demonstrations are generated from image captions, while few-shot exemplars are selected using average CLIP-based image-question similarity.The answer and other samples remain untouched in zero-shot generation, and top-n training samples are used for few-shot reasoning.
B. Performance of the Approaches
SCoRe consistently surpasses baselines across OK-VQA and A-OKVQA settings, while ablations support contributions from its modules and inference remains faster than DIETCOKE.
- B. Performance of the Approaches: 52.1% and 52.4% are SCoRe’s zero-shot results, surpassing DIETCOKE, the previous best method.The two values are reported for SCoRe with Mistral-7B across the evaluated benchmarks.
- B. Performance of the Approaches: Figure 4 reports Pseudo Recall for intermediate outputs.The intermediate outputs include visual and knowledge representations, selected context, relations, and compressed relations.
- B. Performance of the Approaches: 55.7% and 54.9% are SCoRe’s peak accuracies at 16-shot, establishing a new state-of-the-art.Performance gains increase steadily with the shot count in few-shot settings.
- B. Performance of the Approaches: 17.5 seconds per sample is SCoRe’s inference time versus 29.5 seconds for DIETCOKE.The paper describes these time costs as acceptable for visual question answering while achieving improved accuracy.
C. Ablation Studies
The ablation studies show that each SCoRe module improves performance, while structured compression resolves the tension between retrieval breadth and reasoning quality.
- Module effectiveness: 52.1% is achieved by the full model with visual notes, explicit knowledge passages, context selection, and context compression.Performance rises from 47.5% with visual notes alone to 49.0%, 51.2%, and finally 52.1% as modules are added.
- Parameter effects: PR is already strong at k1=1 and improves as the number of titles increases, but gains taper beyond k1=50.PR@20 is almost identical for k1=100 and k1=200.
- Parameter effects: Setting k1=200 and k2=10 balances retrieval efficiency and accuracy.The selected values follow the observed saturation in pseudo recall.
- Parameter effects: Retrieval candidate counts are compressed by tens to hundreds of times while retaining promising results.This reduction addresses the efficiency bottleneck of large-scale retrieval.
- Information loss: Although context acquisition has approximately 10 percentage points higher PR than final context compression, context compression achieves better actual performance.The acquisition output contains redundant and noisy information, whereas compressed structure exposes entity relations more clearly.
D. Case study
The case studies illustrate that SCoRe can produce accurate answers by organizing multimodal evidence into relation triplets, while also revealing failures from conflicting knowledge and insufficient visual specificity.
- Qualitative comparison: SCoRe produces accurate answers such as “snow” and “celebration” by integrating visual, explicit, and implicit knowledge through structured relation triplets.The comparison covers KGenVQA in zero-shot settings and EF-VQA in few-shot settings.
- Failure cases: A conflict between explicit and implicit relations causes SCoRe to output “1886” instead of the retrieved founding date “1892”.The model’s internal knowledge about a formula’s invention overrides the specific brand founding date in the retrieved context.
- Failure cases: Insufficient fine-grained visual knowledge leads SCoRe to answer “birds” instead of identifying specific species such as “cardinal”.The framework recognizes birds and basic colors but lacks specialized taxonomic cues such as subtle beak shapes or plumage patterns.
- Conclusion: SCoRe is presented as an LLM-based framework that extracts structured relationships for knowledge-based visual question answering.The framework is reported to outperform state-of-the-art methods in extensive experiments.