Source-linked AI summary
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, Chuang Gan
TL;DR
Knowledge-based visual reasoning requires models to understand images, connect external knowledge, and reason through answers, a challenging combination. IPVR uses interactive see-think-confirm prompting to select visual concepts, predict answers, and verify rationales. The paper reports better performance than prior few-shot baselines while retaining reasoning traces and computational efficiency relative to fine-tuning methods.
Problem
Knowledge-based visual reasoning is challenging because models must understand image content, connect external knowledge, and perform step-by-step reasoning to answer questions.
Source: §1 · p. 1Method
IPVR iteratively selects question-relevant visual concepts, uses their descriptions for LLM answer prediction, and verifies generated rationales against the image and answer consistency.
Source: §1 · p. 2, §3.2 · pp. 4–5Results
IPVR achieves better performance than previous few-shot baselines while maintaining a step-by-step reasoning trace and greater computational efficiency than fine-tuning methods.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that IPVR keeps the reasoning trace for each step while achieving better performance than existing prompting methods.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Large pre-trained vision and language models have demonstrated remarkable capacities for various tasks. However, solving the knowledge-based visual reasoning tasks remains challenging, which requires a model to comprehensively understand image content, connect the external world knowledge, and perform step-by-step reasoning to answer the questions correctly. To this end, we propose a novel framework named Interactive Prompting Visual Reasoner (IPVR) for few-shot knowledge-based visual reasoning. IPVR contains three stages, see, think and confirm. The see stage scans the image and grounds the visual concept candidates with a visual perception model. The think stage adopts a pre-trained large language model (LLM) to attend to the key concepts from candidates adaptively. It then transforms them into text context for prompting with a visual captioning model and adopts the LLM to generate the answer. The confirm stage further uses the LLM to generate the supporting rationale to the answer, verify the generated rationale with a cross-modality classifier and ensure that the rationale can infer the predicted output consistently. We conduct experiments on a range of knowledge-based visual reasoning datasets. We found our IPVR enjoys several benefits, 1). it achieves better performance than the previous few-shot learning baselines; 2). it enjoys the total transparency and trustworthiness of the whole reasoning process by providing rationales for each reasoning step; 3). it is computation-efficient compared with other fine-tuning baselines.
1. Introduction
Knowledge-based visual question answering requires models to understand image content, retrieve external knowledge, and reason step by step. IPVR combines question-relevant visual concept selection with answer-rationale verification, and the authors report stronger performance than prior few-shot baselines alongside transparency and computational efficiency.
1. Introduction
- KB-VQA requires image understanding, open-world knowledge, and logical reasoning, making it more challenging than traditional visual question answering.A person answering what room is pictured, for example, relates objects such as a sofa and coffee table to the concept “living room.”Source: §1 · p. 1
- Prior prompting methods can use question-independent captions and provide no step-by-step reasoning trace, while neural-symbolic systems are limited in open-world settings.The latter often target simple visual domains and require manually designed symbolic operations.Source: §1 · p. 2
- IPVR selects relevant visual concepts, turns them into textual descriptions for answer prediction, and verifies supporting rationales against the image and answer.It repeats think-confirm iterations until consecutive answers agree.Source: §1 · p. 2
- IPVR outperforms previous few-shot baselines, preserves a step-by-step reasoning trace, and is more computationally efficient than fine-tuning methods.The authors present these as benefits of their modular, interactive, iterative framework.Source: §1 · p. 2
2. Related Work
Prior work applies large pre-trained models to vision-language reasoning through fine-tuning, prompting, LLM-based knowledge retrieval, and neural-symbolic methods. IPVR is positioned as interactive prompting for knowledge-based visual reasoning that aims to retain reasoning transparency without relying on manually designed symbolic operations.
2. Related Work
- Fine-tuning large pre-trained vision-language models can be computationally expensive and time-consuming, while prompt-based methods use textual inputs to guide model predictions.Interactive prompting for knowledge-based visual reasoning entails repeated interaction between language and vision models, including concept extraction and prediction verification.Source: §2 · p. 2
- PICa uses image captions as LLM prompts, but its image context is independent of the question and its answer process lacks a reasoning trace.The paper contrasts this with selecting query-related objects and maintaining a step-by-step process.Source: §2 · p. 3
- LLMs have been used as knowledge sources for KB-VQA, whereas IPVR additionally provides a step-by-step reasoning process alongside the answer.Earlier methods often used neural networks to interpret images and retrieve knowledge from explicit knowledge bases.Source: §2 · p. 3
- Neural module networks offer transparent, stepwise reasoning but have focused on simple physical scenes or underperformed end-to-end methods on real-world datasets.These systems typically decompose questions into operations executed by network modules.Source: §2 · p. 3
3. Method
IPVR processes image-question pairs through see, think, and confirm modules in an iterative loop. It identifies candidate objects, selects and describes query-relevant concepts for answer prediction, then checks supporting rationales against both the image and answer consistency.
3.1. Overall
- IPVR iterates see, think, and confirm: it extracts image concepts, attends to query-relevant ones, predicts an answer, and verifies a supporting rationale before the next iteration.The process ends when consecutive answer predictions agree.Source: §3.1 · pp. 3–4
3.2. Model Details
- The see module detects candidate objects and produces a global image caption to provide visual context for later selection.The method uses a scene parser for object concepts and an image captioner for the global description.Source: §3.1 · p. 3, §3.2 · p. 4
- The think module uses an LLM to select detected categories related to the question, captions the selected region, and predicts an answer from the attended context.The selected category vocabulary is constrained to detected object names; regional captions can describe relationships between objects.Source: §3.2 · p. 4
- The confirm module asks the LLM to generate a rationale that is consistent with the answer and visual input.A rationale should support the answer when reused as context and should match the image.Source: §3.2 · p. 5
- A cross-modality classifier accepts high-similarity rationales into the next prompt, and the answer-rationale loop continues until successive answers agree or the iteration limit is reached.The algorithm returns the answer and the accumulated reasoning process.Source: §3.2 · pp. 4–5
4. Experiments
Experiments compare IPVR with few-shot and supervised baselines on knowledge-based visual reasoning benchmarks, then examine rationale quality, computational cost, failure cases, and component contributions.
4. Experiments
- Experiments use OK-VQA and A-OKVQA, eight in-context examples, and PICa and CoT as learning-in-context baselines.The setup uses OPT-66B; PICa prompts with image captions and object tags, while CoT generates a rationale before its answer.Source: §4 · pp. 5–6
4.1. Compared with Other Methods.
- IPVR gains over PICa and CoT on A-OKVQA and OK-VQA, and exceeds GPV-2 on the A-OKVQA test split; accuracy drops are 7.9 for GPV-2 and 0.4 for IPVR.The OK-VQA variant omits rationale reasoning because its training set has no rationales; IPVR also outperforms baselines in the simplified option-selection setting.Source: Table 1 · p. 6, §4.1 · p. 6
- IPVR's rationales are closer to A-OKVQA ground truth than CoT's, and its reasoning trace exposes visual concepts, regional descriptions, and supporting rationale.Rationale comparisons use BLEU and CLIP sentence similarity; qualitative examples contrast IPVR's visually matched rationale with an inconsistent CoT rationale.Source: Table 2 · p. 7, §4.1 · p. 7, Figure 4 · p. 7
- 13.62 mean LLM queries are used by IPVR; after cost alignment, it still significantly outperforms PICa and CoT.PICa and CoT use k = 14 queries per sample in the aligned comparison.Source: Table 3 · p. 7, §4.1 · p. 8
- IPVR fails when the scene parser misses a key concept such as “pacifier” or when the LLM cannot answer correctly despite adequate visual context.These cases locate failures in either concept detection or answer prediction.Source: Figure 5 · p. 8, §4.1 · p. 8
- Removing any ablated component lowers performance; attend has the largest effect, while rationale generation and CLIP verification each contribute to performance.The authors attribute attend's impact to the visual context it supplies for answer inference; rationale generation also improves transparency.Source: §4.1 · p. 8, Figure 6 · p. 8
5. Conclusion
IPVR adaptively focuses on relevant visual concepts and provides consistent rationales supporting its answer predictions. The authors report better performance than existing prompting methods and retain a trace of each reasoning step.
5. Conclusion
- IPVR adaptively focuses on related image concepts, turns them into natural language, and provides consistent rationales supporting its answer predictions.Source: §5 · p. 8
- Compared with existing prompting methods, IPVR achieves better performance while maintaining transparency through a trace of each reasoning step.Source: §5 · p. 8
- The authors hope IPVR motivates research on interactive prompting between pre-trained models of different modalities for visual commonsense reasoning.Source: §5 · p. 9
A. Overview.
The supplementary material provides additional implementation details, quantitative performance analysis, and qualitative visualizations supporting the main paper.
A. Overview.
- The supplement adds implementation details beyond those in the main paper.Source: A. Overview · p. 12
- It provides additional quantitative performance analysis.Source: A. Overview · p. 12
- It also provides additional qualitative visualizations.Source: A. Overview · p. 12
B. More Implementation Details.
Supplementary implementation details specify IPVR's stopping rule, rationale filtering, and regional-caption generation, with further comparisons reported for caption quality and multiple-choice performance.
B. More Implementation Details.
- IPVR stops when consecutive answers match or after five iterations; it reaches a consistent answer in 2.27 iterations on average.Source: B. More Implementation Details · p. 12
- Rationales are added to the prompt when their CLIP cosine similarity with the image exceeds a threshold set to 0.Source: B. More Implementation Details · p. 12
- Guided regional caption decoding uses caption-question cosine similarity, and its captions are reported as higher quality than naive scene-graph captions on A-OKVQA validation.The caption comparison is reported in Table 4; the supplied text gives no metric values.Source: B. More Implementation Details · p. 12, Table 4 · p. 12
- In the A-OKVQA multiple-choice setting, IPVR with GPT-3 (175B) is reported to outperform fully supervised GPV-2.Source: Table 5 · p. 12
C.1. More Quantitative Analysis.
The supplementary analysis finds that IPVR still outperforms baselines in A-OKVQA multiple-choice, while results vary with the language model and evaluation setting. With GPT-3 175B, it reports 57.5 test accuracy and a smaller validation-to-test drop than two fully supervised methods.
C.1. More Quantitative Analysis.
- IPVR still outperforms baselines in A-OKVQA multiple-choice, though GPV-2 performs better with OPT-66B in this setting than IPVR does.This differs from the reported direct-answer comparison.Source: §C.1 · p. 12
- 57.5 accuracy on the A-OKVQA test split is reported for IPVR using GPT-3 175B, with better performance than fully supervised GPV-2.Source: §C.1 · pp. 12–13
- The validation-to-test accuracy drops are 1.2 for IPVR, 9.7 for KRISP, and 6.6 for GPV-2.The authors suggest frequent validation-set evaluation may contribute to overfitting in fully supervised methods.Source: §C.1 · p. 13
C.2. More Qualitative Analysis.
The qualitative examples show IPVR attending to task-relevant visual context and producing consistent rationales, while its failures expose limits in language-model knowledge and visual perception.
C.2. More Qualitative Analysis.
- In a qualitative example, IPVR attends to semantically related concepts such as “pizza” and “food” and generates the consistent rationale that the restaurant is a pizza restaurant.Source: §C.2 · p. 13
- Failures arise when the LLM cannot connect colorful balloons to the LGBTQ movement or vision models miss essential context such as a donut’s shape; stronger reasoning and perception abilities could improve performance.Figure 10 presents these failure cases and their intermediate results.Source: §C.2 · p. 13, Figure 10 · p. 15
- IPVR provides a modular, step-by-step reasoning trace with related visual concepts, regional descriptions, and a supporting rationale, improving transparency and interpretability.Figure 9 illustrates the reasoning flow.Source: §C.2 · p. 13, Figure 9 · p. 14