Source-linked AI summary
CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answering
Yuyang Hong, Jiaqi Gu, Yujin Lou, Lubin Fan, Qi Yang, Ying Wang, Kun Ding, Yue Wu, Shiming Xiang, Jieping Ye
TL;DR
KB-VQA can suffer conflicts between static VLM knowledge and retrieved information, with visual ambiguity and redundant contexts complicating conflict identification. CC-VQA addresses this through vision-centric conflict reasoning and correlation-guided encoding and decoding, achieving state-of-the-art results with 3.3−6.4% absolute accuracy improvements over competing methods.
Problem
KB-VQA faces conflicts between static parametric VLM knowledge and externally retrieved knowledge, while retrieved contexts may contain visually conflicting or misleading information.
Method
CC-VQA is a training-free method combining visual-centric contextual conflict reasoning with correlation-guided encoding and decoding for conflict mitigation.
Results
3.3−6.4% absolute accuracy improvements over competing methods were achieved across E-VQA, InfoSeek, and OK-VQA, establishing state-of-the-art performance.
Takeaways & Limitations
Visual-semantic conflict analysis and correlation-aware context processing provide the paper’s supported approach to improving knowledge-conflict handling in KB-VQA.
Takeaways & Limitations
CC-VQA requires explicit externalization of model knowledge before visual-centric conflict reasoning.
Abstract
from arXiv · showhide
Knowledge-based visual question answering (KB-VQA) demonstrates significant potential for handling knowledge-intensive tasks. However, conflicts arise between static parametric knowledge in vision language models (VLMs) and dynamically retrieved information due to the static model knowledge from pre-training. The outputs either ignore retrieved contexts or exhibit inconsistent integration with parametric knowledge, posing substantial challenges for KB-VQA. Current knowledge conflict mitigation methods primarily adapted from language-based approaches, focusing on context-level conflicts through engineered prompting strategies or context-aware decoding mechanisms. However, these methods neglect the critical role of visual information in conflicts and suffer from redundant retrieved contexts, which impair accurate conflict identification and effective mitigation. To address these limitations, we propose \textbf{CC-VQA}: a novel training-free, conflict- and correlation-aware method for KB-VQA. Our method comprises two core components: (1) Vision-Centric Contextual Conflict Reasoning, which performs visual-semantic conflict analysis across internal and external knowledge contexts; and (2) Correlation-Guided Encoding and Decoding, featuring positional encoding compression for low-correlation statements and adaptive decoding using correlation-weighted conflict scoring. Extensive evaluations on E-VQA, InfoSeek, and OK-VQA benchmarks demonstrate that CC-VQA achieves state-of-the-art performance, yielding absolute accuracy improvements of 3.3\% to 6.4\% compared to existing methods. Code is available at https://github.com/cqu-student/CC-VQA.
1. Introduction
KB-VQA combines powerful but static VLM knowledge with dynamically retrieved information, creating conflicts that multimodal systems and existing mitigation methods insufficiently address. CC-VQA responds with vision-centric conflict reasoning and correlation-aware encoding and decoding.
- Motivation: Static VLM knowledge and dynamically retrieved information can conflict, causing models to ignore retrieved context or integrate contradictory information inconsistently.These conflicts become more difficult as knowledge bases expand and retrieved or predicted knowledge may be unreliable.
- Motivation: Multimodal retrieval intensifies conflict challenges through crossmodal retrieval limitations, complex visual understanding, and increased hallucination risks.
- CC-VQA: CC-VQA externalizes parametric knowledge and performs vision-centric conflict reasoning against retrieved knowledge to produce conflict-annotated contexts.The method uses visual semantic features to reduce ambiguity and reasoning uncertainty during conflict analysis.
- CC-VQA: Correlation-guided encoding compresses low-correlation statements, while adaptive decoding adjusts output distributions using correlation weights.The design targets sharper conflict detection and more accurate answer generation under contextual redundancy.
2. Related Work
Related work frames KB-VQA as a retrieval, reranking, and generation pipeline and defines knowledge conflict as disagreement between external retrievals and internal model knowledge. CC-VQA is situated among multimodal conflict-mitigation and training-free decoding approaches.
- KB-VQA: KB-VQA commonly follows retrieval, reranking, and generation stages, with methods such as Wiki-LLaVA adding external multimodal documents through hierarchical retrieval.
- Knowledge Conflict: Knowledge conflict denotes discrepancies between retrieved external knowledge and a model’s internal parametric knowledge.
- Conflict Mitigation: Prior conflict-mitigation methods include prompt engineering, auxiliary discriminators, self-reflection, and training-free contrastive decoding.
3. Problem and Observation
KB-VQA conditions a VLM on an image, question, and retrieved context, but conflicts between retrieved and parametric knowledge can reduce answer quality. Two observations motivate visual validation of textual claims and fine-grained correlation analysis of redundant contexts.
- Problem Setup: KB-VQA generates an answer from an image and question while conditioning a VLM on aggregated external knowledge context.The external knowledge base contains entity articles paired with associated images.
- Knowledge Conflict: 16.82% accuracy gain from multimodal retrieval augmentation was accompanied by errors in 10.53% of cases previously answered correctly by the VLM alone.The analysis attributes this degradation to conflicts between retrieval contexts and parametric knowledge.
- Observation 1: Visual semantic features can validate retrieved textual claims about image attributes and help resolve conflicts with parametric knowledge.Relevant attributes include spatial relationships, colors, and shapes in the query image.
- Observation 2: Retrieved contexts are highly redundant: each top-3 context averages 107 sentences, while fewer than 0.3% exceed similarity 0.8.Similarity scores have mean µ = 0.4 and σ = 0.15.
- Observation 2: The correct answers for 90% of questions occur within the top 25% of sentences ranked by image-question similarity.
4. Method
CC-VQA externalizes parametric knowledge, analyzes visual-semantic conflicts across contextual sources, and uses statement-level correlation to prioritize relevant information during generation. Its encoding compresses low-correlation content while adaptive decoding increases the influence of highly correlated sentences.
- Visual-Centric Contextual Conflict Reasoning: CC-VQA generates parametric contexts from the VLM and performs vision-centric conflict analysis across internal and retrieved knowledge.Visual reasoning links contextual conclusions to image features, then summarizes conflict-relevant visual semantics for answer generation.
- Visual-Centric Contextual Conflict Reasoning: Visual rationale extraction identifies image features correlated with each contextual knowledge conclusion, enabling visual-semantic cues for conflict resolution.The resulting visual reasoning output supports prioritization of relevant visual semantics during generation.
- Correlation-Guided Encoding and Decoding: Correlation-aware positional encoding compresses the relative positional space of low-relevance sentences while preserving full positional resolution for highly correlated content.The low-correlation set is defined as the bottom τ percentile, with b = ⌊τ × N⌋ and α = 0.5 scaling position increments for low-correlation tokens.
- Fine-Grained Correlation: Statement-level relevance analysis scores each sentence against the image-question pair after rewriting the question into a disambiguated, image-grounded form.The method associates every sentence across contexts with a relevance metric derived from its relation to the rewritten question and image.
- Correlation-Guided Encoding and Decoding: Correlation-enhanced adaptive decoding gives greater influence to highly correlated contextual sentences because they indicate higher knowledge conflict probability.Fine-grained contextual correlation targets influential sentences rather than adjusting all content at the section level.
5. Experiments
Experiments evaluate CC-VQA on E-VQA, InfoSeek, and OK-VQA using standard task metrics, component ablations, oracle analysis, and qualitative cases. The results show consistent gains from conflict reasoning and correlation-aware processing, including training-free state-of-the-art performance.
- Evaluation Setup: Experiments cover E-VQA, InfoSeek, and OK-VQA, using VQA accuracy for InfoSeek and BEM score for E-VQA.The implementation uses Qwen2.5-VL-7B with frozen EVA-CLIP-8B retrieval and is training-free.
- Main Results: 4.7% and 3.3% further improvements over standard retrieval augmentation are achieved on E-VQA and InfoSeek, respectively.The method also exceeds MMKB-RAG by 5.1% on InfoSeek.
- Main Results: 78.8% accuracy on OK-VQA establishes state-of-the-art performance without fine-tuning.This surpasses existing non-fine-tuned approaches and Wiki-PRF.
- Component Ablation: VCCR adds 1.9% accuracy over Vanilla RAG, while CAD adds 0.8% and CPE adds 0.9% in component ablations.These results support contributions from visual conflict identification, adaptive decoding, and attention to high-correlation sentences.
- Alpha Ablation: Accuracy decreases as α decreases, yet remains high at α = 0.1; α = 0.5 maximizes use of highly relevant sentences.The analysis attributes this behavior to redundant contextual information that can be compressed with limited accuracy loss.
- Case Studies: Qualitative cases show that retrieved knowledge can override correct parametric knowledge, while CC-VQA preserves accurate answers through visual-centric and correlation-guided generation.Figure 6 illustrates externalized parametric knowledge and retrieved facts being synthesized to focus attention on relevant image details.
6. Conclusion
CC-VQA addresses KB-VQA knowledge conflicts by combining visual-semantic conflict analysis with correlation-guided encoding and decoding. Evaluations across three benchmarks report state-of-the-art performance with 3.3 −6.4% absolute improvements over competing methods, while future work targets implicit multimodal conflict reasoning.
- Conclusion: CC-VQA performs visual-semantic conflict analysis across internal and external knowledge contexts.Its second component compresses low-correlation statements and adaptively decodes with correlation-weighted conflict scoring.
- Conclusion: The method achieves state-of-the-art performance on E-VQA, InfoSeek, and OK-VQA.The reported absolute improvement over competing methods is 3.3 −6.4%.
- Limitations and Future Work: The approach requires explicit externalization of model knowledge before visual-centric conflict reasoning.Future work will explore integrating multimodal reasoning to enhance response accuracy.
B.1. Ablation Study of Compression Ratio τ
The compression-ratio ablation studies how τ affects correlation-aware positional encoding on a 10K-sample InfoSeek subset. Compressing more low-correlation sentences improves accuracy, motivating τ = 75%.
- Compression Ratio τ: τ denotes the percentage of lowest-correlation sentences selected for compression in positional encoding.The study evaluates τ on a 10K-sample InfoSeek subset.
- Compression Ratio τ: Accuracy progressively increases as more low-correlation sentences are compressed, with τ = 75% selected for the method.This setting compresses the bottom 75% of sentences ranked by correlation score.
- Compression Ratio τ: Compressing low-correlation sentences helps the model focus on high-correlation sentences most likely to contain answers.The result follows the paper’s stated Observation 2.
B.2. Generalization of CC-VQA
CC-VQA generalizes across models and retrieval sizes, maintaining substantial gains with a stronger model and improving further when more retrieved contexts are used. It also outperforms a stronger thinking model while using fewer output tokens and lower latency.
- Generalization across Models and Retrieval Size: 3.1% accuracy improvement, from 47.7% to 50.8%, is achieved on a large-scale 100M-entry KB with Qwen3-VL-8B.The evaluation tests generalization across models using a stronger model than the primary setting.
- Generalization across Models and Retrieval Size: Top-5 retrieval provides an additional 1% improvement compared with the default top-3 retrieval setting.
- Comparison with Thinking Model: 50.8% accuracy is achieved while consuming 192 output tokens and 8.94 seconds, compared with 817 tokens and 11.79 seconds for Qwen3-VL-8B-Thinking.CC-VQA uses fewer output tokens and has lower latency than the stronger thinking model in this comparison.
B.4. Inference Time
CC-VQA reduces inference-time costs through token compression while remaining training-free and achieving latency comparable to Wiki-PRF. On InfoSeek, it reaches 45.1% accuracy with 76 GB of A800 GPU memory.
- Comparison with Wiki-PRF: 8.94s versus 8.77s latency is comparable to Wiki-PRF despite one additional forward pass, while CC-VQA remains training-free.CC-VQA uses six forward passes versus five for Wiki-PRF.
- Comparison with Wiki-PRF: 45.1% accuracy is achieved on InfoSeek with 76 GB of A800 GPU memory usage.
C.1. Illustration of VCCR
VCCR decomposes conflict analysis into fine-grained extraction and comparison subtasks, using visual features and retrieved information to support answer reasoning. Verification and qualitative cases report accurate assessments, fine-grained concept capture, and improved retrieval-augmented generation.
- Illustration of VCCR: VCCR decomposes model and retrieval-context analysis into subtasks such as extraction and comparison to construct visual conflict reasoning.The module externalizes parametric knowledge, analyzes retrieved data, and consolidates extracted features into discriminative attributes.
- Illustration of VCCR: VCCR achieves over 84% overall accuracy in verification, while its decomposed subtasks maintain high accuracy.
- Illustration of VCCR: Scores of 3 or higher are interpreted as accurate assessments because MLLM evaluation criteria are more stringent than human annotators’ criteria.The score distribution covers 10,000 evaluated samples.
- More Comparison Cases: Qualitative E-VQA cases show correct answers where both the base model and standard RAG fail, across animals, plants, and buildings.The cases include indirectly and directly image-related questions.
- More Comparison Cases: Infoseek examples indicate that correlation computation improves extraction of correct answers from retrieved information across diverse factual questions.Examples include a puzzle author, a rocket launch site, and a small-town time zone.
- Illustration of CC-VQA: The illustrated CC-VQA pipelines apply visual-centric conflict analysis, visual rationale extraction, correlation-aware encoding, and correlation-enhanced adaptive decoding.The cases cover location-related E-VQA and time-related InfoSeek questions, alongside butterfly and red-panda examples.