Source-linked AI summary
Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering
Zihao Zhu, Jing Yu, Yujing Wang, Yajing Sun, Yue Hu, Qi Wu
TL;DR
FVQA requires external knowledge and fine-grained selection of question-relevant evidence beyond visible image content. Mucko addresses this with iterative reasoning over visual, semantic, and factual graph layers, achieving reported state-of-the-art performance and interpretable evidence selection.
Problem
Existing FVQA methods have limited ability to adaptively select question-oriented, complementary evidence across visual, semantic, and factual modalities.
Method
Mucko represents images as multi-layer heterogeneous graphs and uses modality-aware graph convolutions for intra-modal selection and cross-modal reasoning.
Results
Mucko outperforms state-of-the-art approaches on FVQA and obtains interpretable results on the benchmark dataset.
Takeaways & Limitations
Attention weights and gate values indicate which modalities and entities contribute more to each answer.
Takeaways & Limitations
Mucko fails when multiple answers are reasonable for the same question.
Abstract
from arXiv · showhide
Fact-based Visual Question Answering (FVQA) requires external knowledge beyond visible content to answer questions about an image, which is challenging but indispensable to achieve general VQA. One limitation of existing FVQA solutions is that they jointly embed all kinds of information without fine-grained selection, which introduces unexpected noises for reasoning the final answer. How to capture the question-oriented and information-complementary evidence remains a key challenge to solve the problem. In this paper, we depict an image by a multi-modal heterogeneous graph, which contains multiple layers of information corresponding to the visual, semantic and factual features. On top of the multi-layer graph representations, we propose a modality-aware heterogeneous graph convolutional network to capture evidence from different layers that is most relevant to the given question. Specifically, the intra-modal graph convolution selects evidence from each modality and cross-modal graph convolution aggregates relevant information across different modalities. By stacking this process multiple times, our model performs iterative reasoning and predicts the optimal answer by analyzing all question-oriented evidence. We achieve a new state-of-the-art performance on the FVQA task and demonstrate the effectiveness and interpretability of our model with extensive experiments.
1 Introduction
FVQA extends visual question answering by requiring external knowledge beyond image content. Mucko addresses this need with a multi-layer heterogeneous graph and modality-aware reasoning that selects question-relevant evidence across visual, semantic, and factual information.
- Motivation: FVQA requires external knowledge because some questions cannot be answered from visible image content alone.Answering such questions may require localizing an object, recognizing its semantic identity, and connecting it to a relevant fact.
- Limitations of Existing Methods: Existing FVQA methods build fact graphs or concatenate image, question, and entity embeddings, limiting fine-grained selection of relevant multimodal evidence.The homogeneous representation provides visual information equally to graph nodes and limits adaptive evidence capture across modalities.
- Proposed Approach: Mucko represents each image as a heterogeneous graph with visual, semantic, and factual layers.The visual layer preserves object appearance and relationships, the semantic layer bridges visual and factual information, and the fact layer stores corresponding knowledge.
- Proposed Approach: Its modality-aware heterogeneous graph convolution selects question-oriented evidence within each layer and aggregates complementary evidence across layers.These procedures are called Intra-Modal Knowledge Selection and Cross-Modal Knowledge Reasoning.
- Contributions: Attention selects relevant evidence while graph convolution adaptively aggregates features, and attention weights and gate values support interpretation of modality and entity contributions.The paper reports significant improvement over state-of-the-art solutions and provides visualized case studies.
2 Related Work
Related work has increasingly modeled visual question answering with graphs and external facts. Mucko extends these directions by combining visual, semantic, and factual graph layers for fine-grained cross-modal evidence collection.
- Visual Question Answering: Traditional VQA models use global visual features, while attention mechanisms highlight question-relevant objects but may ignore informative relationships between objects.Graph-based representations emerged to model objects and their relationships more explicitly.
- Visual Question Answering: Recent graph-based VQA work enriches visual representations with language, but LSTM-based semantics lack fine-grained correlations with visual information.Mucko addresses this limitation by using multiple graph layers spanning visual, semantic, and factual perspectives.
- Fact-based Visual Question Answering: FVQA combines an image with a knowledge base of fact triplets and generally selects one entity from a fact graph as the answer.Existing methods include query-mapping and learning-based approaches.
3 Methodology
Mucko represents an image with visual, semantic, and factual graph layers, then performs question-guided intra-modal selection and cross-modal reasoning to identify the answer entity.
- Task and Overview: FVQA predicts an answer entity from a supporting fact while leveraging an image, question, and external knowledge base.Facts are represented as triplets < e1, r, e2 >, and the answer is selected as either e1 or e2.
- Multi-Modal Graph Construction: Mucko constructs three graph layers: a visual graph for object appearance and relationships, a semantic graph for high-level abstractions, and a fact graph for external knowledge.The visual graph uses detected objects and spatial relations; semantic information comes from dense captions; candidate facts form the fact graph.
- Intra-Modal Knowledge Selection: Question-guided node and edge attention selects relevant evidence independently within the visual, semantic, and fact graphs.Node attention evaluates relevance to the question, while edge attention evaluates relational importance under neighboring-node and question guidance.
- Intra-Modal Knowledge Selection: Intra-modal graph convolution updates each layer by aggregating attention-weighted neighborhood information, producing question-oriented node representations.The same convolutional operations are applied independently to the three graph layers, with modality-specific node and edge representations.
- Cross-Modal Knowledge Reasoning: Cross-modal convolutions transfer complementary visual and semantic information to fact entities, followed by fact-to-fact aggregation for global answer selection.The fused fact-entity representations are iteratively refined across multiple reasoning steps, and the entity with the largest predicted probability is selected.
4 Experiments
Experiments show that Mucko consistently outperforms prior methods on FVQA, while ablations and visualizations support the value of multi-modal evidence selection, cross-modal reasoning, and interpretability. Parameter analyses identify top-100 candidate facts, top-3 relation types, and two reasoning steps as the strongest tested settings.
- Comparison with State-of-the-Art Methods: Mucko improves top-1 accuracy by 3.71% and top-3 accuracy by 5.69% over the state-of-the-art model.It consistently outperforms all compared approaches on every reported metric.
- Ablation Study: Removing the visual or semantic graph reduces top-1 accuracy by 1.1% or 3.94%, respectively, relative to the full model.Removing both graphs causes a significant decrease, with visual information having the greater impact in the reported ablation comparison.
- Ablation Study: Replacing visual-to-fact or semantic-to-fact convolution with concatenation decreases performance, supporting cross-modal convolution for gathering complementary evidence.The decrease also occurs when both cross-modal convolutions are replaced simultaneously.
- Interpretability: Attention weights and gate values visualize Mucko’s selection of relevant visual, semantic, and factual evidence during reasoning.Factual knowledge often provides predominant clues, while semantic evidence becomes more important for questions involving complex relationships.
- Interpretability: Mucko can fail when multiple answers are reasonable for the same question, such as when both “wedding” and “party” may have cakes.The reported prediction of “party” is considered reasonable by human judgment.
- Parameter Analysis: The highest downstream accuracy is obtained with top-100 candidate facts, top-3 relation types, and two reasoning steps.These settings are used in the full model.
5 Conclusion
The paper proposes Mucko for visual question answering requiring external knowledge by combining multi-layer graph representations with iterative cross-modal reasoning. It reports improved state-of-the-art performance and interpretable results on the benchmark dataset.
- 5 Conclusion: Mucko represents images with visual, semantic, and factual graph layers and iteratively selects and gathers intra-modal and cross-modal evidence.The model uses modality-aware heterogeneous graph convolution for multi-layer cross-modal knowledge reasoning.
6 Supplementary Materials
Mucko is evaluated on Visual7W+KB and OK-VQA alongside memory-based, graph-based, VQA, knowledge-based, and ensemble baselines. It outperforms the compared methods, including ensembles and models designed for general or knowledge-based VQA.
- OK-VQA: OK-VQA contains 14,031 images and 14,055 questions spanning categories including science and technology, history, and sports.Its questions are manually generated and require supporting knowledge from open-domain resources rather than specific knowledge bases.
- Visual7W+KB: The Visual7W+KB comparison includes memory-based KDMN variants and the graph-based Out of the Box model.The baselines include versions without external knowledge, with attention-based knowledge incorporation, with dynamic memory, and with ensemble modeling.
- Visual7W+KB: 7.98% on top-1 accuracy and 13.52% on top-3 accuracy over state-of-the-art models on Visual7W+KB.The method is a single model and surpasses the existing ensembled model.
- OK-VQA: Mucko consistently outperforms all compared models on OK-VQA overall performance, including general VQA, knowledge-based, and ensemble models.The comparison includes BAN and MUTAN, ArticleNet-based models, and BAN/AN and MUTAN/AN oracle ensembles.
- OK-VQA: The OK-VQA results indicate that general VQA requires effective external-knowledge incorporation beyond a well-designed VQA model alone.Mucko also exceeds single and ensemble knowledge-based VQA models in the reported comparison.