PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering

Jing Yu, Zihao Zhu, Yujing Wang, Weifeng Zhang, Yue Hu, Jianlong Tan

arXiv:2009.00145v1cs.AIcs.CV

TL;DR

Knowledge-based VQA requires evidence beyond image content, but existing approaches can combine information without fine-grained selection. This paper represents visual, semantic, and factual information as graphs and applies recurrent cross-modal reasoning. It reports state-of-the-art performance on three benchmarks, while noting that its global-assessment GNN is relatively simple.

  • Problem

    Knowledge-based VQA requires external knowledge beyond visible image content, and existing solutions may jointly embed information without fine-grained selection of question-oriented, complementary evidence.

    Source: §1 · pp. 2–3

  • Method

    The model represents visual, semantic, and factual information as graphs and uses recurrent GRUC modules for parallel visual and semantic reasoning before graph-based global assessment.

    Source: §1 · p. 5

  • Results

    The model reportedly achieves state-of-the-art performance on FVQA, Visual7W-KB, and OK-VQA, with ablations examining component contributions.

    Source: §1 · p. 6, §5 · p. 30

  • Takeaways & Limitations

    The model's attention weights and fusion gate values expose which concepts and modalities contribute to answers and how evidence is selected across modalities.

    Source: §4.5 · p. 26

    The authors describe the global-assessment GNN as relatively simple and suggest that a more effective global assessment approach may improve performance.

    Source: §4.4.4 · p. 26

Abstract

from arXiv · show

Knowledge-based Visual Question Answering (KVQA) requires external knowledge beyond the visible content to answer questions about an image. This ability is challenging but indispensable to achieve general VQA. One limitation of existing KVQA solutions is that they jointly embed all kinds of information without fine-grained selection, which introduces unexpected noises for reasoning the correct answer. How to capture the question-oriented and information-complementary evidence remains a key challenge to solve the problem. Inspired by the human cognition theory, in this paper, we depict an image by multiple knowledge graphs from the visual, semantic and factual views. Thereinto, the visual graph and semantic graph are regarded as image-conditioned instantiation of the factual graph. On top of these new representations, we re-formulate Knowledge-based Visual Question Answering as a recurrent reasoning process for obtaining complementary evidence from multimodal information. To this end, we decompose the model into a series of memory-based reasoning steps, each performed by a G raph-based R ead, U pdate, and C ontrol ( GRUC ) module that conducts parallel reasoning over both visual and semantic information. By stacking the modules multiple times, our model performs transitive reasoning and obtains question-oriented concept representations under the constrain of different modalities. Finally, we perform graph neural networks to infer the global-optimal answer by jointly considering all the concepts. We achieve a new state-of-the-art performance on three popular benchmark datasets, including FVQA, Visual7W-KB and OK-VQA, and demonstrate the effectiveness and interpretability of our model with extensive experiments.

1. Introduction

KVQA needs question-oriented evidence connecting image content to semantic concepts and external facts, but existing approaches can use information indiscriminately. The paper represents these modalities as graphs and reasons across them recurrently to select complementary evidence before assessing answers.

1. Introduction

  • Earlier methods can miss visual concepts when matching question keywords, while indiscriminate image information can add noise to entity representations.The paper contrasts keyword matching and fixed image-question-entity embeddings in with selective evidence gathering.Source: §1 · pp. 2–3
  • The paper represents visual appearance, semantic abstraction, and factual knowledge with distinct graphs, with the visual and semantic graphs grounded in the fact graph.The semantic graph supplies high-level abstraction, which the paper says brings remarkable improvement in KVQA.Source: §1 · pp. 4–5
  • GRUC performs parallel visual and semantic reasoning through control, read, and update units, iteratively integrating knowledge under cross-modal constraints.After multiple steps, the model adaptively fuses evidence and uses a graph neural network to assess concepts for an answer.Source: §1 · p. 5
  • The model reportedly outperforms state-of-the-art approaches on FVQA, Visual7W-KB, and OK-VQA, with ablations examining component contributions.The introduction also presents the model as interpretable through attention weights and fusion gate values.Source: §1 · p. 6

2. Related Work

Prior VQA work develops attention and relational reasoning, while knowledge-based approaches add external facts but can provide visual information indiscriminately. This paper situates its approach among graph methods by reasoning across modality-specific subgraphs with modality-oriented controllers.

2.1. Visual Question Answering

  • Visual VQA research progressed from attention to multi-step relational reasoning, but these approaches do not incorporate external knowledge during progressive reasoning.The discussion notes that one-step relation networks have high computational cost and that multi-step approaches address compositional reasoning.Source: §2.1 · pp. 6–7

2.2. Incorporating External Knowledge in VQA

  • Knowledge-based VQA methods include query mapping and graph reasoning, but prior approaches may omit visual evidence during reasoning or provide it wholesale.The paper contrasts these methods with its multimodal-graph and memory-based cross-modal reasoning approach.Source: §2.2 · pp. 7–8

2.3. Graph Neural Networks

  • Unlike prior heterogeneous graph methods that unify node and edge types, this work uses multiple modality-specific subgraphs and modality-oriented controllers for parallel reasoning.The paper also distinguishes GRUC from GGS-NN, which performs convolution and recurrent propagation in the same graph.Source: §2.3 · pp. 8–9

3. Methodology

The method represents image-question evidence and external facts in modality-specific graphs, selects question-oriented knowledge, and recurrently gathers complementary evidence. It then jointly assesses concepts to predict an answer.

3. Methodology

  • The model constructs visual, semantic, and fact graphs, selects question-oriented knowledge within each graph, and uses GRUC for iterative cross-modal reasoning.The reasoning gathers complementary evidence from visual and semantic graphs under question and fact guidance.Source: §3 · pp. 9–10
  • At the end of reasoning, evidence from three modalities is fused into a representation for each concept.This follows the iterative gathering of complementary visual and semantic evidence.Source: §3 · p. 10
  • Graph convolutional networks jointly analyze concepts, and a binary classifier predicts the optimal answer.This is the model's global assessment and answer-prediction stage.Source: §3 · p. 10

3.1. Multimodal Graph Construction

The method builds separate visual, semantic, and fact graphs to preserve modality-specific evidence and relations. These graphs represent detected objects and spatial relations, caption-derived concepts and relations, and retrieved factual entities and relations.

3.1.1. Visual Graph Construction

  • The visual graph is fully connected over detected objects, whose nodes carry visual features and whose edges encode relative spatial relationships.The described detector yields K = 36 objects; bounding-box features provide object geometry.Source: §3.1.1 · p. 10

3.1.2. Semantic Graph Construction

  • Dense captions provide local semantic descriptions, which are parsed into a graph of object names or attributes and their relationships.The graph representation retains relations among concepts for explicit cross-modal reasoning.Source: §3.1.2 · pp. 10–11

3.1.3. Fact Graph Construction

  • Candidate facts are ranked by word-embedding similarity to question words and detected visual concepts; the top 100 are further filtered by predicted relation type.The relation classifier retains facts matching the top three predicted relationships, and the resulting fact graph organizes entities and relations.Source: §3.1.3 · p. 11

3.2. Intra-Modal Knowledge Selection

The model selects question-relevant evidence separately within the visual, semantic, and fact graphs, then updates node representations through intra-modal graph convolution.

3.2. Intra-Modal Knowledge Selection

  • Question-guided attention scores graph nodes for relevance to the question before their representations are updated.The attention uses the node representation and question embedding; its parameters are learned, and the question embedding is encoded by an LSTM.Source: §3.2 · p. 12
  • A second question-guided attention step evaluates each edge in relation to its neighboring node and the question.The edge-attention inputs combine neighbor and relation representations with the current node and question representations.Source: §3.2 · p. 12
  • Intra-modal graph convolution uses the learned node and edge attention weights to aggregate neighborhood information and update node representations independently in each graph.The visual-to-visual, semantic-to-semantic, and fact-to-fact convolutions share operations but use modality-specific node and edge representations.Source: §3.2 · pp. 11–12

3.3. Cross-Modal Knowledge Reasoning

Cross-modal reasoning supplements fact-graph entities with question-relevant visual and semantic evidence. Recurrent GRUC steps gather this evidence in parallel, then fuse the modalities into concept representations.

3.3. Cross-Modal Knowledge Reasoning

  • Fact-graph entities provide the candidate answer concepts, while visual and semantic knowledge complements their otherwise insufficient evidence.The model treats each fact-graph entity as a concept and gathers complementary knowledge from the other two graphs without requiring explicit one-to-one mappings.Source: §3.3 · p. 13, §3.3.1 · p. 14

3.3.1. The Recurrent GRUC Module

  • GRUC modules operate in parallel over visual and semantic graph memories for each fact-graph concept, iteratively gathering question-relevant knowledge.Each module has Control, Read, and Update units; the control state guides memory reads, while the read context updates the control state for the next step.Source: §3.3.1 · pp. 14–15
  • The Read Unit attends over graph-structured memory to retrieve contextual knowledge that both guides the next reasoning step and contributes to final modality representations.The Update Unit adaptively revises memory entries using their previous states, neighboring entries, and the current control state.Source: §3.3.1 · pp. 15–16

3.3.2. Multimodal Feature Fusion Module

  • After T reasoning steps, a gate fuses fact, visual, and semantic knowledge into each concept representation.The resulting concept representations are used in global assessment.Source: §3.3.2 · pp. 16–17

3.4. Global Assessment and Answer Prediction

The model compares concept representations globally with a graph neural network and predicts each concept’s answer probability using a binary classifier.

3.4. Global Assessment and Answer Prediction

  • Concept embeddings are compared globally by a graph neural network, whose outputs feed a binary classifier for answer prediction.The entity associated with the concept having the largest probability is selected as the final answer.Source: §3.4 · p. 17
  • Weighted binary cross-entropy addresses training-sample imbalance because each sample has one annotated answer entity and the remaining entities serve as negatives.The loss assigns separate weights to positive and negative samples.Source: §3.4 · p. 17

4. Experiments

The experiments compare GRUC on FVQA, Visual7W-KB, and OK-VQA, using top-1 and top-3 accuracy. GRUC outperforms the reported baselines across the benchmarks, with less pronounced improvement on OK-VQA than on the other two datasets.

4.1. Datasets and Evaluation Metrics

  • The evaluation uses FVQA, Visual7W-KB, and OK-VQA, with top-1 and top-3 accuracy averaged over five test splits.FVQA asks for an entity in a supporting fact; Visual7W-KB has 16,850 open-domain question-answer pairs, while OK-VQA has 14,055 questions and uses ConceptNet for supporting knowledge retrieval.Source: §4.1 · pp. 17–18

4.2. Implementation Details

  • Implementation uses 300-dimensional GloVe word embeddings and a 512-dimensional LSTM question encoder, with the maximum question length set to 20 words.The model selects the top 12 dense captions, uses 36 detected objects, and is trained for 10 epochs with Adam and dropout ratio 0.5.Source: §4.2 · pp. 18–19

4.3.1. Experimental results on FVQA

  • 10.28% on top-1 accuracy and 10.95% on top-3 accuracy were the boosts over state-of-the-art model OB on FVQA.The authors report that GRUC consistently outperforms all approaches on all metrics.Source: §4.3.1 · pp. 19–20, Table 1 · p. 20

4.3.2. Experimental results on Visual7W-KB

  • 8.13% on top-1 accuracy and 16.51% on top-3 accuracy were the improvements over state-of-the-art models on Visual7W-KB.GRUC is a single model and outperforms the existing ensembled model [21].Source: Table 2 · p. 21, §4.3.2 · p. 21

4.3.3. Experimental results on OK-VQA

  • GRUC consistently outperforms compared models overall and in each OK-VQA knowledge category, including standard VQA, knowledge-based, and ensemble models.The reported categories include Vehicles and Transportation, Sports and Recreation, and Science and Technology; the full category list is in the cited table and prose.Source: §4.3.3 · pp. 21–22, Table 3 · p. 22
  • OK-VQA improvement is less pronounced than on FVQA and Visual7W-KB, which the authors attribute partly to free-form questions that are more diverse and complex.They also note that OK-VQA requires a wider range of knowledge than a specific knowledge base provides.Source: §4.3.3 · pp. 22–23

4.4. Ablation Study

The ablations test the value of each knowledge modality, GRUC reasoning, intra-modal selection, and global assessment. Removing these components generally reduces performance, with the reported effects varying by component.

4.4.1. Influence of Different Knowledge Modalities

  • Removing either the semantic or visual graph lowers top-1 and top-3 accuracy on all three datasets; removing both causes a significant decrease.The authors report a greater impact from removing visual information and say factual knowledge alone is insufficient.Source: §4.4.1 · p. 24

4.4.2. Influence of the GRUC Module

  • Replacing GRUC with simple concatenation decreases performance by more than 10%, while removing neighborhood information causes smaller decreases across the three datasets.The authors link preserving neighborhood structure with richer semantics for answer prediction.Source: §4.4.2 · p. 25

4.4.3. Influence of the Intra-Modal Knowledge Selection

  • Removing intra-modal knowledge selection remarkably lowers all metrics on all three datasets.The authors interpret the result as evidence that filtering knowledge before reasoning adds improvement.Source: §4.4.3 · p. 25

4.4.4. Influence of the Global Assessment

  • Removing global assessment lowers performance on every dataset, but less than the other ablations; the authors suggest a more effective GNN could improve it.The authors describe the GNN used for global assessment as relatively simple.Source: §4.4.4 · p. 26

4.5. Interpretability

The visualizations expose how GRUC selects and combines evidence across modalities, and illustrate both a comparison with OB and several kinds of failure. The authors report that the dominant evidence source varies with the question.

4.5. Interpretability

  • Attention weights and gate values visualize GRUC’s modality-specific evidence selection and cross-modal read attention.Figure 6 marks attended objects, relations, graph neighbors, read attention at step T, and gate values.Source: §4.5 · p. 26, Figure 6 · p. 27
  • Factual knowledge supplies predominant clues in most cases, while semantic evidence increases for complex relationships and visual evidence for a single-object question.The authors illustrate the contrast with questions about a single object and a relationship between objects.Source: §4.5 · p. 26
  • In the illustrated laptop-versus-keyboard case, GRUC distinguishes the entities using relevant visual and semantic evidence, while OB does not.The authors attribute OB’s difficulty to its lack of feature selection.Source: §4.5 · p. 28
  • Failures include ambiguous answers or insufficient image evidence, as well as missing required facts in the provided knowledge base.The authors give wedding versus party as an ambiguity and report no ConceptNet fact about Samuel Fox.Source: §4.5 · p. 28

4.6. Parameter Analysis

Parameter analysis reports the best performance with 10 dense captions and three GRUC reasoning steps. The authors relate the reasoning-step result to insufficient evidence at lower counts and over-smoothing at higher counts.

4.6. Parameter Analysis

  • Evaluations of different numbers of dense captions show that 10 captions achieve the best performance on both top-1 and top-3 accuracy.Source: §4.6 · pp. 28–29
  • 3 reasoning steps achieve the best performance on all three datasets; fewer steps may leave knowledge inadequately extracted, while too many may cause over-smoothing.Figure 8 reports top-1 and top-3 accuracy for FVQA, Visual7W+KB, and OK-VQA across one to five steps.Source: Figure 8 · p. 29, §4.6 · p. 29

5. Conclusion

The paper presents GRUC as a graph-based recurrent model for cross-modal reasoning over visual, semantic, and factual knowledge graphs. It reports state-of-the-art performance on three benchmarks, while identifying open-domain knowledge as a weakness.

5. Conclusion

  • The model represents visual, semantic, and factual knowledge as graphs, with the semantic graph providing high-level abstraction.The conclusion says the graph representations unify modalities and support relational reasoning across them.Source: §5 · pp. 29–30
  • Each GRUC step uses modality-oriented controllers to reason in parallel over visual and semantic information, and the model consistently outperforms state-of-the-art approaches on FVQA, Visual7W-KB, and OK-VQA.The authors also report that visualizations reveal knowledge selection across modalities.Source: §5 · pp. 29–30
  • The model performs less well when open-domain knowledge is required; the authors identify adaptively incorporating diverse knowledge bases as future work.The proposed coverage includes commonsense, Wikipedia, and professional knowledge.Source: §5 · p. 30

Related papers

Loading 2009.00145v1…