PaperlayerView exact revision on arXiv

Source-linked AI summary

KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA

Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, Marcus Rohrbach

arXiv:2012.11014v1cs.CVcs.CL

TL;DR

Knowledge-based VQA asks questions whose answers require outside information not present in the image or annotated for retrieval. KRISP combines implicit knowledge from multimodal BERT reasoning with explicit graph reasoning while preserving symbolic semantics. It outperforms prior work on OK-VQA and uses both knowledge types, including for rare answers.

  • Problem

    Open-domain knowledge-based VQA lacks annotated knowledge at training and test time, requiring models to answer questions using information beyond the image.

  • Method

    KRISP combines a BERT-pretrained multimodal transformer with a symbolic graph network over diverse knowledge sources while retaining symbolic representations through answer prediction.

  • Results

    KRISP outperforms prior work on OK-VQA and experiments show it uses both implicit and symbolic knowledge while generalizing to rare answers.

  • Takeaways & Limitations

    Preserving symbolic representations through prediction supports explicit graph reasoning and generalization to rare answers in open-domain knowledge VQA.

  • Takeaways & Limitations

    The main state-of-the-art comparison uses OK-VQA version 1.0, while other experiments use version 1.1 with changed answer stemming and vocabulary.

Abstract

from arXiv · show

One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image. In this work we study open-domain knowledge, the setting when the knowledge required to answer a question is not given/annotated, neither at training nor test time. We tap into two types of knowledge representations and reasoning. First, implicit knowledge which can be learned effectively from unsupervised language pre-training and supervised training data with transformer-based models. Second, explicit, symbolic knowledge encoded in knowledge bases. Our approach combines both - exploiting the powerful implicit reasoning of transformer models for answer prediction, and integrating symbolic representations from a knowledge graph, while never losing their explicit semantics to an implicit embedding. We combine diverse sources of knowledge to cover the wide variety of knowledge needed to solve knowledge-based questions. We show our approach, KRISP (Knowledge Reasoning with Implicit and Symbolic rePresentations), significantly outperforms state-of-the-art on OK-VQA, the largest available dataset for open-domain knowledge-based VQA. We show with extensive ablations that while our model successfully exploits implicit knowledge reasoning, the symbolic answer module which explicitly connects the knowledge graph to the answer vocabulary is critical to the performance of our method and generalizes to rare answers.

1. Introduction

Open-domain knowledge-based VQA requires information beyond the image and beyond knowledge reliably learned from image-question-answer training pairs. KRISP combines implicit knowledge from BERT-style multimodal reasoning with explicit symbolic knowledge graphs, preserving symbolic semantics and using diverse sources for OK-VQA.

  • Open-domain VQA questions require external knowledge that is neither present in the image nor given as annotated facts at training or test time.
  • Training image-question-answer triplets alone is not scalable and can introduce dataset biases, motivating external sources such as Wikipedia and knowledge graphs.
  • Implicit knowledge is encoded in neural-network parameters learned from annotated data or large-scale unsupervised language training, including BERT-based multimodal models.
  • Preserving symbolic knowledge from input through answer prediction supports rare entities and rare knowledge because reasoning logic is shared across symbols, without requiring ground-truth knowledge facts.
  • KRISP combines a multimodal BERT-pretrained transformer with a graph network and draws on four diverse knowledge sources to cover OK-VQA’s varied knowledge needs.
  • KRISP establishes a new state-of-the-art on OK-VQA and analyzes knowledge fusion, explicit graph reasoning, and answer prediction through ablations.

2. Related Work

Related VQA work spans multimodal fusion, explicit reasoning over image content, and knowledge-based methods. KRISP is positioned among approaches that either retain symbolic structure with annotations or embed it for easier integration while losing symbol semantics.

  • Multimodal Vision and Language Modeling: Multimodal vision-language models use fusion strategies including bilinear models, self-attention, and increasingly transformer-based image-text integration.
  • Multimodal Vision and Language Modeling: BERT-pretrained multimodal transformers import implicit language knowledge while learning additional knowledge from multimodal training.
  • Some VQA methods extract programs or image symbols for explicit reasoning, but focus on information shown in the image rather than external knowledge.
  • Knowledge in Computer Vision: Knowledge-based vision research uses attributes, web-mined knowledge, word embeddings, class hierarchies, and knowledge graphs for recognition, retrieval, and related tasks.
  • Symbolic Knowledge for VQA: Symbolic VQA methods either retain symbols until answering but require correct-fact annotations, or embed symbolic knowledge and lose symbol semantics while integrating easily with standard VQA.

3. The KRISP Model

KRISP combines a BERT-pretrained multimodal transformer with symbolic graph reasoning over image and question concepts. It constructs and filters a large multi-source graph, then uses typed, directed message passing and separate implicit and symbolic answer predictions.

  • KRISP is organized into implicit multimodal reasoning, symbolic knowledge reasoning, and integration of their answer predictions.
  • Implicit Knowledge Reasoning: The implicit branch processes question tokens and visual features with a BERT-pretrained transformer, then mean-pools transformer outputs into z_implicit.
  • Symbolic Knowledge: Visual concept predictions provide symbols for image entities, enabling the graph network to connect pictured concepts with question reasoning, such as identifying a sheep before inferring “ewe.”
  • Knowledge Graph Construction: The graph combines trivia, commonsense, scientific, and situational knowledge from sources including DBPedia, ConceptNet, VisualGenome, and hasPart-style relations.
  • Knowledge Graph Construction: Because raw sources contain millions of edges and much irrelevant information, KRISP filters entities and edges using question, answer, and visual concepts before graph processing.
  • Integration: The model produces separate implicit and symbolic answer vectors and selects the highest-scoring answer across both branches, with symbolic prediction reusing z_implicit for late fusion.
  • Graph Network: An RGCN passes messages using distinct transformations for edge types and directions, preserving semantic differences such as is_a versus has_a.

4. Results

KRISP’s results show that combining symbolic graph reasoning with implicit transformer knowledge improves OK-VQA performance, especially for rare answers. Ablations indicate that symbolic structure, bidirectional fusion, and BERT pre-training each contribute, while qualitative cases reveal both useful graph support and homonym-related errors.

  • Model Analysis and Ablations: A random knowledge graph causes a significant performance drop, showing that KRISP benefits from symbolic knowledge rather than graph computation alone.The random-graph variant preserves the nodes and architecture but randomizes their edges.
  • Model Analysis and Ablations: Removing BERT pre-training lowers KRISP performance by 6%, confirming that implicit knowledge is an important component of the combined model.The corresponding Multi-modal BERT decrease is 7%, while the remaining KRISP advantage suggests partial overlap but mostly non-overlapping knowledge.
  • Model Analysis and Ablations: Both MMBERT-to-graph features and late fusion matter: removing either lowers performance by about 1%, while removing both lowers it by over 7%.Without a connection, the graph cannot incorporate implicit knowledge; without backpropagation, it cannot correct errors from its input.
  • Model Analysis and Ablations: Directed edge types matter, as replacing RGCN with GCN lowers performance by about 2%; removing both question and image graph inputs also lowers performance by about 2%.Word2vec node inputs have the smallest effect, reducing performance by less than 1%.
  • Subpart Analysis: 31.47% MMBERT accuracy with graph-mediated backpropagation exceeds the 29.26% MMBERT baseline, while the joint model uses graph predictions 47% of the time.The oracle selecting the better submodel reaches 36.71%, indicating non-redundant MMBERT and graph predictions.
  • Long-Tail Analysis: KRISP predicts 1349 unique answers versus MMBERT’s 1247, and 780 versus 719 among correct predictions, indicating better long-tail generalization.The answer-frequency-rank analysis likewise finds that KRISP selects less common answers on average.
  • Qualitative Analysis: Qualitative examples show graph-supported answers such as “gas” for a pot’s heat source, but homonyms such as “navy” can confuse symbolic reasoning.Another example answers “concrete” instead of “brick,” likely because available knowledge supported that answer.
  • State-of-the-Art Comparison: KRISP reaches 38.35% accuracy on OK-VQA v1.0 versus 33.66% for the best prior state of the art, and 38.90% on v1.1.The authors report slightly better performance on v1.1, whose answer vocabulary handles stemming more coherently.

5. Conclusion

KRISP combines implicit transformer reasoning with explicit knowledge-graph reasoning for knowledge-based VQA. It outperforms prior work on OK-VQA and generalizes to rare answers.

  • KRISP incorporates explicit knowledge-graph reasoning with implicit reasoning in a multi-modal transformer.
  • KRISP outperforms prior works on OK-VQA, the largest available open-domain knowledge VQA dataset.
  • Extensive ablations show that preserving symbolic representations from input to prediction is important to the architecture's performance.
  • Experiments and examples indicate that KRISP uses both implicit and symbolic knowledge and generalizes to rare answers.

A. Methodology Additional Details

The additional methodology details describe how four knowledge sources are combined, filtered, and converted into a task-specific symbolic graph.

  • The graph combines DBPedia, VisualGenome, hasPart KB, and ConceptNet knowledge sources.
  • DBPedia preprocessing translates graph triplets into lowercase multi-word English expressions.
  • The combined graph removes stop-word concepts to avoid nonmeaningful edges.
  • Filtering retains entities from questions, answers, and visual concepts, then limits the graph to 25 common and useful edge types.
  • The final graph contains 361,999 edges, 7,643 nodes, and 25 edge types.

A.2. Image Symbols

KRISP derives image symbols using four pretrained classifiers and detectors, then uses filtered answer vocabularies for OK-VQA evaluation.

  • KRISP obtains image symbols from four models: ImageNet and Places365 classifiers, a Visual Genome Faster R-CNN, and an LVIS Mask R-CNN.
  • The image-symbol models include ResNet-152, ResNet-18, Faster R-CNN, and an EQL-loss-trained Mask R-CNN.
  • The answer vocabulary includes answers appearing at least 10 times in training annotations.
  • The vocabulary contains 2,253 answers for OKVQA v1.0 and 2,250 for v1.1.

A.4. Graph Network to Multi-modal BERT Baseline

This baseline summarizes graph-network hidden states into one vector and supplies it as an additional input to Multi-modal BERT.

  • The graph network computes node representations with graph convolutions using input symbols and word2vec features, without implicit input.
  • A dummy node and edge type aggregate hidden states from all graph nodes into a symbolic graph summary.
  • The graph summary is linearly embedded to the transformer input size and appended to Multi-modal BERT inputs.
  • Alternative self-attention-based graph summarization methods performed worse than the dummy-node aggregation method.

B. Network / Training Hyperparameters

This section records the network and training parameters used for KRISP and its MMBERT baseline and subparts.

  • Table 7 reports network parameters for the MMBERT baseline and subparts.
  • Table 9 reports the training meta-parameters used to train all models.

C. Variance Values for Tables

This section reports sample standard deviations for runs presented in the paper’s tables.

  • Sample standard deviations are provided for the runs reported in Tables 10 and 11.

D. Additional Ablations

The additional ablations examine knowledge-graph sources and image-symbol inputs, showing that combining sources and symbols performs best overall.

  • The additional ablations include final experiments summarized in Tables 12 and 13.
  • Tables 8 and 9 provide graph-network and training hyperparameters relevant to interpreting the ablation experiments.
  • Knowledge Graph Ablation: Using all four knowledge-graph sources outperforms using DBPedia, VisualGenome, hasPart, or ConceptNet alone; DBPedia and ConceptNet are the strongest individual graphs.
  • Image Symbol Ablation: Using any single image-classifier symbol set performs worse than using all four, while VisualGenome detections are the most significant inputs.

E. More Qualitative Examples

The paper provides additional qualitative examples comparing KRISP with the implicit-knowledge Multi-modal BERT baseline and showing related graph knowledge.

  • Figure 5 presents additional qualitative examples from KRISP.
  • Each example shows the question, image, and answers predicted by KRISP and the Multi-modal BERT implicit-knowledge baseline.
  • The figure also displays graph knowledge judged relevant to the question, answers, or image.
Loading 2012.11014v1…