Source-linked AI summary
KAT: A Knowledge Augmented Transformer for Vision-and-Language
Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao
TL;DR
Knowledge-based VQA raises whether multimodal transformers can use explicit knowledge alongside their implicit knowledge, while existing methods leave retrieval quality and integration unresolved. KAT combines both sources in an end-to-end encoder-decoder that reasons over them during answer generation, achieving a +6 points absolute state-of-the-art result on OK-VQA. The paper also reports improved interpretability and identifies remaining challenges in semantic alignment and multi-source integration.
Problem
Existing knowledge-retrieval approaches leave open the quality and relevance of retrieved knowledge and how implicit and explicit knowledge should be integrated for reasoning.
Method
KAT jointly integrates implicit commonsense knowledge and explicit knowledge in an end-to-end encoder-decoder that generates answers autoregressively.
Results
+6 points absolute on OK-VQA, with KAT achieving a strong state-of-the-art result.
Takeaways & Limitations
Explicit and implicit knowledge provide complementary pieces of knowledge, and their integration improves performance and interpretability of model predictions.
Takeaways & Limitations
Semantic matching between image regions and knowledge entries lacks implicit commonsense knowledge.
Abstract
from arXiv · showhide
The primary focus of recent work with largescale transformers has been on optimizing the amount of information packed into the model's parameters. In this work, we ask a different question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these challenges, we propose a novel model - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result (+6 points absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an end to end encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. An additional benefit of explicit knowledge integration is seen in improved interpretability of model predictions in our analysis.
1 Introduction
Knowledge-intensive visual question answering requires models to connect image content with external knowledge and reason over both implicit commonsense and explicit facts. KAT addresses retrieval quality and joint reasoning as central challenges in OK-VQA.
- OK-VQA requires knowledge not present in the input, making it a test bed for integrating implicit and explicit knowledge.Implicit knowledge includes commonsense, while explicit knowledge can come from sources such as Wikidata.
- Answering OK-VQA questions can require visual grounding, explicit knowledge, and implicit reasoning in a single chain.Examples include grounding “organism” to “bird” before applying that birds evolved from reptiles, or recognizing boats and harbors before using knowledge about anchors.
- Existing approaches may retrieve generic or irrelevant explicit knowledge and often focus on explicit sources that cannot answer many knowledge-based questions alone.The paper argues that questions often require jointly reasoning over explicit and implicit knowledge.
- KAT improves knowledge extraction by prompting a frozen GPT-3 model for tentative answers and evidence, and by using a CLIP-based contrastive retriever for explicit knowledge.The retrieved explicit knowledge is designed to be more relevant to the image and question.
2 Related Work
Prior work brings external knowledge into knowledge-based VQA through structured and unstructured resources, APIs, and multimodal models used as implicit knowledge bases. KAT instead jointly reasons over explicit and implicit knowledge in a unified generative architecture, using Wikidata entities rather than a dataset-specific corpus.
- Vision-Language Transformer: Multimodal transformers provide implicit cross-modal knowledge, but that knowledge is not sufficient for many knowledge-based questions.Other multimodal models, including CLIP and ALIGN, primarily align visual and language representations through contrastive learning.
- Knowledge-based VQA: Knowledge-based VQA extends visual question answering with external knowledge beyond the image, and OK-VQA is an open-ended benchmark covering diverse knowledge categories.KAT focuses on OK-VQA because of its large-scale knowledge-based questions and open-ended nature.
- Knowledge-based VQA: Prior methods aggregate structured knowledge graphs, unstructured knowledge bases, or neural-symbolic inference, often linking images to knowledge through detectors and scene classifiers.External APIs and OCR have also been used to enrich these associations.
- KAT: KAT jointly reasons over explicit and implicit knowledge in a single unified architecture to augment generative language models.The approach extends retrieval-and-generation frameworks from open-domain question answering to visual question answering.
- KAT: Unlike dataset-specific approaches such as Vis-DPR, KAT collects entities from Wikidata and is not limited to the training set.Its distinction from PICa is the integration of explicit and implicit knowledge during reasoning.
3 Method
KAT integrates explicit knowledge retrieved from image regions with implicit commonsense knowledge retrieved from GPT-3, then jointly reasons over both sources during autoregressive answer generation. Its end-to-end encoder-decoder design combines retrieval, reasoning, and generation for OK-VQA.
- Overview: KAT defines explicit knowledge as information from external knowledge bases and implicit knowledge as knowledge stored in large-scale language models.
- Problem Formulation: KAT formulates OK-VQA as autoregressive sequence-to-sequence answer generation from an image and question rather than classification.The model uses an encoder and decoder and generates answers from the entire vocabulary.
- Explicit Knowledge Extraction: A contrastive-learning-based retriever matches image regions with Wikidata entries to select the top-m relevant entries as explicit knowledge.Image patches are encoded and compared with indexed entity representations using normalized inner products.
- Knowledge Base Construction: The explicit knowledge base is a Wikidata subset covering eight categories of animals, vehicles, and other common objects.The construction filters the English Wikidata dump to common real-world objects and produces 423,520 entity triplets.
- Knowledge Reasoning Module: KAT concatenates explicit and implicit knowledge embeddings into a global representation and uses cross-attention before decoder layers generate the answer.The attended representation is a weighted sum of knowledge values, enabling joint reasoning over both knowledge sources.
4 Experiment
The experiments fine-tune T5-based models on OK-VQA and compare KAT with existing approaches. KAT achieves higher performance under both explicit-only and combined implicit-explicit knowledge settings.
- Dataset and setup: OK-VQA contains 14,031 images and 14,055 questions requiring outside knowledge beyond the images.The dataset covers a variety of knowledge categories and uses the standard VQA challenge evaluation metric.
- Dataset and setup: KAT fine-tunes T5-base and T5-large models on OK-VQA using AdamW and retrieves 40 entities.The models contain 220M and 770M parameters, respectively; training uses a 3e−5 learning rate with a 2K-iteration warmup and 10K total iterations.
- Comparison with Existing Approaches: KAT outperforms existing approaches on OK-VQA, achieving 44.25% with explicit knowledge alone.This score is 4.85% and 5.9% higher than MAVEx and KRISP, respectively.
- Comparison with Existing Approaches: With implicit knowledge from GPT-3 as additional input, KAT outperforms PICa-Full by 6.41%.The comparison evaluates KAT's integration of knowledge from different types during answer generation.
5 Ablation Study
The ablations show that model capacity, explicit-knowledge quantity, retrieval backbone, and adaptive knowledge reasoning all affect performance, while jointly using implicit and explicit knowledge is consistently beneficial. Category analyses further show complementary strengths across knowledge types and large gains from their joint use.
- Model architectures and knowledge types: Larger KAT models consistently outperform KAT-base, indicating greater capacity for implicit knowledge reasoning.KAT-large shows a consistent improvement over KAT-base.
- Model architectures and knowledge types: ∼4% performance gain from integrating explicit and implicit knowledge supports their complementary roles.The two knowledge sources provide complementary pieces of knowledge.
- Effectiveness of knowledge reasoning: 2.43% lower performance for KAT without reasoning shows that concatenating knowledge sources is less effective than adaptive attention during answer generation.The non-reasoning variant may introduce noise, whereas KAT adaptively attends to different knowledge sources.
- Extracting explicit knowledge: 47.6% accuracy using only implicit knowledge is slightly below PICa-Full, while adding more explicit knowledge entries consistently improves KAT.The results indicate that model complexity alone is insufficient and that explicit knowledge contributes to performance.
- Extracting explicit knowledge: The retrieval backbone and number of knowledge sources are interrelated, and adaptive attention can further reduce the gap between CLIP-ViT/16 and RN50.As more entities are retrieved, the knowledge reasoning module can adaptively attend to different explicit knowledge.
- Category and qualitative results: Joint reasoning outperforms explicit-only and implicit-only models by large margins across OK-VQA categories, with explicit knowledge stronger for Brands, Companies, and Products and Weather and Climate.Implicit-only performance is better for most categories, while explicit knowledge helps when fine-grained visual recognition is important.
6 Conclusion
KAT highlights the complementary role of implicit knowledge from scaled models and explicit knowledge from structured knowledge bases. The paper reports significant improvement while identifying retrieval, reasoning, image-region alignment, and multi-base integration as remaining challenges.
- Conclusion: KAT makes the complementary relationship between implicit and explicit knowledge explicit while achieving significant improvement on knowledge-based VQA.The approach is described as conceptually simple yet effective.
- Conclusion: Further progress remains possible through improved retrieval and reasoning, better image-region alignment with external semantics, and efficient integration of multiple knowledge bases.These challenges are identified as directions for future work.
A Figure of Explicit Knowledge
Explicit knowledge is extracted by cropping image regions with a sliding window and retrieving entries from a wiki knowledge base using CLIP.
- Explicit knowledge extraction: A sliding window crops image regions before CLIP retrieves explicit knowledge entries from the knowledge base.The described extraction example uses CLIP with a wiki knowledge base.
B Examples of Prompts of Implicit Knowledge
The appendix provides examples of prompts used to obtain implicit knowledge from GPT-3.
- Implicit knowledge prompts: GPT-3 is queried for implicit knowledge using prompts constructed from experimental inputs.The appendix presents concrete prompt examples and rationale evidence.
C Analysis on More Examples
The examples illustrate how KAT combines fine-grained explicit knowledge and commonsense implicit knowledge before generating open-ended answers. This combination supports semantically reasonable predictions and stronger knowledge-aware representations.
- Analysis of predictions: The model’s examples analyze predictions across different knowledge types and aspects.
- Effectiveness of explicit knowledge retriever: Fine-grained explicit knowledge entries, such as golden retriever and Macbook Pro, constrain the model’s search space and benefit answer generation.
- Effectiveness of implicit knowledge retriever: Implicit knowledge retrieved from GPT-3 supplies commonsense evidence that requires further inference and complements explicit knowledge.
- Answer generation & classification: KAT treats OK-VQA as an open-end generation task rather than classification, enabling answers such as Iphone or Hercules.Classification approaches use dataset-specific vocabularies and therefore have limited generalization ability.
- Answer generation & classification: Using explicit and implicit knowledge enhances semantic alignment and produces representations with stronger knowledge-awareness.