PaperlayerView exact revision on arXiv

Source-linked AI summary

KnowIT VQA: Answering Knowledge-Based Questions about Videos

Noa Garcia, Mayu Otani, Chenhui Chu, Yuta Nakashima

arXiv:1910.10706v3cs.CVcs.CL

TL;DR

KnowIT VQA addresses video questions that require temporal reasoning and knowledge beyond a clip. The paper introduces a 24,282-pair sitcom dataset and a model that combines video, language, and show-specific knowledge. In human evaluation, accuracy rose after participants saw the associated knowledge annotations, while the dataset remained challenging for people unfamiliar with the show.

  • Problem

    Image content alone leaves temporal coherence and questions requiring knowledge beyond a clip unaddressed, while a common framework for these question types is missing.

    Source: Introduction · p. 1

  • Method

    The paper introduces KnowIT VQA, a 24,282-sample video dataset, and ROCK, which retrieves show information from a knowledge base and fuses it with language and spatio-temporal video representations.

    Source: Data Splits · p. 4, ROCK Model · p. 5

  • Results

    0.683 to 0.823 accuracy after participants saw the associated KNOWLEDGE annotations, supporting their relevance to the questions.

    Source: Human Evaluation · p. 5

  • Takeaways & Limitations

    KnowIT VQA is the largest human-generated KBVQA dataset and the first collection addressing four question types, including knowledge and temporal coherence.

    Source: Dataset Comparison · p. 4

    Automatically generating similar knowledge explanations from videos remains future work, which the authors identify as difficult to capture.

    Source: Knowledge Base · p. 5

Abstract

from arXiv · show

We propose a novel video understanding task by fusing knowledge-based and video question answering. First, we introduce KnowIT VQA, a video dataset with 24,282 human-generated question-answer pairs about a popular sitcom. The dataset combines visual, textual and temporal coherence reasoning together with knowledge-based questions, which need of the experience obtained from the viewing of the series to be answered. Second, we propose a video understanding model by combining the visual and textual video content with specific knowledge about the show. Our main findings are: (i) the incorporation of knowledge produces outstanding improvements for VQA in video, and (ii) the performance on KnowIT VQA still lags well behind human accuracy, indicating its usefulness for studying current video modelling limitations.

Introduction

Image-based VQA focuses on visual content, leaving temporal coherence and questions requiring knowledge beyond the clip unresolved. KnowIT VQA combines video understanding with knowledge-based reasoning, and the proposed model fuses video, language, and show-specific knowledge.

Introduction

  • Image-only VQA misses temporal coherence and cannot answer questions requiring knowledge beyond the visual content; a common framework for these question types was missing.Examples include how a conversation ends and who owns the place shown.Source: Introduction · p. 1
  • KnowIT VQA is a video KBVQA dataset with natural questions designed to require familiarity with the sitcom.The paper presents the sitcom as a testbed for modelling knowledge-based questions about the world.Source: Introduction · p. 1
  • The proposed model acquires, processes, and maps show-specific knowledge to infer question motivation, then fuses it with video and language content to predict answers.Source: Introduction · p. 2

Related Work

VideoQA and KBVQA address complementary challenges, but video datasets often emphasize one modality and KBVQA approaches vary in how they obtain knowledge. KnowIT VQA combines freely proposed questions about knowledge and temporal coherence in videos.

Video Question Answering

  • Many video datasets emphasize textual or visual content separately; only a few combine information sources, and reasoning beyond video content remains difficult.The passage names PororoQA and TVQA as examples of datasets using multiple information sources.Source: Related Work · p. 2

Knowledge-Based Visual Question Answering

  • KBVQA informs models with external knowledge, but methods for acquiring and incorporating that knowledge remain at an early stage.Approaches range from specific image-focused knowledge bases to generic external sources such as DBpedia.Source: Related Work · p. 2
  • KnowIT VQA uses freely proposed questions to study knowledge and temporal coherence together, and the authors describe it as the first video collection exploring external-knowledge questions.The passage contrasts this with datasets that impose templates or obtain questions directly from knowledge bases.Source: Related Work · p. 2

KnowIT VQA Dataset

KnowIT VQA is a knowledge-informed video question-answering dataset built from The Big Bang Theory, designed to test questions that require show familiarity alongside visual, textual, and temporal reasoning. It contains 24,282 human-generated question-answer pairs and spans four question types.

KnowIT VQA Dataset

  • KnowIT VQA uses sitcom videos to model knowledge and temporal coherence, with questions intended to require familiarity with the show.The dataset comprises videos from The Big Bang Theory and is presented as a testbed for knowledge-based visual question answering.Source: KnowIT VQA Dataset · p. 2

Video Collection

  • Videos cover the first nine seasons of The Big Bang Theory: 207 episodes are segmented into 12,264 uniform 20-second clips, with subtitles aligned to transcript speakers and timestamps.Transcripts also provide scene information used for segmentation.Source: Video Collection · p. 3

QA Generation

  • Workers familiar with the show wrote each clip’s question, correct answer, and three relevant wrong answers, with a second collection round used to gather quality feedback and diverse questions.For context, workers saw the clip and subtitles and could consult the episode transcript and summaries; collection took about three months.Source: QA Generation · p. 3

Knowledge Annotations

  • The dataset contains visual, textual, temporal, and knowledge-based questions; knowledge-based questions are 62% of the set, while recurrent knowledge accounts for 6.08%.Visual-based questions are 22%, textual-based 12%, and temporal-based 4%; recurrent knowledge is distinguished from episode-specific knowledge.Source: Knowledge Annotations · pp. 3–4, Figure 2 · p. 3

Data Splits

  • The dataset has 24,282 samples from 12,087 clips, with episodes randomly assigned to training, validation, and test sets so clips and questions from one episode stay together.Correct answers are slightly longer than wrong answers, a bias also noted in prior QA datasets (Tapaswi et al. 2016; Lei et al. 2018).Source: Data Splits · p. 4

Dataset Comparison

  • With 24,282 questions, KnowIT VQA is the largest human-generated KBVQA dataset and the first collection addressing four question types.About 34% of its videos overlap with TVQA; its knowledge annotations required a larger set of clips to approximate viewers’ acquired knowledge.Source: Dataset Comparison · p. 4

Human Evaluation

Human evaluation tested whether the clips and knowledge annotations support answering KnowIT VQA questions and compared workers with different familiarity with the show. Accuracy increased after rookies were shown the annotations, while evaluation results also supported the relevance of video and show knowledge.

Evaluation Design

  • Workers were grouped as masters, who had watched at least nine seasons, or rookies, who had watched no episodes; question evaluation varied the available QA, subtitle, and video context.Each question was answered by three workers in each group; the evaluation also assessed knowledge annotations.Source: Human Evaluation · p. 4

Evaluation on the questions

  • Masters mostly relied on knowledge acquired from watching the show, whereas rookies admitted not knowing the answer in most responses.The accuracy gap between Subs and Video groups also supports the relevance of video content.Source: Human Evaluation · p. 4

Evaluation on the knowledge

  • After participants answered with subtitles and video, they were shown associated KNOWLEDGE and answered again; accuracy increased from 0.683 before KNOWLEDGE exposition to 0.823 after, verifying annotations’ relevance to questions.Source: Human Evaluation · p. 5

ROCK Model

ROCK retrieves show-specific knowledge and fuses it with language and spatio-temporal video representations to answer questions. Its pipeline builds a natural-language knowledge base, cleans near duplicates, and ranks knowledge instances for each question and candidate answer.

ROCK Model

  • ROCK retrieves show-related knowledge and fuses it with language and spatio-temporal video representations to reason over questions and predict answers.The model relies on language instances representing show information in a knowledge base.Source: ROCK Model · p. 5

Knowledge Base

  • The knowledge base uses AMT-provided KNOWLEDGE annotations as natural-language sentences to represent show-specific information, rather than generic knowledge graphs.The collection is cleaned from 24,282 to 19,821 instances.Source: Knowledge Base · p. 5
  • Knowledge instances with cosine similarity above 0.998 are connected, clustered by graph connectivity, and reduced to one randomly selected instance per cluster.The authors report that 0.998 was a good tradeoff between near duplicates and semantically similar instances.Source: Knowledge Base · p. 5

Knowledge Retrieval Module

  • For each question and candidate answer, the retrieval module constructs a query input, scores its relevance to knowledge instances with BERT, and ranks those instances.The input combines the question, candidate answer, and knowledge instance; candidate answers are sorted by a prior relevance score for order invariance.Source: Knowledge Retrieval Module · p. 6

S · BERTS(xij) + bS), (3)

The scoring model uses BERT-based question–knowledge matching, while a separate prior score orders candidate answers to make the model invariant to their input order.

S · BERTS(xij) + bS), (3)

  • BERTS is fine-tuned on matching and non-matching question–knowledge pairs using the supplied loss, then used to rank knowledge instances for each question.The top-ranked instances are selected as the most relevant knowledge samples.Source: S · BERTS(xij) + bS), (3) · p. 6
  • A prior score predicts how likely each candidate answer is correct, and the answers are sorted by that score to make their ordering invariant.The candidate question–answer input is tokenized to 120 tokens for the BERT prior scorer.Source: S · BERTS(xij) + bS), (3) · p. 6

E BERTE(ec) + bE, (5)

The answer predictor combines video and language representations, while a prior BERT score sorts candidate answers. Video reasoning encodes visual content and combines it with captions, subtitles, the question, candidate answer, and retrieved knowledge.

E BERTE(ec) + bE, (5)

  • Candidate answers are sorted in descending prior-score order to provide an answer-order-invariant input.The prior score is predicted from each question–candidate answer pair.Source: E BERTE(ec) + bE, (5) · p. 6

Video Reasoning Module

  • Visual representations are built from sampled frames using image, object-and-attribute concept, facial, or caption features.Image and concept features are projected to 512 dimensions; facial features use a cast-trained face recognizer and a projected bag-of-faces representation.Source: Video Reasoning Module · p. 6
  • For each candidate answer, BERT-reasoning encodes ordered frame captions, subtitles, the question, the answer, and retrieved knowledge as a language representation.The captions are ordered by timestamp, and the retrieved knowledge instances are concatenated into the input.Source: Video Reasoning Module · p. 6
  • The answer predictor concatenates a visual representation with each candidate's language representation, scores the candidates, and selects the highest-scoring answer.The prediction model is fine-tuned using multi-class cross-entropy loss.Source: Video Reasoning Module · p. 7

Experimental Results

The experiments compare ROCK and baselines on KnowIT VQA, testing question-answer, subtitle, visual, and knowledge inputs. Knowledge and subtitle-based models improve results in different comparisons, while visual models remain below ROCKSQA and ROCK remains below human masters.

Experimental Results

  • Answer heuristics performed poorly overall, with only Longest beating random; correct answers tended to be longer, but similarities showed no strong bias.The baselines included word-count heuristics and word2vec/BERT answer-similarity methods.Source: Experimental Results · p. 7
  • Sentence-similarity methods performed worse than random, while TFIDF, LSTM Emb./BERT, and ROCKQA outperformed human workers.These answer-prediction classifiers used questions and candidate answers as input.Source: Experimental Results · p. 7
  • LSTM BERT and ROCKSQA improved accuracy by 5.7% over question-answer-only models, whereas LSTM Emb. did not improve over those models.The authors suggest that word embeddings may have difficulty encoding long subtitle sequences.Source: Experimental Results · p. 7
  • ROCKVSQA outperformed TVQA by 6.6%, with Concepts highest; no visual model outperformed ROCKSQA.The authors interpret this as evidence of limitations in current video modelling approaches.Source: Table 4 · p. 7, Experimental Results · p. 8
  • Knowledge raises accuracy 6.5% over non-knowledge methods, but ROCK remains below human masters; annotated KNOWLEDGE lifts ROCKGT to 0.731.Image, Concepts, and Facial representations performed the same for the knowledge models.Source: Table 4 · p. 7, Experimental Results · p. 8
  • The proposed prior-scoring method retrieved knowledge best, and using all candidate answers together provided more context than using only questions.The comparison included question-only input and parameter-sharing QA input.Source: Experimental Results · p. 8, Table 5 · p. 8

Conclusion

The paper presents a dataset and video reasoning model for knowledge-based visual question answering in videos. Its evaluation finds potential in knowledge-based models but a substantial gap from human performance.

Conclusion

  • The paper presents a dataset for knowledge-based visual question answering in videos and a video reasoning model combining multimodal video information with task-specific knowledge.Source: Conclusion · p. 8
  • The evaluation indicates the potential of knowledge-based models for video understanding problems.Source: Conclusion · p. 8
  • A substantial gap from human performance remains, and the authors hope the dataset will encourage stronger models that help reduce it.Source: Conclusion · p. 8

Related papers

Loading 1910.10706v3…