Source-linked AI summary
TVQA: Localized, Compositional Video Question Answering
Jie Lei, Licheng Yu, Mohit Bansal, Tamara L. Berg
TL;DR
Video QA lacked a large benchmark that jointly tested visual understanding, subtitle comprehension, and temporal grounding. TVQA introduces such a dataset with compositional questions, evaluates baseline systems, and presents a multimodal neural model; experiments show that both visual and textual understanding are necessary, while a gap remains to human performance.
Problem
Video-QA research has had less coverage than image-QA, and existing datasets impose limitations on clip length, domain, modality, or question structure.
Method
TVQA combines 60–90-second TV clips, aligned subtitles, compositional human-written questions, temporal annotations, baseline experiments, and a multi-stream end-to-end language-and-vision model.
Results
Both visual and textual understanding are necessary for TVQA, while the best performance uses all contextual sources, including subtitles and videos.
Takeaways & Limitations
TVQA provides a benchmark for multimodal video QA with grounded moments, subtitle dialogue, and full-clip evaluation.
Takeaways & Limitations
A significant gap remains between the proposed baselines and human QA accuracy.
Abstract
from arXiv · showhide
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks. However, due to data limitations, there has been much less work on video-based QA. In this paper, we present TVQA, a large-scale video QA dataset based on 6 popular TV shows. TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video. Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts. We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task. The dataset is publicly available at http://tvqa.cs.unc.edu.
1 Introduction
TVQA addresses limitations in prior video-QA datasets with a large, natural, multimodal benchmark built around compositional questions. It supports answering questions about grounded moments, localizing those moments, and answering from full clips.
- Motivation: Prior video-QA datasets were limited by short clips, text-only summaries, cartoon content, or template-generated questions.These constraints reduced coverage of visual information, natural video understanding, or compositional questioning.
- Motivation: TVQA uses subtitle dialogue alongside video because answering real-world questions can require combining what systems see with what they hear.The dataset targets multimodal understanding of ongoing actions and interactions.
- Dataset design: Workers wrote compositional questions with a main question and a grounding clause that identifies the relevant moment.For example, a question can ask what people are discussing when they are sitting on a couch.
- Tasks and baselines: The dataset supports QA on grounded clips, question-driven moment localization, and QA on full video clips.The paper provides baselines for both QA tasks and introduces a language-and-vision model, while leaving moment localization for future work.
2 Related Work
Related work spans image- and video-based QA, textual reasoning, and natural-language grounding. TVQA differs by requiring joint vision and language understanding while connecting questions to moments in video.
- QA datasets: Prior work includes image-QA and video-QA datasets covering diverse domains, formats, and question-generation strategies.Examples include VQA, Visual7W, CLEVR, TGIF-QA, MovieFIB, LSMDC, TRECVID, MovieQA, and PororoQA.
- Textual reasoning: Text-QA datasets evaluate language understanding, whereas TVQA questions require both vision understanding and language understanding.Text-QA methods are not directly applicable, but they can inspire suitable TVQA models.
- Natural-language grounding: Natural-language grounding research localizes objects or moments in visual media from language descriptions.This line of work provides relevant background for TVQA’s question-driven moment localization task.
3 TVQA Dataset
TVQA is a multimodal video-QA dataset built from long-running television shows, with questions designed to combine video, subtitles, and temporal localization. Its analyses characterize question content, localized moments, show-specific vocabulary, and the importance of combining visual and textual information.
- Dataset construction: TVQA contains 925 episodes from six TV shows spanning 461 hours, segmented into clips of roughly 60–90 seconds.The shows cover sitcoms, medical dramas, and crime drama.
- Question and answer collection: Workers viewed videos with aligned subtitles and wrote compositional questions whose second part localized the relevant moment.Each question also received start and end timestamps, one correct answer, and four challenging wrong answers.
- Dataset analysis: Each clip contributes seven five-choice questions, while questions average 13.5 words and emphasize object, action, person, reasoning, and location content.Only one of the five candidate answers is correct.
- Dataset analysis: Most questions localize moments shorter than 15 seconds within the longer 60–90-second clips.This creates a temporal-localization component in addition to answer selection.
- Dataset analysis: Question vocabulary differs across shows: sitcoms emphasize everyday objects and activities, medical dramas medical terms, and Castle detective terms.Even shows within the same genre exhibit distinct topics, such as “game” and “laptop” for BBT versus “bar” and “beer” for HIMYM.
- Human evaluation: Human accuracy reaches 89.41% with video, subtitles, and questions, compared with 32.61% using question–answer pairs alone.The study reports that adding visual and textual sources significantly improves human performance.
4 Methods
TVQA uses a multi-stream neural model that jointly processes subtitles, questions, candidate answers, and multiple visual representations. Context-query matching produces stream-specific answer scores that are combined for prediction.
- Model overview: The model takes a video clip, subtitles, a question, and five candidate answers as inputs.Its contextual streams include regional visual, visual concept, and subtitle information.
- Video features: Regional visual features are extracted from detected object regions, while visual concept features represent unique detected objects and attributes.Faster R-CNN supplies detections; concept words are embedded with GloVe.
- Sequence encoders: BiLSTMs encode subtitle, question, answer, concept, regional, and ImageNet feature sequences into contextual representations.Regional and ImageNet features are projected into word-vector space before BiLSTM encoding.
- Context-query matching: A context-matching module uses question and answer representations as queries against each visual or subtitle context.It produces context-aware question and answer representations for subsequent fusion.
- Answer prediction: Fused stream representations are processed by another BiLSTM, temporally max-pooled, and converted into candidate-answer probabilities.When multiple streams are used, their scores are summed to obtain the final score.
5 Experiments
Experiments compare simple retrieval and similarity baselines with the proposed multimodal model on full and timestamp-localized clips. Results show that subtitles and video complement question-answer information, localization helps, and human-written negatives remain challenging.
- Setup: The dataset is split into 80% training, 10% validation, and 10% test data, with evaluation using multiple-choice accuracy.The test set is divided into public and reserved subsets.
- Baseline comparison: 30.22% accuracy is achieved by the longest-answer baseline, versus 20% random chance.Retrieval and question-answer similarity baselines perform poorly without video or subtitle context.
- Baseline comparison: 49.59% accuracy is achieved by TFIDF subtitle-answer similarity, substantially improving over question-answer similarity baselines.The authors relate this result to questions involving specific objects, locations, and other concepts mentioned in subtitles.
- Model variants: The best model uses all contextual sources, including subtitles and video with visual concept features.The model variants compare different contextual inputs and feature types.
- Localization: Timestamp annotation consistently improves model accuracy compared with using full-length videos and subtitles.This comparison indicates that localizing relevant moments benefits question answering.
- Negative answers: Human-written negative answers are more challenging than randomly sampled negatives.Performance is much higher when negatives are randomly sampled within the same show.
- Qualitative analysis: Failure cases include distraction by locations and questions requiring common sense beyond textual or visual cues.The examples are drawn from predictions of the proposed joint model.
6 Conclusion
The paper presents TVQA as a large-scale, localized, compositional video-QA benchmark and reports that successful answering requires both visual and textual understanding. A substantial gap from human QA accuracy remains, motivating stronger multimodal and temporal reasoning methods.
- Conclusion: TVQA provides a large-scale, localized, compositional video question-answering dataset with QA tasks defined with and without timestamps.The paper also supplies baseline experiments for future comparison.
- Conclusion: The experiments show that TVQA requires both visual and textual understanding.This conclusion concerns the multimodal nature of the benchmark.
- Limitations and future work: A significant gap remains between the proposed baselines and human performance on QA accuracy.The authors identify multimodal interaction, human-object relations, and temporal cues as directions for future work.
A Appendix
The appendix characterizes TVQA’s question, answer, and character-specific vocabulary distributions and provides additional qualitative prediction examples. Questions are generally long, while answer lengths show no dominant peak.
- Answer distributions: TVQA has no peak answer length.The appendix explicitly notes the absence of a dominant answer-length mode.
- Character vocabulary: Top unique nouns are reported separately for characters in Grey’s Anatomy and How I Met Your Mother.The appendix uses these tables to characterize show-specific vocabulary.
- Qualitative examples: Additional TVQA examples show correct predictions in upper rows and incorrect predictions in lower rows.Ground-truth answers are highlighted in green and model predictions are marked with check symbols.