TVQA+: Spatio-Temporal Grounding for Video Question Answering

Jie Lei; Licheng Yu; Tamara L. Berg; Mohit Bansal

Source-linked AI summary

TVQA+: Spatio-Temporal Grounding for Video Question Answering

Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal

arXiv:1904.11574v2Apr 25, 2019cs.CVcs.AIcs.CL

TL;DR

Video QA systems have often lacked spatial evidence grounding alongside temporal localization, despite both being relevant to answering questions. The paper introduces TVQA+ with grounded bounding boxes and STAGE, a unified framework for spatial-temporal QA. The authors report improved QA performance from temporal and spatial predictions, interpretable visualizations, and state-of-the-art performance that remains below human performance.

Problem
Existing video QA datasets provide limited spatial annotation, motivating systems that jointly localize relevant moments and referenced visual regions for answering questions.
Method
The paper augments TVQA into TVQA+ with grounded bounding boxes and proposes STAGE to jointly perform moment localization, object grounding, and video question answering.
Results
Temporal and spatial predictions improve QA performance, while STAGE produces explainable spatio-temporal visualizations and achieves state-of-the-art performance.
Takeaways & Limitations
Joint spatio-temporal grounding supports video QA with evidence visualizations that help explain what the model has learned.
Takeaways & Limitations
STAGE still has a large performance gap compared with human performance, and its full model is not overwhelming on “why” and “how” questions.

Abstract

from arXiv · show

We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos. We first augment the TVQA dataset with 310.8K bounding boxes, linking depicted objects to visual concepts in questions and answers. We name this augmented version as TVQA+. We then propose Spatio-Temporal Answerer with Grounded Evidence (STAGE), a unified framework that grounds evidence in both spatial and temporal domains to answer questions about videos. Comprehensive experiments and analyses demonstrate the effectiveness of our framework and how the rich annotations in our TVQA+ dataset can contribute to the question answering task. Moreover, by performing this joint task, our model is able to produce insightful and interpretable spatio-temporal attention visualizations. Dataset and code are publicly available at: http: //tvqa.cs.unc.edu, https://github.com/jayleicn/TVQAplus

1 Introduction

The paper argues that video QA needs both temporal moment localization and spatial grounding of referenced visual concepts. It introduces TVQA+ and STAGE to jointly ground evidence and answer questions, with reported gains and interpretable visualizations.

Motivation: Existing video QA datasets, including TVQA, provide temporal annotations but generally lack spatial annotations for answer-relevant regions.The paper motivates spatial grounding as complementary to locating the relevant video moment.
Dataset: TVQA+ augments TVQA with grounded frame-level bounding boxes for visual concepts mentioned in questions and correct answers.The dataset contains 29.4K multiple-choice questions, 310.8K bounding boxes, and 2.5K object and people categories.
Task: The proposed task requires systems to localize relevant moments, detect referred objects and people, and answer questions.The task evaluates question answering alongside temporal and spatial grounding.
Method: STAGE is an end-to-end framework that jointly combines moment localization, object grounding, and question answering.The model uses grounded evidence to produce spatio-temporal attention visualizations.
Findings: The authors report significant gains over baselines, benefits from both temporal and spatial supervision, and interpretable spatio-temporal visualizations.Ablation studies examine how the annotations and model components contribute to task performance.

2 Related Work

Prior video QA work typically predicts answers without jointly grounding them in time and space. This paper connects video moment retrieval and visual object grounding through a unified spatio-temporal QA task.

Video Question Answering: Existing video question answering datasets and tasks commonly focus on predicting answers from video and language inputs.The related-work discussion contrasts these tasks with the proposed grounded formulation.
Video Question Answering: The proposed task additionally grounds answers in both spatial and temporal domains.This extends answer prediction with localization of relevant moments and referred visual regions.
Language-Guided Retrieval: Language-guided retrieval includes referring-object identification and moment retrieval, while this work integrates both goals simultaneously.The integrated task requires grounding referred moments and objects at once.

3 Dataset

TVQA+ augments TVQA with spatial bounding boxes linked to visual concepts while retaining temporal annotations, enabling question answering, temporal localization, and spatial localization.

Dataset construction: TVQA+ extends TVQA, whose questions combine visual and subtitle information with timestamps identifying the minimum answering context.The original TVQA contains 152.5K multiple-choice questions from 21.8K clips across six television shows.
Dataset construction: 29,383 QA pairs from 4,198 clips were selected for bounding-box annotation in the The Big Bang Theory subset.
Data collection: Visual concepts are extracted from nouns in questions and correct answers, with frequent non-visual nouns manually removed.CoreNLP yields 152,722 words from a 9,690-word vocabulary; 165 frequent non-visual nouns are removed.
Data collection: Workers refine temporal timestamps, sample one frame every two seconds, and annotate bounding boxes for referenced concepts in each sampled frame.Original videos with subtitles provide context during annotation, while semi-automated face annotation performed poorly under partial occlusion.
Dataset analysis: TVQA+ contains 148,468 annotated images and 310,826 bounding boxes, supporting question answering, temporal localization, and spatial localization.The dataset follows the original TVQA data splits and covers 2,527 categories, averaging 2.09 boxes per image and 10.58 boxes per question.
Dataset analysis: Most boxes occupy small image areas and most localized spans are shorter than 10 seconds, with an average span length of 7.2 seconds.The largest spans reach 20 seconds, compared with an average full-clip length of 61.49 seconds.

4 Methods

STAGE is a unified framework that uses video, subtitles, and question-answer hypotheses to jointly ground relevant objects and temporal spans before predicting answers. Its training combines answer, spatial-attention, and temporal-span objectives, while hard span-based evidence supports interpretable localization.

STAGE Architecture: STAGE encodes video regions and text, computes QA-guided attention over objects and subtitles, fuses the modalities, localizes temporal spans, and predicts answers.It uses frame-wise regional visual representations, contextual encoding, and joint frame-wise video-text representations.
Formulation: The task takes a question with five candidate answers, a 60-second video, and aligned subtitle sentences as input.Frames are sampled at 0.5 FPS, with each frame paired with two neighboring subtitle sentences.
STAGE Architecture: STAGE computes attention from each QA word to object regions and subtitle words to produce QA-aware visual and subtitle representations.Object and subtitle attention are computed separately before video-text fusion.
Temporal Localization: The span predictor estimates start and end probabilities for joint video-text temporal localization, rather than applying span prediction to text alone.The model uses aligned joint embeddings to predict temporal spans relevant to the video question-answering task.
Answer Prediction: Hard localized span features are combined with global video features for answer prediction, producing more interpretable evidence than soft temporal attention.Candidate spans are proposed from predicted start and end probabilities, then pooled into local representations before classification.
Training and Supervision: The overall training loss combines answer, attention, and span objectives, while spatial supervision treats boxes with IoU ≥0.5 as positive.The attention and span losses apply to the ground-truth hypothesis; the weights are watt=0.1 and wspan=0.5.

5 Experiments

Experiments evaluate STAGE on spatio-temporal QA using answer, temporal localization, joint answer-span, and object-grounding metrics. STAGE improves over baselines, while ablations show benefits from aligned fusion, temporal and spatial supervision, and local span features, with remaining weaknesses on some question types.

5.1 Metrics: STAGE evaluates answer accuracy, temporal localization, joint answer-span accuracy, and object grounding using QA Acc., Temp. mIoU, ASA, and Grd. mAP.ASA counts a prediction as correct when the answer is correct and the predicted span has IoU ≥0.5; Grd. mAP uses IoU threshold 0.5.
5.2 Comparison with Baseline Methods: 32.49% temporal mIoU and 27.34% Grd. mAP measure STAGE’s localization of relevant moments and referred objects or people.The results still leave a large gap between STAGE and human performance.
5.3 Model Analysis: 68.31% QA Acc. for the backbone exceeds the 65.79% baseline, supporting the contribution of aligned fusion and the STAGE design changes.The backbone removes span-related components and explicit attention supervision, replaces CNN encoders with RNN encoders, and removes aligned fusion.
5.3 Model Analysis: 121.92% relative gain in Grd. mAP follows the addition of spatial supervision, which also improves other task performance.Temporal supervision enables temporal grounding and improves performance on other tasks before spatial supervision adds further improvements.
5.3 Model Analysis: Local features from max-pooled span-proposal regions produce the best performance across all metrics.These regions contain more relevant cues for answering questions.
Accuracy by Question Type: QA accuracy increases for “what,” “who,” and “where” questions, but the full model lacks overwhelming performance on “why” and “how” questions.The authors identify textual reasoning as future work for these question types.
TVQA Results: Performance improves to 70.23% on the full TVQA dataset after adding temporal supervision, while the backbone gains 3.91% relative over the best published result.STAGE variants using GloVe still achieve better results than the comparison systems.

6 Conclusion

The paper introduces TVQA+ and the spatio-temporal video QA task, then presents STAGE as an end-to-end framework for jointly grounding moments, objects or people, and answers. Experiments show that temporal and spatial predictions improve QA and provide explainable results, although performance remains below human results.

Conclusion: TVQA+ supports a spatio-temporal video QA task requiring systems to localize relevant moments, detect referred objects or people, and answer questions.The dataset and task combine temporal and spatial grounding with question answering.
Conclusion: STAGE is an end-to-end trainable framework that jointly performs moment localization, object or people grounding, and question answering.The framework is designed to perform all three subtasks together.
Conclusion: Temporal and spatial predictions improve QA performance while providing explainable results.The paper presents these predictions as producing interpretable outputs alongside better QA.
Conclusion: STAGE achieves state-of-the-art performance but remains separated from human performance by a large gap.The authors identify this gap as room for further improvement.

A.1 Timestamp Annotation

The timestamp annotation process addresses loose temporal spans in TVQA by refining longer annotations with Amazon Mechanical Turk workers. Refined timestamps are substantially shorter and consistently improve STAGE performance on the TVQA+ validation set.

Timestamp refinement: 8.7% of 150 randomly sampled training questions had spans at least 5 seconds longer than needed, motivating timestamp refinement.Questions with localized spans longer than 10 seconds, representing 41.33% of questions, were selected for refinement.
Timestamp refinement: Models using refined timestamps consistently outperform models using the original timestamps on the TVQA+ validation set.The comparison is reported in Table 7 across the evaluated models.

A.2 Bounding Box Annotation

Bounding-box annotation links visual concepts in questions and answers to regions in sampled video frames. Manual collection was used to ensure accuracy after automated face annotation produced poor quality, while face retrieval remained limited by partial occlusion.

Bounding-box collection: Workers receive a question, its correct answer, and sampled frames, then draw a box around each highlighted visual concept such as “laptop.”Each task presents one concept-word and sampled-frame pair; wrong answers are not annotated.
Bounding-box collection: Only 3.13% of sampled wrong answers were groundable, compared with 46% of correct answers.The authors checked 200 sampled question-answer pairs before restricting annotation to correct answers.
Face annotation: Automated character-face annotation produced poorer quality than expected, so the authors manually collected boxes to ensure accuracy.Manual collection was also used to create ground-truth face labels for evaluating face retrieval.
Face annotation: 55.6 F1, 74.4 Precision, and 44.4 Recall were insufficient for further research on face retrieval, partly because television faces are often partially occluded.The evaluation covered the 12 most frequently appearing characters in TVQA+.

A.3 Quality

TVQA+ bounding-box quality was controlled through stringent worker qualifications, ongoing monitoring, and in-house verification.

Workers were restricted to English-speaking countries and required at least 3,000 accepted HITs with a 95% acceptance rate.Qualified workers were also well paid, and workers producing poor annotations were disqualified.
An in-house check found 95.5% of 200 sampled QA

A.4 Training Details

The model uses Adam-based training with early stopping and can incorporate LXMERT features in place of Faster R-CNN and BERT features. Qualitative examples show that STAGE generally localizes correct evidence but struggles with unusual or small objects and some temporal intervals.

Training Details: Training uses Adam with a 1e-3 initial learning rate, 3e-7 weight decay, batch size 16, and a maximum of 100 epochs.Training stops when QA Acc. fails to improve for five consecutive epochs.
Training Details: CNN hidden size is set to 128.
Training Details: LXMERT features can replace Faster R-CNN object features and BERT question features for video frame-question inputs.LXMERT is pretrained on image-text, image captioning, and image question answering datasets.
Qualitative Analysis: Correct STAGE examples usually exhibit both correct temporal and spatial localization.The examples include grounding visualizations with predicted spans, confidence colors, and ground-truth boxes shown in green.
Qualitative Analysis: Incorrect object localization is frequent for unusual or small objects, while incorrect temporal localization is another frequent failure reason.The failure analysis identifies unusual objects, small objects, and incorrect temporal intervals as recurring difficulties.