Source-linked AI summary
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, Gunhee Kim
TL;DR
Most VQA research has focused on images, leaving video-specific spatio-temporal reasoning insufficiently addressed. This paper introduces three video VQA tasks, the TGIF-QA dataset, and a dual-LSTM model with spatial and temporal attention; video-based methods outperform image-based methods, though differences may not be significant.
Problem
Most existing VQA work focuses primarily on images, while video VQA requires spatio-temporal reasoning from videos.
Method
The paper introduces three video VQA tasks, the TGIF-QA dataset, and a dual-LSTM approach with spatial and temporal attention.
Results
Video-based methods outperform image-based methods on the reported video QA tasks, although the differences may not be significant.
Takeaways & Limitations
TGIF-QA provides a benchmark connecting video understanding with video VQA through repetition and state-transition questions requiring multiple-frame analysis.
Takeaways & Limitations
The authors identify the sequence-to-sequence model and multimodal fusion design as areas for improvement, including possible use of 3D convolution.
Abstract
from arXiv · showhide
Vision and language understanding has emerged as a subject undergoing intense study in Artificial Intelligence. Among many tasks in this line of research, visual question answering (VQA) has been one of the most successful ones, where the goal is to learn a model that understands visual content at region-level details and finds their associations with pairs of questions and answers in the natural language form. Despite the rapid progress in the past few years, most existing work in VQA have focused primarily on images. In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways. First, we propose three new tasks designed specifically for video VQA, which require spatio-temporal reasoning from videos to answer questions correctly. Next, we introduce a new large-scale dataset for video VQA named TGIF-QA that extends existing VQA work with our new tasks. Finally, we propose a dual-LSTM based approach with both spatial and temporal attention, and show its effectiveness over conventional VQA techniques through empirical evaluations.
1. Introduction
TGIF-QA extends VQA from images to videos by introducing spatio-temporal reasoning tasks, a large-scale animated-GIF dataset, and a dual-LSTM model with attention.
- Motivation: Existing VQA work focuses primarily on images, while video VQA lacks large-scale datasets with well-defined tasks.Prior video datasets either extend image-style questions or require comprehensive understanding of textual movie data.
- Contributions: The paper introduces three video VQA tasks requiring spatio-temporal reasoning: repetition count, repeating action, and state transition.The tasks analyze multiple frames and complement standard frame-based VQA questions.
- Method: The proposed approach uses LSTMs for both question-answer pairs and video input, with spatial and temporal attention mechanisms.These mechanisms selectively attend to specific parts of the video while modeling visual-textual associations.
- Contributions: TGIF-QA contains 165K question-answer pairs collected from 72K animated GIFs.The dataset extends the TGIF animated-GIF resource to video VQA.
2. Related Works
The paper positions TGIF-QA as a video extension of image-centric VQA, emphasizing frame-level spatio-temporal reasoning over movie-script story comprehension and using spatial and temporal attention.
- Datasets and techniques: Most VQA datasets and techniques are image-centric, focusing on concepts such as objects, colors, locations, and spatial attention.TGIF-QA extends this line of work to short video clips rather than static images.
- Datasets: TGIF-QA creates question-answer pairs from short animated GIFs, which provide concise formats and cohesive visual stories.This distinguishes the dataset's source material from movie clips and scripts used in related video VQA work.
- Tasks: The paper uses open-ended and multiple-choice question formats, consistent with established VQA task formulations.Open-ended questions require predicting an answer word, while multiple-choice questions require selecting among candidates.
- Datasets and tasks: Movie-based video VQA datasets address action recognition or story comprehension, whereas TGIF-QA emphasizes visual spatio-temporal reasoning in animated GIFs.Its questions focus on frame-level repetition and state transitions rather than movie scripts and synopses.
- Techniques: The proposed model extends image-VQA attention approaches with both spatial and temporal attention for video inputs.The paper describes temporal attention as a distinctive component for capturing complex spatio-temporal video patterns.
3. TGIF-QA Dataset
TGIF-QA extends video VQA with three video-specific spatio-temporal tasks, a frame-based task, and QA pairs collected from animated GIFs. The dataset combines automated and crowdsourced construction and complements existing video VQA datasets with distinctive tasks and dynamic content.
- Dataset construction: The dataset contains 165,165 QA pairs collected from 71,741 animated GIFs, including 53,083 frame-QA pairs and 112,082 video-QA pairs.Frame QA uses 39,479 GIFs, while video QA uses 53,247 GIFs.
- Video QA tasks: TGIF-QA introduces repetition count, repeating action, and state transition tasks that require analyzing temporal changes in videos.Repetition count asks for the number of action repetitions; repeating action identifies a repeated action; state transition identifies what occurred before or after a state.
- Frame QA: Frame QA asks an open-ended question answerable from one or more video frames, using object, number, color, and location answers.The questions are generated automatically from TGIF captions using an NLP-based technique.
- Dataset construction: Crowdsourcing collected repetition and state-transition annotations from 595 workers, with validation clips used to filter negligent workers and expand the gold-standard answer set.Collected fields included repetition subjects, verbs, objects, counts, transition types, previous states, and next states.
- Dataset construction: QA generation applies linguistic post-processing, templates, grammatical correction, and multiple-choice option selection, with diverse negative answers selected using word and phrase embeddings.Verbs are represented with 300D GloVe vectors, while skip-thought vectors guide phrase selection.
- Comparison with other datasets: Compared with existing video VQA datasets, TGIF-QA uses short social-media GIFs, supports both open-ended and multiple-choice questions, and includes video-specific tasks despite being smaller than LSMDC-QA.Its answer verbs are more dynamic, especially in motion and contact categories, suggesting a need for spatio-temporal reasoning.
4. Approach
The ST-VQA approach encodes video and question/answer text with dual-layer LSTMs, then decodes answers using task-specific outputs. Spatial and temporal attention mechanisms preserve and selectively use region- and frame-level video information.
- Model overview: ST-VQA takes a video, question, and optional multiple-choice answer phrase, producing either a word or compatibility-score vector.The model uses separate dual-layer LSTMs for video and QA inputs.
- Feature representation: Video features combine frame-level ResNet-152 descriptors with sequence-level C3D descriptors, sampled every four frames to reduce redundancy.ResNet-152 and C3D features are extracted from pretrained models.
- Video and text encoders: Question and answer sequences are represented as word embeddings and encoded with dual-layer LSTMs, carrying visual information into the text encoder.The question encoder initializes from the final video-encoder state, while the answer encoder initializes from the final question-encoder state.
- Video and text encoders: The video encoder concatenates spatial and frame features at each time step and processes them sequentially with a dual-layer LSTM.The hidden-state dimension is D = 512.
- Answer decoders: Answer decoders use pairwise hinge loss for multiple-choice and repetition-count tasks, ℓ2 loss for counts, and softmax classification for frame QA.The multiple-choice decoder scores candidate answers, while open-ended decoders output integer or vocabulary-word answers.
- Attention mechanisms: Spatial attention selects regions within each frame, while temporal attention selects frames before passing attended information to the answer decoder.The two mechanisms address the spatial and temporal dimensions of video information separately.
5. Experiments
The experiments compare video- and image-based VQA baselines, feature representations, and attention mechanisms across TGIF-QA task types. Video-based methods outperform image-based methods, while temporal attention achieves the best performance in three tasks.
- Baselines and variants: The study compares two image-based VQA methods, one video-based method, and seven variants of the proposed model.Image-based baselines are adapted using frame aggregation or frame-wise averaging; the variants test input features and attention mechanisms.
- Evaluation: Accuracy is used for three task types, while repetition counting is evaluated with mean ℓ2 loss.The reported table distinguishes the repetition count metric from the other task metrics.
- Results: Video-based methods outperform image-based methods, although the differences may not be significant because C3D features already capture some spatio-temporal information.This result supports the need for spatio-temporal reasoning while qualifying the observed comparison.
- Input features: The Concat approach, which uses both ResNet and C3D features, achieves the best performance among the four input-feature baselines across all tasks.The comparison includes Text, ResNet, C3D, and Concat representations.
- Attention mechanisms: Temporal attention achieves the best performance in three tasks, while combining spatial and temporal representations supports selective attention over video content.Spatial attention selects regions within frames, whereas temporal attention selects frames.
- Qualitative analysis: Qualitative examples show that questions such as counting repeated head movements require spatio-temporal reasoning rather than a single-frame analysis.The proposed method combines ResNet and C3D representations with spatial and temporal attention mechanisms.
6. Conclusion
The paper concludes by presenting ST-VQA as an extension of VQA with new spatio-temporal tasks, a large-scale dataset, and a dual-LSTM model with spatial and temporal attention. It identifies 3D convolution and more principled visual-textual fusion as directions for improving the model.
- Contributions: The work contributes three video-VQA tasks, a dataset with 165K QA pairs from 72K animated GIFs, and a dual-LSTM model with spatial and temporal attention.These contributions extend existing image-focused VQA work to video.
- Future work: The authors plan to improve the sequence-to-sequence ST-VQA model by adopting 3D convolution and better combining visual-textual information.They cite multimodal compact bilinear pooling as one possible fusion approach.
7. Document Changelog
The document changelog records revisions from the initial CVPR 2017 camera-ready version through added statistics and updated Table 5 results. The document also includes acknowledgements and funding information.
- Revisions: Version 1 is identified as the initial CVPR 2017 camera-ready version.
- Revisions: Version 2 added statistics and results, including a text-only baseline, for the extended dataset.
- Revisions: Version 3 updated the results in Table 5 and uploaded relevant files to the repository.
- Acknowledgements: The acknowledgements thank collaborators and institutions and note partial support from Seoul National University’s Big Data Institute and Yahoo Research.