Source-linked AI summary
Leveraging Video Descriptions to Learn Video Question Answering
Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, Min Sun
TL;DR
Video QA requires scalable training data beyond costly manual annotation, especially for videos in the wild and free-form answer spaces. The paper harvests online videos and descriptions, generates candidate QA pairs automatically, and uses self-paced learning to mitigate noisy pairs; self-paced learning is effective and extended SS outperforms baselines.
Problem
Building large-scale video QA datasets is costly, while existing approaches have limited support for videos in the wild and more challenging answer spaces.
Method
The approach harvests videos with user-curated descriptions, automatically generates candidate QA pairs, and uses a ratio test with self-paced learning to reduce the effect of non-perfect pairs.
Results
Self-paced learning is effective, and the extended SS method outperforms other baselines and extended methods.
Takeaways & Limitations
The approach produces a large-scale video QA dataset with minimal human effort while improving the effectiveness of extended video QA models.
Takeaways & Limitations
The task remains limited by automatically generated QA pairs, 10% of which are observed to be irrelevant or inconsistent with the corresponding video content.
Abstract
from arXiv · showhide
We propose a scalable approach to learn video-based question answering (QA): answer a "free-form natural language question" about a video content. Our approach automatically harvests a large number of videos and descriptions freely available online. Then, a large number of candidate QA pairs are automatically generated from descriptions rather than manually annotated. Next, we use these candidate QA pairs to train a number of video-based QA methods extended fromMN (Sukhbaatar et al. 2015), VQA (Antol et al. 2015), SA (Yao et al. 2015), SS (Venugopalan et al. 2015). In order to handle non-perfect candidate QA pairs, we propose a self-paced learning procedure to iteratively identify them and mitigate their effects in training. Finally, we evaluate performance on manually generated video-based QA pairs. The results show that our self-paced learning procedure is effective, and the extended SS model outperforms various baselines.
Introduction
The paper targets scalable video question answering by replacing much of the manual training-data construction with automatically generated QA pairs from harvested video descriptions. It introduces self-paced learning to reduce the impact of imperfect pairs and reports that the extended SS model performs best among the evaluated baselines.
- Motivation: Video QA offers more specific visual understanding than open-ended captioning and supports standard accuracy evaluation.Questions focus attention on relevant video content while reducing the space of relevant answers.
- Motivation: Manual construction of video-QA benchmarks requires substantial human effort, especially when annotators must inspect videos and localize answer context.MovieQA illustrates this costly process, producing 14944 QA pairs from 408 movies.
- Approach: The proposed pipeline harvests online videos and descriptions, then automatically generates candidate QA pairs instead of manually constructing training pairs.The collected dataset contains 18100 videos and 175076 candidate QA pairs.
- Approach: Automatic QA generation scales but introduces noise: 10% of generated pairs are irrelevant or inconsistent with their videos.The paper reports that noisy training pairs can harm supervised video-QA frameworks.
- Approach: A ratio test and self-paced learning iteratively identify non-perfect pairs and mitigate their effects during training.The method compares losses on the original video and a dummy video to identify questionable pairs.
- Results: Across experiments on manually generated QA pairs, self-paced learning is effective and the extended SS method outperforms other baselines.The evaluation uses 2000 videos and about 2500 manually generated ground-truth QA pairs.
Related Work
Prior video-QA datasets rely heavily on manual annotation or constrained question formats, while this paper pursues scalable, open-domain free-form QA from harvested videos and descriptions. Its approach builds on automatic question generation and extends vision-language QA models for video.
- Image-QA: Image-QA methods commonly require manually collected, correct visual question-answer data for training.The related work includes models combining image and question representations or separate question and answer LSTMs.
- Question generation: The adopted question-generation method ranks candidate QA pairs statistically and generates them from open-domain video-description sentences.This differs from approaches that restrict questions to objects, numbers, colors, and locations.
- Video-QA: Existing video-QA work includes query answering, multimodal movie QA, and automatically generated fill-in-the-blank questions.MovieQA requires substantial annotation, while another dataset automatically generates 390744 fill-in-the-blank questions from caption data.
- Video-QA: The proposed task answers free-form natural-language questions about videos rather than fill-in-the-blank questions.Videos and descriptions are harvested from an online repository without additional manual effort to create descriptions.
- Dataset: The dataset contains 18100 open-domain videos, with automatically generated training and validation QA pairs and human-generated testing QA pairs.The training, validation, and testing sets contain 151263, 21352, and 2461 QA pairs, respectively.
Video Question Answering Dataset
The dataset harvests curated Internet videos and descriptions, then automatically generates candidate QA pairs and supplements them with human cleaning and evaluation data. Its questions and answers emphasize actions, human roles, and diverse video content rather than the object-, color-, and number-focused patterns common in image QA.
- Data collection: The dataset begins with videos from an online curated repository, each accompanied by several owner-submitted sentences refined by staff editors.The videos are open-domain, handheld-camera recordings with variable quality and camera motion.
- Question generation: Candidate QA pairs are automatically generated from each description sentence using a state-of-the-art question-generation method.The method is expected to produce some imperfect pairs, which are addressed later in the paper.
- Human quality control: Turkers clean a subset by labeling QA pairs as correct, irrelevant, or correctable, then revising the correctable cases.Only about 10% of QA pairs require the second correction step.
- Question and answer analysis: Questions include diverse 5W1H forms and many action- or motion-oriented queries, distinguishing them from image-QA questions focused mainly on objects, colors, and numbers.Auxiliary verbs such as Does, Did, and Do are common because many questions concern event-description verbs.
- Question and answer analysis: Answers contain substantial human-role and action content, while Yes and No each account for 32.5% of the whole set.The answer distribution excludes Yes/No when showing its eight manually defined categories.
Our Method
Video-QA learns an answer-generating function from video, question, and answer sequences while reducing the influence of automatically generated pairs that conflict with video content. The method combines a ratio-based filtering test, self-paced retraining, and extensions of four sequence or attention architectures.
- Video-QA formulation: Video-QA predicts a natural-language answer from a question and a video observation represented as sequences of image observations.The learned function is written as a = f(v, q), with training triplets automatically generated from videos and descriptions.
- Self-paced learning: A ratio test identifies training pairs whose losses indicate weak relevance or consistency with the associated video.The test compares the loss for the original video with the loss after changing the video content.
- Self-paced learning: Self-paced learning replaces videos passing a thresholded ratio test with a dummy video and repeatedly retrains the model.The procedure stops when no additional videos satisfy the ratio test.
- Extended methods: The paper extends MN, VQA, SA, and SS for video QA using combinations of sequence encoding, embeddings, and soft attention.All extended methods are trained end-to-end with the self-paced procedure.
Experiments and Results
Experiments train and evaluate extended video-QA methods using candidate training data, validation candidates, and manually generated test QA pairs. Self-paced learning improves the extended methods, and E-SS achieves the strongest reported average accuracy among them.
- Experimental setup: The split uses 14,100 training videos with 151,263 candidate QA pairs, 2,000 validation videos with 21,352 candidates, and 2,000 test videos with 2,461 ground-truth pairs.
- Evaluation: Performance is evaluated with classification accuracy, relaxed WUPS at thresholds 0.0 and 0.9, and separate Yes/No and Others results.The answer space contains 1K candidates, making exact classification accuracy stricter than word-similarity scoring.
- Extended methods: 31.0% average Acc is achieved by self-paced E-SS, which outperforms the other extended methods by jointly encoding videos and questions sequentially.
- Ablations: All extended methods perform worse without video observations, showing that their non-visual variants cannot answer using the question alone.
- Self-paced learning: Self-paced learning improves all extended methods, while E-SA gains least because soft attention can select different visual observations under noisy training pairs.
- Self-paced learning: 2.6% improvement in average accuracy is obtained by E-SS over its Train-all version after self-paced learning.
Conclusions
The scalable approach produces a large video-QA dataset with minimal human effort, while extended models and self-paced learning are effective. Future work will expand the dataset and improve handling of non-perfect training examples.
- 18,100 videos and 175,076 QA pairs were generated with minimal human effort.The approach produces a large-scale video-based question-answering dataset.
- Extended video-QA models and the self-paced learning procedure were effective.The procedure addresses non-perfect training examples.
- Future work will increase the Video-QA dataset's scale and improve handling of more non-perfect training examples.