Source-linked AI summary
ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, Dacheng Tao
TL;DR
VideoQA lacks large, fully human-annotated benchmarks for complex and long videos. This paper introduces ActivityNet-QA, analyzes its properties and baseline performance, and studies video representation strategies, finding that dynamic sampling improves baseline performance by at least 1%.
Problem
Existing VideoQA datasets are often small, automatically annotated, and short, limiting their suitability for evaluating complex video understanding.
Method
The paper constructs ActivityNet-QA from 5,800 ActivityNet videos with 58,000 crowdsourced QA pairs, then analyzes the dataset and evaluates VideoQA baselines and feature sampling strategies.
Results
Replacing fixed-stride sampling with dynamic sampling improves all baselines by at least 1%.
Takeaways & Limitations
ActivityNet-QA provides a large, human-annotated benchmark for studying VideoQA on long web videos and video feature representation.
Takeaways & Limitations
The studied models are baselines, leaving significant room for improved video features and more powerful spatio-temporal reasoning models.
Abstract
from arXiv · showhide
Recent developments in modeling language and vision have been successfully applied to image question answering. It is both crucial and natural to extend this research direction to the video domain for video question answering (VideoQA). Compared to the image domain where large scale and fully annotated benchmark datasets exists, VideoQA datasets are limited to small scale and are automatically generated, etc. These limitations restrict their applicability in practice. Here we introduce ActivityNet-QA, a fully annotated and large scale VideoQA dataset. The dataset consists of 58,000 QA pairs on 5,800 complex web videos derived from the popular ActivityNet dataset. We present a statistical analysis of our ActivityNet-QA dataset and conduct extensive experiments on it by comparing existing VideoQA baselines. Moreover, we explore various video representation strategies to improve VideoQA performance, especially for long videos. The dataset is available at https://github.com/MILVLG/activitynet-qa
Introduction
VideoQA extends visual question answering to image sequences, requiring fine-grained content understanding and spatio-temporal reasoning. ActivityNet-QA addresses limitations of existing datasets with a large, human-annotated benchmark built from long web videos.
- VideoQA extends ImageQA by requiring models to understand image sequences and answer more diverse questions.
- Accurate VideoQA requires fine-grained video understanding together with spatio-temporal reasoning over visual content.
- Existing VideoQA datasets are limited by small scale, automatic annotations, short videos, and restricted video representation.
- 58,000 QA pairs were collected from 5,800 ActivityNet videos, which are long, complex web videos representing 200 action classes.
- The paper analyzes ActivityNet-QA and compares results from existing VideoQA baselines.
ActivityNet-QA Dataset
ActivityNet-QA is constructed from sampled ActivityNet videos with crowdsourced questions spanning controlled and free-form types. The dataset includes quality controls, bilingual alignment, statistical analysis, and video-feature representation procedures.
- Video collection: The dataset samples 5,800 videos from ActivityNet’s 20,000 videos while considering split allocation, class diversity, and balance.
- QA generation: Crowdsourcing is used to generate QA pairs rather than relying on automatically extracted captions or narrative descriptions.
- QA generation: The controlled question types assess coarse temporal action understanding, spatial reasoning, and temporal relationships between objects.
- QA generation: Free-type answers are categorized into Yes/No, Number, Color, Object, Location, and Other classes.
- QA quality control: Questions and answers are quality-controlled through length limits, separate question and answer annotators, double-checking, and majority voting.
- Statistical analysis: Average question and answer lengths are 8.67 and 1.85, respectively, with maximum English lengths of 20 and 5 words.
- Video representation: Video feature representation proceeds through video-unit generation, sampling, and unit feature extraction.
- QA generation: Each video receives ten QA pairs: one motion, one spatial relationship, one temporal relationship, and seven free-type questions.
Methods
The paper evaluates ActivityNet-QA with baseline VideoQA models while varying long-video feature representation through sampling and two-stream fusion strategies.
- Baseline evaluation: The experiments assess dataset difficulty using VideoQA baselines paired with different video feature types.The paper explicitly focuses on ActivityNet-QA and factors influencing performance rather than complex-model comparisons.
- Video sampling: Long videos are divided into non-overlapping units of 16 consecutive frames, averaging 270 units per video.Because this sequence is too large and variable for existing models, the authors sample a fixed number T of units.
- Video sampling: Fixed Stepsize and Dynamic Stepsize sample a fixed number of video units, with Dynamic Stepsize favoring units likely to contain meaningful actions.Dynamic Stepsize uses an external temporal action proposal model to build action scores over units before sampling.
- Feature extraction: Appearance and motion features are extracted from each sampled unit using VGG-16 on its central frame and C3D on all 16 frames.Both networks use pretrained weights: ImageNet for VGG-16 and Sport-1M for C3D.
- Feature fusion: Mean Pooling, Concat, and Eltwise Product fuse the two streams into 4096-D, 8192-D, and 4096-D features, followed by L2 normalization.These fused representations can be combined flexibly with the end-to-end baselines because the baselines are decoupled from video features.
- Baseline architectures: E-VQA, E-MN, and E-SA encode questions and video information with LSTM-based, memory-network, and soft-attention architectures for answer prediction.E-VQA fuses question and video features by element-wise multiplication; E-MN performs multiple inference steps over memory; E-SA attends to video features using the encoded question.
Experiments
Experiments evaluate ActivityNet-QA baselines with standard VideoQA metrics and investigate video-feature sampling and fusion strategies. Dynamic sampling improves baseline accuracy, while long-video temporal reasoning remains difficult and overall performance is unsatisfactory.
- Experimental Setup: ActivityNet-QA models are trained on 3,200 videos and 32,000 QA pairs, tuned on 1,800 videos and 18,000 QA pairs, and tested on 800 videos and 8,000 QA pairs.All models use TensorFlow, Adam optimization, early stopping, and pretrained 300-dimensional GloVe question embeddings.
- Evaluation Criteria: The evaluation uses accuracy and WUPS, with WUPS@0.0 and WUPS@0.9 computed using thresholds γ = 0.0 and γ = 0.9.Accuracy counts identical predicted and ground-truth answers, whereas WUPS accounts for word-level answer ambiguities.
- Baseline Results: E-SA slightly outperforms E-VQA and E-MN on both accuracy and WUPS, but overall performance remains far from satisfactory.Temporal-relationship questions have lower accuracy than other question types, indicating that long-video temporal reasoning remains difficult for the baselines.
- Video Feature Representation: All baselines improve by at least 1% when fixed-step sampling is replaced with dynamic-step sampling.The authors attribute this to dynamic sampling better capturing representative and diverse key-frames, especially in long videos.
- Video Feature Representation: Mean Pooling achieves the best accuracy and robustness among the three fusion strategies evaluated for E-SA with FS sampling.The comparison examines sampling frequencies T={20, 40, 60}.
- Qualitative Analysis: The methods more often answer static-frame questions correctly than questions requiring temporal reasoning.Qualitative examples separate cases where at least one method succeeds from cases where all methods fail.
Conclusion and Future Work
ActivityNet-QA is a large-scale benchmark for complex web-video question answering, distinguished by long ActivityNet videos, crowdsourced annotations, and bilingual alignment. The baseline analyses leave substantial room for stronger video representations and VideoQA models, while auxiliary ActivityNet information may improve fine-grained understanding.
- Future Work: The evaluated models are baselines, leaving significant room for improved discriminative video representations and more powerful spatio-temporal reasoning models.The paper specifically identifies better visual feature learning and more accurate spatio-temporal reasoning as improvement directions.
- Future Work: Auxiliary ActivityNet information, such as dense captions, could help models better understand fine-grained video semantics.This proposed future direction connects additional annotations with fine-grained semantic understanding.