Source-linked AI summary
Video Question Answering via Attribute-Augmented Attention Network Learning
Yunan Ye, Zhou Zhao, Yimeng Li, Long Chen, Jun Xiao, Yueting Zhuang
TL;DR
Image-focused visual question answering may not model video temporal dynamics adequately. The paper proposes attribute-augmented attention with multi-step reasoning, and reports that reasoning further improves performance.
Problem
Image question-answering approaches may be ineffective for video because they do not sufficiently model temporal dynamics.
Source: §1 · p. 1Method
The framework jointly detects frame-level attributes and learns a unified video representation, with multi-step reasoning for video question answering.
Source: §1 · pp. 1–2Results
Multi-step reasoning is reported to further improve the proposed attention network's video-question-answering performance.
Source: §1 · p. 1Takeaways & Limitations
The paper constructs a large-scale video-question-answering dataset and evaluates both multiple-choice and open-ended tasks.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Video Question Answering is a challenging problem in visual information retrieval, which provides the answer to the referenced video content according to the question. However, the existing visual question answering approaches mainly tackle the problem of static image question, which may be ineffectively for video question answering due to the insufficiency of modeling the temporal dynamics of video contents. In this paper, we study the problem of video question answering by modeling its temporal dynamics with frame-level attention mechanism. We propose the attribute-augmented attention network learning framework that enables the joint frame-level attribute detection and unified video representation learning for video question answering. We then incorporate the multi-step reasoning process for our proposed attention network to further improve the performance. We construct a large-scale video question answering dataset. We conduct the experiments on both multiple-choice and open-ended video question answering tasks to show the effectiveness of the proposed method.
1 INTRODUCTION
The introduction argues that image question-answering approaches may not transfer effectively to video, where answering can depend on temporal dynamics. It presents an attribute-augmented attention framework, multi-step reasoning, and evaluation on two video question-answering task formats.
1 INTRODUCTION
- Existing visual question-answering methods mainly address static images, while their lack of temporal-dynamics modeling may limit their effectiveness on video [17].Video information may be scattered across frames, with some frames redundant or irrelevant to the question.Source: §1 · p. 1
- The proposed framework jointly detects frame-level attributes and learns a unified video representation, then adds multi-step reasoning to further improve performance.The authors also describe the approach as modeling temporal dynamics and semantic attributes.Source: §1 · pp. 1–2
- The paper constructs a large-scale video question-answering dataset and evaluates its method on multiple-choice and open-ended tasks.Source: §1 · p. 2
2 VIDEO QUESTION ANSWERING VIA ATTENTION NETWORK LEARNING
The section formulates video question answering as learning an attribute-augmented attention network that returns an answer relevant to a referenced video. It builds question-conditioned temporal representations from frame features and detected attributes, then refines them through recursive reasoning.
2.1 Problem Formulation
- Given a video, question, and attributes, the goal is to learn a network that returns the answer relevant to the referenced video.Source: §2.1 · p. 2
2.2 Attribute-Augmented Attention Network Learning
- Pretrained detectors supply frame attributes, which are embedded and element-wise multiplied with each bidirectional-LSTM frame state to form joint attribute/frame representations.The detector use is attributed to [9].Source: §2.2 · p. 2
- Question-conditioned temporal attention scores each attribute-augmented frame, and softmax weights are summed to form a question-oriented video representation.Source: §2.2 · pp. 2–3
- The multi-step reasoning process recursively updates the representation, returning the joint question-oriented video representation after the R-th update.Source: §2.2 · p. 3
- For multiple-choice tasks, a softmax classifies the updated representation; for open-ended tasks, an LSTM decoder generates freeform answers.Source: §2.2 · p. 3
- Training optimizes an objective with a trade-off between training loss and regularization using SGD with diagonal Ada-Grad.Source: §2.2 · p. 3
3 EXPERIMENTS
The experiments use a generated video question-answering dataset and evaluate methods on open-ended and multiple-choice tasks. The section describes the data preparation and comparison setup, but the supplied passages do not provide the results themselves.
3.1 Data Preparation
- The dataset is generated from YouTube2Text descriptions and contains 1,987 videos and 122,708 descriptions.Generated questions cover what, who, and other queries; the data are split into training, validation, and testing sets.Source: §3.1 · p. 3
- Each video is represented by 40 sampled frames, with 2,048-dimensional ResNet features; question and answer semantics use 256-dimensional word2vec vectors.Open-ended answers use an <eos> token, with <Unk> for out-of-vocabulary words.Source: §3.1 · p. 3
3.2 Performance Comparisons
- The comparisons extend image question-answering methods to video and evaluate attribute removal and different reasoning-step configurations of r-ANL.Accuracy is used for both task formats, and the best validation-set hyperparameters and parameters are used for testing.Source: §3.2 · pp. 3–4
- Table 2 reports overall experimental results for open-ended and multiple-choice video question answering, across different question types.The supplied passages do not include the table’s cell values.Source: §3.2 · p. 4, Table 2 · p. 4
4 CONCLUSION
The conclusion presents attribute-augmented attention as an approach to video question answering, combining frame-and-text representation learning with question-oriented attention and multi-step reasoning. It also reports constructing a dataset and evaluating the method experimentally.
4 CONCLUSION
- The method learns joint representations of visual frames and textual attributes, then uses attention to form question-oriented video representations for answering.The conclusion presents these as successive components of the proposed approach.Source: §4 · p. 4
- Multi-step reasoning is incorporated into the attention network to further improve the method’s performance.Source: §4 · p. 4
- The paper constructs a large-scale video question-answering dataset and evaluates the proposed method through extensive experiments.Source: §4 · p. 4