Source-linked AI summary
Motion-Appearance Co-Memory Networks for Video Question Answering
Jiyang Gao, Runzhou Ge, Kan Chen, Ram Nevatia
TL;DR
Video QA requires reasoning over long, varied image sequences in which motion and appearance interact, while different questions may require different numbers of frames. The paper proposes a motion-appearance co-memory network with co-memory attention, multi-level contextual facts, and dynamic fact ensembles. On TGIF-QA, it significantly outperforms state-of-the-art methods across all four tasks.
Problem
Video QA requires temporal reasoning over image sequences and joint reasoning of videos and text, with motion, appearance, and question-dependent temporal extent complicating representation.
Method
The motion-appearance co-memory network uses joint motion-appearance attention, temporal conv-deconv contextual facts, and dynamic fact ensemble mechanisms built on DMN concepts.
Results
The model significantly outperforms state-of-the-art performance on all four TGIF-QA tasks.
Takeaways & Limitations
The paper provides a video QA architecture that jointly models motion and appearance while dynamically constructing temporal representations for different questions.
Takeaways & Limitations
The evaluation reports an evaluation mistake in the prior work used for the TGIF-QA count-task comparison, with updated performances listed by its authors.
Abstract
from arXiv · showhide
Video Question Answering (QA) is an important task in understanding video temporal structure. We observe that there are three unique attributes of video QA compared with image QA: (1) it deals with long sequences of images containing richer information not only in quantity but also in variety; (2) motion and appearance information are usually correlated with each other and able to provide useful attention cues to the other; (3) different questions require different number of frames to infer the answer. Based these observations, we propose a motion-appearance comemory network for video QA. Our networks are built on concepts from Dynamic Memory Network (DMN) and introduces new mechanisms for video QA. Specifically, there are three salient aspects: (1) a co-memory attention mechanism that utilizes cues from both motion and appearance to generate attention; (2) a temporal conv-deconv network to generate multi-level contextual facts; (3) a dynamic fact ensemble method to construct temporal representation dynamically for different questions. We evaluate our method on TGIF-QA dataset, and the results outperform state-of-the-art significantly on all four tasks of TGIF-QA.
1. Introduction
Video QA requires temporal reasoning over varied visual information, where motion and appearance interact and questions may require different temporal extents. The proposed motion-appearance co-memory network addresses these properties with new temporal and memory mechanisms and outperforms prior methods on all four TGIF-QA tasks.
- Video QA jointly requires temporal reasoning over image sequences and reasoning between videos and text.
- Motion and appearance provide mutually useful attention cues, while different questions require representations from different numbers of frames.
- The proposed model significantly outperforms state-of-the-art performance on all four TGIF-QA tasks.
- The model uses co-memory attention to jointly model motion and appearance information.
- Temporal conv-deconv networks build multi-level contextual facts for video QA.
- Dynamic fact ensemble produces temporal facts dynamically during each cycle of fact encoding.
2. Related Work
Related work applies attention and memory mechanisms to image and video QA, while video methods additionally address temporal structure and motion. Existing approaches nonetheless leave gaps in motion analysis or dynamic memory updating that motivate this work.
- Image question answering: Image QA commonly uses attention to guide visual focus and memory to refine answers through multiple reasoning iterations.
- Video question answering: Video QA extends attention-based reasoning to temporal periods and combines video information with language for answering questions.
- Dynamic Memory Networks: The general DMN architecture repeatedly attends to facts, updates episodic memory with question information, and generates answers from the final memory.
- Video question answering: Some video QA methods use motion together with appearance, whereas memory-based methods cited here lack motion analysis and dynamic memory updating.
- Video temporal analysis: Video temporal analysis methods localize actions or activities using temporal boundaries and, in some cases, language queries.
3. General Dynamic Memory Networks
DMN represents inputs as facts, encodes questions as embeddings, and iteratively retrieves question-relevant information through attention and memory updates. Its final memory state is passed to an answer module to generate task-specific predictions.
- The DMN framework contains input, question, episodic memory, and answer modules.
- Fact module: The fact module converts input data into a sequence of vectors called facts.
- Question module: The question module encodes the question sentence with a GRU and uses its final hidden state as the question embedding q.
- Episodic memory module: The episodic memory module repeatedly attends to facts and updates memory across cycles to retrieve information relevant to the question.
- Episodic memory module: An attention-based GRU uses the attention gate instead of the original update gate to incorporate ordering and positional information in videos.
- Episodic memory module: The contextual feature c_t updates episodic memory m_t together with the question embedding q and the previous-cycle memory.
- Answer module: The final memory m_T is passed to the answer module, which combines it with q and may use task-specific decoders such as softmax.
4. Motion-Appearance Co-Memory Networks
The network builds multi-level temporal facts and uses co-memory attention plus dynamic fact ensemble to select question-relevant representations from motion and appearance streams. Separate memory updates then produce a final joint memory for answer decoding across TGIF-QA task types.
- Multi-level Contextual Facts: Two-stream motion and appearance features are processed into multi-level contextual facts with shared temporal resolution but different contextual coverage.Temporal convolution models context at multiple scales, while deconvolution recovers temporal resolution; the resulting fact set is built separately for motion and appearance.
- Motion-appearance Co-Memory Module: Co-memory attention uses motion and appearance memories jointly to generate separate attention gates for the two modalities.Each modality provides cues for the other during attention generation and memory updating.
- Motion-appearance Co-Memory Module: Dynamic fact ensemble combines attention over fact levels and temporal steps to construct question- and cycle-specific temporal facts.The method applies softmax over fact levels and uses the resulting attention scores in later fact encoding.
- Motion-appearance Co-Memory Module: Attention-based GRUs separately encode motion and appearance facts, update their memories, and concatenate the final memories for answer generation.The final output memory combines the motion and appearance memories after the specified number of update cycles.
- Answer Module: Separate decoders address TGIF-QA multiple-choice, repetition-count, and Frame QA settings using regression, integer prediction, or classification objectives.Repeating action and state transition use hinge-loss multiple-choice decoding; repetition count uses rounded integer regression with ℓ2 loss; Frame QA uses softmax classification with cross-entropy.
5. Evaluation
The evaluation uses TGIF-QA’s four video-question-answering tasks and tests co-memory attention, contextual facts, dynamic fact ensemble, memory-update cycles, and overall state-of-the-art performance. The full model significantly outperforms state-of-the-art methods across all four tasks.
- 5.1. Dataset: TGIF-QA contains 165k question-answer pairs from 71k animated Tumblr GIFs spanning repetition count, repetition action, state transition, and frame QA.The first three tasks require temporal reasoning; frame QA requires selecting the relevant frame from the video.
- 5.4. Experiments on TGIF-QA: The experiments evaluate co-memory attention, multi-level contextual facts, dynamic fact ensemble, memory-update cycles, and comparisons with prior methods.The study also compares fact concatenation and memory concatenation as direct motion-appearance baselines.
- 5.4. Experiments on TGIF-QA: Co-memory attention outperforms fact-concat and mem-concat on all four TGIF-QA tasks.The mechanism jointly uses motion and appearance information when generating attention.
- 5.4. Experiments on TGIF-QA: Dynamic fact ensemble provides better results than using a single temporal convolutional fact representation.Attention-based fusion forms a weighted average of multi-level contextual facts rather than selecting only one.
- 5.4. Experiments on TGIF-QA: Two memory-update cycles give the best performance on Action, Trans, and Count, while Frame has similar results for two and three cycles.In Trans, T = 2 improves performance over T = 1 by 3.3%, and performance begins to saturate at T = 3.
- 5.4. Experiments on TGIF-QA: The full co-memory model uses T = 2, dynamic fact ensemble, and three-layer contextual facts, and significantly outperforms state-of-the-art methods on all four tasks.The ensemble uses soft fusion of the three-layer contextual facts.
6. Conclusion
Video QA requires reasoning over long image sequences and jointly using motion and appearance information. The proposed motion-appearance co-memory network combines co-memory attention, multi-level contextual facts, and dynamic fact ensemble, and significantly outperforms state-of-the-art performance on TGIF-QA.
- 6. Conclusion: Video QA involves longer, more varied image sequences than image QA, with motion and appearance providing correlated attention cues.Different questions can also require different numbers of frames to infer an answer.
- 6. Conclusion: The model uses co-memory attention, a temporal conv-deconv network for multi-level contextual facts, and dynamic fact ensemble for question-dependent temporal representations.These mechanisms are designed specifically for video QA.
- 6. Conclusion: The method significantly outperforms state-of-the-art performance on all four TGIF-QA tasks.