Source-linked AI summary
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, Heng Huang
TL;DR
VideoQA requires reasoning over complex questions and temporally distributed visual evidence, while existing methods can struggle with feature integration and global question semantics. The paper introduces heterogeneous visual memory, question memory, and iterative multimodal fusion, achieving consistent outperformance of state-of-the-art methods on four benchmark datasets.
Problem
VideoQA must connect complex question semantics with relevant video content, but existing methods can separate feature integration from attention learning and rely on local question semantics.
Method
The model uses heterogeneous memory for appearance and motion, question memory for global context-aware text representations, and multimodal fusion for iterative multi-step reasoning.
Results
The approach consistently outperforms state-of-the-art methods on four VideoQA benchmark datasets.
Takeaways & Limitations
The framework jointly models global video context, complex question semantics, and visual-textual interactions through progressively refined attention.
Abstract
from arXiv · showhide
In this paper, we propose a novel end-to-end trainable Video Question Answering (VideoQA) framework with three major components: 1) a new heterogeneous memory which can effectively learn global context information from appearance and motion features; 2) a redesigned question memory which helps understand the complex semantics of question and highlights queried subjects; and 3) a new multimodal fusion layer which performs multi-step reasoning by attending to relevant visual and textual hints with self-updated attention. Our VideoQA model firstly generates the global context-aware visual and textual features respectively by interacting current inputs with memory contents. After that, it makes the attentional fusion of the multimodal visual and textual representations to infer the correct answer. Multiple cycles of reasoning can be made to iteratively refine attention weights of the multimodal data and improve the final representation of the QA pair. Experimental results demonstrate our approach achieves state-of-the-art performance on four VideoQA benchmark datasets.
1. Introduction
VideoQA requires linking complex question semantics to relevant visual content across video frames. The proposed framework addresses this with heterogeneous visual memory, context-aware question memory, and iterative multimodal fusion.
- 1. Introduction: VideoQA must associate relevant visual contents in frame sequences with the subject queried by a complex question.The task requires understanding the question before identifying and localizing the corresponding visual evidence.
- 1. Introduction: Existing approaches encode video frames and question words, then use temporal or spatial attention to associate visual contents with the question.Temporal attention targets relevant events, while spatial attention targets details and queried subjects within frames.
- 1. Introduction: Separating feature integration from attention learning can produce incorrect attention when appearance and motion features are combined.The paper identifies straightforward early fusion and separated integration steps as limitations of existing approaches.
- 1. Introduction: The heterogeneous memory integrates appearance and motion features while co-learning spatiotemporal attention through attentional memory operations.Its multi-input visual features interact within memory to improve video understanding.
- 1. Introduction: The question memory learns global context-aware text representations to avoid being trapped by irrelevant local semantics in complex questions.It stores sequential text information, learns relevance between words, and highlights queried subjects.
- 1. Introduction: The multimodal fusion layer iteratively refines joint visual-question attention and combines both modalities with learned soft weights for multi-step reasoning.The framework supports repeated reasoning cycles before inferring the answer.
2. Related Work
VideoQA extends visual question answering to temporally structured video, where contextual information and motion-appearance integration complicate reasoning. The paper proposes memory-based multimodal fusion to address these challenges.
- 2. Related Work: Visual Question Answering reasons about answers to questions related to image content, using attention over regions, facts, and question words.Prior methods include stack attention, object-proposal alignment, fact-based attention, co-attention, and external memory.
- 2. Related Work: VideoQA extends VQA to video clips and requires memorizing contextual information across the temporal scale.Existing methods commonly use both motion and appearance features to represent video content.
- 2. Related Work: The proposed model differs by combining motion and appearance through heterogeneous external memory with attentional read and write operations.It also constructs global context-aware visual and question features through memory interactions.
- 2. Related Work: The multimodal fusion layer combines visual and question features with soft attentional weights and supports multi-step reasoning.This design is presented as a distinguishing feature from existing VideoQA work.
3. Our Approach
The model encodes video and questions, then uses heterogeneous and question memories to produce global context-aware features for multimodal, iterative answer reasoning.
- 3.1. Video and text representation: The architecture encodes appearance, motion, and question features with separate LSTM-based encoders before memory-based processing.Appearance and motion features are encoded individually, while question word embeddings are processed by another LSTM encoder.
- 3.2. Heterogeneous video memory: Heterogeneous memory integrates motion and appearance features through attentional write operations that learn joint attention and memorize spatiotemporal patterns.Separate hidden states determine modality-specific contents, while a global hidden state stores integrated context-aware features.
- 3.2. Heterogeneous video memory: The memory performs attentional reads from memory slots and updates motion, appearance, and global hidden states to produce final video features.The global hidden state across video time steps is used as the context-aware video representation.
- 3.3. External question memory: Question memory persistently stores encoded text, interacts current inputs with memory contents, and generates global context-aware question features.Its slot-writing and reading operations update a single question-memory hidden state across word positions.
- 3.4. Multimodal fusion and reasoning: The multimodal fusion layer uses an LSTM controller to attend to visual and question features, combine modality-weighted contents, and update its state iteratively.The resulting controller state is combined with temporal video attention to form the final answer representation.
4. Experiments and Discussions
Experiments evaluate the model across four VideoQA benchmarks, where it generally surpasses prior methods across multiple question types and settings. Attention visualizations and ablations further examine how the model’s memories and iterative reasoning contribute to performance.
- Benchmark evaluation: The evaluation covers TGIF-QA, MSVD-QA, MSRVTT-QA, and YouTube2Text-QA using dataset-specific features, splits, question types, and answer formats.The datasets include counting, action, transition, FrameQA, open-ended, and multiple-choice tasks.
- Benchmark results: 4.02 average L2 loss is achieved on TGIF-QA counting, while accuracies reach 0.739 for Action, 0.778 for Trans., and 0.538 for FrameQA.These results improve on the listed ST-VQA and Co-Mem comparisons.
- Benchmark results: 0.337 overall accuracy is achieved on MSVD-QA, 5.3% better than the prior best 0.320, despite slight underperformance on How, When, and Where questions.The paper attributes the limited overall effect of these categories to their small absolute number of instances under class imbalance.
- Benchmark results: 0.330 overall accuracy is achieved on MSRVTT-QA, with the model outperforming competing models on what, who, and how question types.The comparison includes ST-VQA, Co-Mem, and AMU.
- Benchmark results: On YouTube2Text-QA, overall accuracy reaches 0.808 versus 0.520 for multiple-choice questions and 0.301 versus 0.262 for open-ended questions.The model does not use the additional frame-level attribute supervision used by r-ANL.
5. Conclusion
The proposed end-to-end VideoQA framework uses external memories and multimodal fusion to capture global contexts, complex question semantics, and their interactions. Across four benchmark datasets, it consistently outperforms state-of-the-art methods.
- The framework introduces external memory modules to capture global contexts in video frames, complex question semantics, and their interactions.
- Its multimodal fusion layer performs multi-step reasoning with gradually refined attention over visual and textual modalities.
- Across four benchmark VideoQA datasets, the approach consistently outperforms state-of-the-art methods.