Source-linked AI summary
Attention-Based Multimodal Fusion for Video Description
Chiori Hori, Takaaki Hori, Teng-Yok Lee, Kazuhiro Sumi, John R. Hershey, Tim K. Marks
TL;DR
Video description systems must generate sentences from asynchronous, heterogeneous video cues, while existing attention mechanisms primarily select temporal or spatial information. The paper introduces multimodal attention to select among image, motion, and audio features during decoding, and reports competitive Youtube2Text performance with gains over temporal attention alone. No supported passage specifies a broader limitation beyond audio sometimes containing unrelated background music.
Problem
Video description requires selecting relevant features when video events and sentence words are not synchronized, and different descriptive content may depend on different modalities.
Method
The paper extends encoder-decoder sentence generation with multimodal attention that selectively weights image, motion, and audio features according to the decoder state.
Results
Multimodal Attention consistently outperformed Simple Multimodal, while the full method achieved results competitive with current state-of-the-art temporal-attention systems on Youtube2Text.
Takeaways & Limitations
Multimodal attention provides a context-sensitive way to fuse video modalities for description generation.
Takeaways & Limitations
Audio features do not always help because some YouTube videos contain background music unrelated to video content.
Abstract
from arXiv · showhide
Currently successful methods for video description are based on encoder-decoder sentence generation using recur-rent neural networks (RNNs). Recent work has shown the advantage of integrating temporal and/or spatial attention mechanisms into these models, in which the decoder net-work predicts each word in the description by selectively giving more weight to encoded features from specific time frames (temporal attention) or to features from specific spatial regions (spatial attention). In this paper, we propose to expand the attention model to selectively attend not just to specific times or spatial regions, but to specific modalities of input such as image features, motion features, and audio features. Our new modality-dependent attention mechanism, which we call multimodal attention, provides a natural way to fuse multimodal information for video description. We evaluate our method on the Youtube2Text dataset, achieving results that are competitive with current state of the art. More importantly, we demonstrate that our model incorporating multimodal attention as well as temporal attention significantly outperforms the model that uses temporal attention alone.
1. Introduction and Related Work
Video description generates natural-language summaries from videos and has applications in retrieval, accessibility, and broader machine-intelligence research. The paper motivates multimodal attention because relevant visual, motion, and audio cues vary with the content being described.
- Task and motivation: Video description automatically generates a natural-language sentence summarizing an input video.The task is also called video captioning.
- Related work: Image captioning applies encoder-decoder generation to produce a natural-language description from a single image, with attention enabling focus on specific image parts.This work builds on attention-based captioning and extends attention to video modalities.
- Existing approaches: Encoder-decoder RNN systems generate descriptions from video features including appearance, motion, optical flow, trajectories, and audio.These systems can be trained end-to-end on videos paired with text descriptions.
- Attention motivation: Temporal misalignment between video features and sentence words makes irrelevant input features a source of clutter during word selection.Attention mechanisms help retrieve relevant features from corresponding input portions.
- Multimodal motivation: Different modalities can provide reliable cues at different times, but estimating each modality’s reliability for fusion remains challenging.Appearance, motion, and audio may support different descriptive content.
- Contribution: The paper proposes attention across modalities in addition to spatio-temporal attention, allowing end-to-end context-sensitive fusion for video description.The experiments report that adding multimodal attention to temporal attention significantly outperforms temporal attention alone.
2. Encoder-decoder-based sentence generator
The sentence generator encodes a sequence of image features and iteratively decodes a word sequence with an LSTM-based encoder-decoder. At test time, beam search retains multiple high-probability hypotheses before selecting a completed sentence.
- Architecture: The encoder-decoder architecture generates word sequences from image-sequence inputs using recurrent neural networks.Figure 1 presents an LSTM-based encoder-decoder example.
- Encoding: An LSTM encoder maps a sequence of extracted image features into hidden states for sentence generation.Features may come from pretrained CNNs such as GoogLeNet, VGGNet, or C3D; C3D can capture dynamic video features.
- Decoding: The decoder predicts the next word iteratively from its previous state, beginning with <sos> and ending with <eos>.The decoder uses an LSTM network to infer the next-word probability distribution.
- Training and inference: During training, the reference word sequence Y = y1, . . . , yM is provided to the generator.The training phase conditions learning on the reference description.
- Training and inference: During testing, beam search keeps multiple hypotheses with high cumulative probabilities at each step and selects one reaching the end-of-sentence token.This searches over possible word sequences rather than retaining only one partial hypothesis.
3. Attention-based sentence generator
The attention-based sentence generator uses temporal or spatial attention to emphasize input features relevant to predicting each next word. Its encoder processes feature sequences, while the decoder conditions word probabilities on an attention-derived content vector.
- Attention-based generation emphasizes features from specific times or spatial regions according to the current context, improving next-word prediction.
- The input sequence is formed from extracted features, which may be processed by a BLSTM, GRU, feed-forward layer, or used directly as CNN features.
- Attention weights emphasize hidden activation vectors at input time steps that are most important for predicting the next output word.
- For each output word, the relevant input content is represented as a weighted sum of hidden activation vectors.
- The decoder is an Attention-based Recurrent Sequence Generator with an LSTM whose output probabilities are conditioned on the attention-derived content vector.
- The attention weights are computed using parameter matrices, vectors, and a scalar score for each output-input position.
4. Attention-based multimodal fusion
The multimodal fusion method extends attention from temporal or spatial selection to selecting among multiple feature modalities. Decoder-state-dependent modality weights allow the system to vary the relative contribution of image, motion, audio, or other feature types during sentence generation.
- Video description can combine image, motion, and audio features, but effective fusion requires estimating each modality’s reliability.
- Simple feature fusion combines modality-specific content vectors with fixed weights that do not change with decoder context.
- Fixed fusion weights limit the ability to exploit multiple feature types because relative modality weights cannot change with context.
- Multimodal attention lets the decoder selectively attend to specific modalities or feature types based on its current state when predicting each word.
- The multimodal attention weights are computed similarly to temporal attention for K input-feature sequences.
- Unlike simple fusion, the proposed architecture changes feature-level attention weights according to decoder state and content vectors for successive output words.
5. Experiments
The experiments evaluate multimodal attention on YouTube2Text using multiple feature types, fusion strategies, and captioning metrics. Multimodal attention generally improves over simpler or unimodal alternatives, while audio benefits depend on data quality.
- Dataset: The YouTube2Text corpus contains 1,970 video clips with 80,839 descriptions, averaging about 41 sentences per clip and 8 words per sentence.
- Features: The system incorporates image, spatiotemporal, and audio features, with audio available for 1,649 clips, covering 84% of the corpus.
- Compared methods: The evaluation compares temporal-attention unimodal models, simple multimodal fusion, proposed multimodal attention, and prior temporal-attention systems.
- Metrics: Caption quality is evaluated with BLEU, METEOR, and CIDEr; CIDEr is described as more robust to incorrect ground-truth descriptions than BLEU or METEOR.
- Results: Simple Multimodal outperformed Unimodal models, while Multimodal Attention consistently outperformed Simple Multimodal.
- Results: Audio improved some baselines and produced the best BLEU with multimodal attention, but multimodal attention without audio produced the best CIDEr.Background music and other unrelated audio can limit the benefit of the audio feature.
6. Conclusion
The paper introduces multimodal attention for encoder-decoder video description. The mechanism attends to input modalities as well as time and outperforms temporal attention alone on YouTube2Text.
- The proposed multimodal attention mechanism selectively attends to image, motion, and audio modalities during recurrent encoder-decoder sentence generation.
- The method provides a way to fuse multimodal information while retaining temporal attention over video features.
- On YouTube2Text, the approach is competitive with temporal-attention state-of-the-art methods and outperforms temporal attention alone.