Source-linked AI summary
Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning
Xin Wang, Yuan-Fang Wang, William Yang Wang
TL;DR
Video captioning must combine audio and visual cues, yet selective cross-modal fusion and temporal alignment across granularities remain underexplored. HACA aligns and fuses global and local modality contexts; on MSR-VTT, it reports new state-of-the-art results on several captioning metrics.
Problem
Video captioning needs audio and visual cues, but selective cross-modal attention and alignment of modality transitions at different granularities remain underexplored.
Source: §1 · p. 1Method
HACA uses hierarchical attentive encoders and separate global and local cross-modal decoders to align and fuse multimodal contexts.
Source: §1 · p. 2, §2 · p. 2, §2.4 · p. 3Results
On MSR-VTT, HACA raises BLEU-4 from 41.4 to 43.1 and achieves new state-of-the-art results on BLEU-4, METEOR, and ROUGE-L.
Source: §1 · p. 2, §4.1 · p. 5Takeaways & Limitations
HACA can be plugged into existing reinforcement-learning captioning methods, and the framework can incorporate modalities such as optical flow and C3D features.
Source: §5 · p. 5
Abstract
from arXiv · showhide
A major challenge for video captioning is to combine audio and visual cues. Existing multi-modal fusion methods have shown encouraging results in video understanding. However, the temporal structures of multiple modalities at different granularities are rarely explored, and how to selectively fuse the multi-modal representations at different levels of details remains uncharted. In this paper, we propose a novel hierarchically aligned cross-modal attention (HACA) framework to learn and selectively fuse both global and local temporal dynamics of different modalities. Furthermore, for the first time, we validate the superior performance of the deep audio features on the video captioning task. Finally, our HACA model significantly outperforms the previous best systems and achieves new state-of-the-art results on the widely used MSR-VTT dataset.
1 Introduction
Video captioning benefits from audio as well as visual cues, but prior fusion methods do not selectively attend across modalities or capture their temporal transitions at different granularities. HACA aligns and fuses global and local multimodal contexts, and the paper also tests deep audio features for captioning.
- Audio can distinguish video content, such as singing from talking, when visual features alone are insufficient for an accurate caption.The paper argues that understanding the audio signature is necessary for accurate description.Source: §1 · p. 1
- Prior fusion methods lack selective cross-modal attention, while different modalities’ temporal transitions at varying granularities remain underexplored.The paper motivates aligning both global and local transitions.Source: §1 · p. 1
- HACA uses hierarchical encoders and attentive representations to align and dynamically fuse global and local contexts across modalities.Its attention includes visual, audio, and decoder attention.Source: §1 · p. 2
- The paper is the first to use deep audio features for video captioning and reports their effectiveness over hand-crafted MFCC features.Source: §1 · pp. 1–2
2 Proposed Model
HACA uses hierarchical recurrent encoders to represent each modality at global and local temporal granularity, then aligns and fuses those representations through separate attentive decoders.
2 Proposed Model
- HACA encodes each input modality with a hierarchical attentive encoder that learns both local and global representations.The framework uses one encoder per modality and can extend beyond the visual and audio inputs shown.Source: §2 · p. 2
2.1 Feature Extractors
- ResNet extracts deep visual features, while VGGish extracts deep audio features for the model.Both are pretrained CNN models.Source: §2.1 · p. 3
2.2 Attention Mechanism
- Soft attention computes a context vector at each step as a weighted sum over the input feature sequence.The attention weights can be learned to emphasize features useful for predicting the system’s internal state.Source: §2.2 · p. 3
2.3 Hierarchical Attentive Encoder
- The hierarchical encoder uses low-level and high-level LSTMs, with the high-level LSTM processing feature chunks at a lower temporal resolution.Attention summarizes low-level outputs within the current chunk before they are passed to the high-level LSTM; separate visual and audio encoders produce high- and low-level feature sequences.Source: §2.3 · p. 3
2.4 Globally and Locally Aligned Cross-modal Attentive Decoder
- Separate global and local decoders align modality representations at matching granularities and fuse their contexts for word prediction.The global decoder aligns long-range, high-level contexts; the local decoder combines fine-grained context with the global decoder’s output. Cross-modal attention selectively weights modality contexts, while decoder attention attends to prior hidden states.Source: §2 · p. 2, §2.4 · p. 3
2.5 Cross-Entropy Loss Function
- The model’s training objective is a cross-entropy loss over the generated and ground-truth word sequences.The passage identifies the model parameters and ground-truth sequence in introducing the loss.Source: §2.5 · p. 4
3 Experimental Setup
Experiments use MSR-VTT, with ResNet visual features and VGGish audio features, and evaluate captions using four automatic metrics.
Dataset and Preprocessing
- MSR-VTT contains 10,000 clips, split into 6,513 training, 497 validation, and 2,990 test videos, each with 20 human-annotated captions.Frames are sampled at 3 fps for ResNet features, and pretrained VGGish processes the raw WAV audio.Source: §3 · p. 4
Evaluation Metrics
- Caption quality is evaluated with BLEU, METEOR, ROUGE-L, and CIDEr-D using the standard MS-COCO evaluation code.Source: §3 · p. 4
Training Details
- Hyperparameters are tuned on validation data, with training capped at 50 epochs and test-time beam search of size 5.The setup also specifies modality-specific encoder dimensions and chunk sizes, dropout of 0.5, and batch size 64.Source: §3 · p. 4
4 Results
HACA outperforms prior MSR-VTT systems on BLEU-4, METEOR, and ROUGE-L, while component analyses support deep audio features, decoder attention, and hierarchical context alignment. Its validation CIDEr curves also show stronger performance than the compared models.
4.1 Comparison with State Of The Arts
- HACA sets new state-of-the-art results on BLEU-4, METEOR, and ROUGE-L, raising BLEU-4 from 41.4 to 43.1 with supervised learning only.Its CIDEr score is second best, below CIDEnt-RL, which directly optimizes CIDEr with reinforcement learning.Source: §4.1 · pp. 4–5
4.2 Result Analysis
- HACA outperforms HACA(w/o align), supporting the effectiveness of aligning global and local contexts.ATT(v) is comparable to top-ranked challenge results; CM-ATT(va) improves over it, and adding decoder attention further benefits captioning.Source: §4.2 · p. 5
4.3 Effect of Deep Audio Features
- Deep VGGish audio features outperform hand-crafted MFCC features in the cross-modal attention model.Table 2 compares performance across audio-feature choices.Source: §4.3 · p. 5, Table 2 · p. 5
4.4 Learning Curves
- HACA consistently outperforms HACA(w/o align) and CM-ATT on validation CIDEr curves, with the largest reported model capacity.The curves use greedy decoding during training; test scores are higher because beam search is used at test time.Source: §4.4 · p. 5, Figure 3 · p. 5
5 Conclusion
The conclusion presents HACA as a generic architecture for globally and locally aligned cross-modal attention, with stated extensions to reinforcement-learning methods and additional modalities.
5 Conclusion
- HACA is a generic video-captioning architecture that learns globally and locally aligned cross-modal attention.Source: §5 · p. 5
- The architecture can be plugged into existing reinforcement-learning methods for video captioning to further boost performance.Source: §5 · p. 5
- Optical-flow and C3D features are among the additional modalities that can be incorporated into HACA alongside deep visual and audio features.Source: §5 · p. 5