Source-linked AI summary
Memory-Attended Recurrent Network for Video Captioning
Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, Yu-Wing Tai
TL;DR
Video captioning models typically process only one source video at a time, limiting their access to recurring words’ varied visual contexts across training data. MARN adds a memory-based decoder that captures these correspondences and explicitly models adjacent-word compatibility, achieving the best reported results on two datasets.
Problem
Encoder-decoder video-captioning models can attend only to the currently processed video, limiting their ability to capture a word’s varied visual contexts across training videos.
Method
MARN combines an attention-based recurrent decoder with a memory storing each vocabulary word’s visual context, word embedding, and auxiliary features.
Results
MARN achieves the best reported results among state-of-the-art methods on the MSR-VTT and MSVD video-captioning datasets.
Takeaways & Limitations
The memory lets MARN use visual contexts from multiple training videos and explicitly model compatibility between adjacent words during decoding.
Abstract
from arXiv · showhide
Typical techniques for video captioning follow the encoder-decoder framework, which can only focus on one source video being processed. A potential disadvantage of such design is that it cannot capture the multiple visual context information of a word appearing in more than one relevant videos in training data. To tackle this limitation, we propose the Memory-Attended Recurrent Network (MARN) for video captioning, in which a memory structure is designed to explore the full-spectrum correspondence between a word and its various similar visual contexts across videos in training data. Thus, our model is able to achieve a more comprehensive understanding for each word and yield higher captioning quality. Furthermore, the built memory structure enables our method to model the compatibility between adjacent words explicitly instead of asking the model to learn implicitly, as most existing models do. Extensive validation on two real-word datasets demonstrates that our MARN consistently outperforms state-of-the-art methods.
1. Introduction
Video captioning generates natural-language descriptions of video content, but conventional encoder-decoder models attend only to the currently processed video. MARN addresses this limitation with memory over words’ visual contexts across training videos and explicitly models adjacent-word compatibility.
- Motivation: Video captioning generates natural-language sequences describing visual content while requiring understanding of temporal dynamics.The task supports applications including visual question answering, video retrieval, and assistance for visually impaired people.
- Existing approach: Encoder-decoder methods use an encoder to extract visual features and a decoder to generate captions sequentially, with attention selecting relevant content.The decoder’s attention is limited to the single source video currently being processed.
- Limitation: A word may occur across multiple similar but nonidentical video scenes, yet existing decoders cannot explore its full visual correspondence across training data.This limitation restricts the contextual information available for candidate words such as “pouring.”
- Proposed method: MARN uses a memory structure to store descriptive information for vocabulary words and their relevant visual contexts across training videos.The memory is built on top of an attention-based recurrent decoder and supports a more comprehensive understanding of each word.
- Proposed method: MARN explicitly models compatibility between adjacent words rather than learning that compatibility implicitly through next-word prediction.The model was evaluated on the MSR-VTT and MSVD datasets and achieved the best results compared with other state-of-the-art methods.
2. Related Work
Prior video-captioning work progressed from templates to encoder-decoder systems and attention-based improvements, but generally remained limited to one source video. MARN adds a memory mechanism during decoding to incorporate visual contexts associated with candidate words across training data.
- Video Captioning: Traditional video-captioning methods use templates, word roles, grammar rules, and hierarchical structures to generate captions.Examples model source-video components with conditional random fields or semantic relationships between actions and visual features.
- Video Captioning: MP-LSTM and S2VT established encoder-decoder video-captioning benchmarks, while attention mechanisms subsequently produced significant performance gains.MP-LSTM uses CNNs for visual encoding and LSTM decoding; S2VT shares an LSTM across encoding and decoding.
- Video Captioning: Recent methods improve either video encoding or caption decoding through multimodal fusion, frame selection, spatial-temporal representations, or decoding refinements.The cited examples include VideoLAB, PickNet, and TSA-ED.
- Video Captioning: Most existing encoder-decoder methods still attend only to the video being processed and therefore miss candidate words’ varied visual contexts across training data.MARN mitigates this limitation by incorporating memory into the decoding phase.
- Memory-based Models: MARN is an assistant memory module that supports the basis captioning model rather than serving as the dominant task module.The paper positions it as the first use of a memory network in visual captioning, according to the authors.
3. Memory-Attended Recurrent Network
MARN combines video feature encoding, a basis recurrent captioning decoder, and an attended memory decoder. Its memory aggregates word-specific visual contexts across related videos, while attention mechanisms combine 2D and 3D information and regularize temporal attention.
- MARN consists of an encoder, an Attention-based Recurrent Decoder, and an Attended Memory Decoder.The encoder extracts source-video features; the recurrent decoder generates captions, and the memory decoder assists caption generation.
- Encoder: The encoder extracts separate 2D frame features and 3D temporal features before downstream decoding.2D features are obtained for sampled frames, while 3D features are extracted over every 16 frames.
- Attention-based Recurrent Decoder: The basis decoder predicts words sequentially with a GRU and temporal attention over both 2D and 3D visual features.The two feature types are attended separately and their context representations are concatenated for decoding.
- Attention-based Recurrent Decoder: Sharing the attention function across 2D and 3D features projects them toward a similar feature space and acts as regularization against overfitting.The model treats the feature types separately during encoding and fuses their hidden representations during decoding rather than concatenating them early.
- Memory Structure: The memory stores each vocabulary word’s visual contexts, word embedding, and optional auxiliary features, retaining top-k relevant features from each related video.This construction captures visual contexts from multiple scenes in which the same word appears while reducing feature redundancy.
4. Experiments
Experiments evaluate MARN on MSR-VTT and MSVD through ablations, qualitative comparisons, human judgments, and comparisons with existing methods. The results show benefits from attended memory and attention-coherent loss, with MARN achieving strong benchmark performance.
- Experimental Setup: Experiments use MSR-VTT and MSVD, with standard splits, filtered training-set vocabularies, and CIDEr, METEOR, ROUGE-L, and BLEU evaluation.MSR-VTT contains 10,000 clips and MSVD contains 1,970 clips; the vocabulary contains approximately 11K and 4K words, respectively.
- Ablation Study: The Attended Memory Decoder improves CIDEr by 1.1% on MSR-VTT and 1.8% on MSVD over the basis decoder.The ablation compares the sole attention-based recurrent decoder with the integrated system containing the Attended Memory Decoder.
- Ablation Study: Word embeddings and visual contexts provide the major memory gains, while auxiliary category information adds a smaller gain on MSR-VTT.Visual context supplies full-spectrum context and source-video matching, whereas word embeddings measure compatibility with the previously predicted word.
- Ablation Study: Attention-Coherent Loss raises CIDEr from 46.8 to 47.1 on MSR-VTT and from 91.7 to 92.2 on MSVD.The improvement is described as a small margin on both datasets.
- Comparison on MSR-VTT: On MSR-VTT, MARN achieves the best METEOR, ROUGE-L, and CIDEr results while ranking second on BLEU-4.The basis model performs comparably to state-of-the-art methods, suggesting a performance ceiling for encoder-decoder models using attention alone.
- Comparison on MSVD: On MSVD, MARN performs significantly better than other methods on every reported metric except BLEU-4, where PickNet and RecNet achieve the best result.The basis decoder also outperforms other methods substantially, and the Attended Memory Decoder further boosts performance.
5. Conclusion
MARN combines an attention-based recurrent decoder with a memory-based decoder that captures each candidate word’s visual contexts across training videos. The model achieves superior quantitative and qualitative performance on two real-world datasets.
- MARN uses an attention-based recurrent network as its basis caption decoder and a memory-based decoder to assist decoding.
- Its memory captures the full-spectrum correspondence between candidate words and visual contexts across training videos.
- MARN generates more precise captions and shows superior quantitative and qualitative performance on two real-world datasets.