Multimodal Memory Modelling for Video Captioning

Junbo Wang; Wei Wang; Yan Huang; Liang Wang; Tieniu Tan

Source-linked AI summary

Multimodal Memory Modelling for Video Captioning

Junbo Wang, Wei Wang, Yan Huang, Liang Wang, Tieniu Tan

arXiv:1611.05592v1Nov 17, 2016cs.CV

TL;DR

Video captioning must map visual sequences to language while handling long-term visual-textual dependencies. M3 introduces shared multimodal memory with repeated read/write interactions between video and text, and it outperforms state-of-the-art methods on MSVD and MSR-VTT using BLEU and METEOR.

Problem
Video captioning remains challenging because mapping visual sequences to language requires modelling long-term visual-textual dependencies beyond the limited capacity of LSTM memory.
Method
M3 attaches shared external memory to CNN-based video encoding and LSTM-based decoding, repeatedly reading and writing visual and textual representations.
Results
M3 outperforms state-of-the-art methods on MSVD and MSR-VTT in BLEU and METEOR.
Takeaways & Limitations
Shared multimodal memory supports long-term visual-textual interaction while guiding global visual attention for video captioning.
Takeaways & Limitations
The authors identify video representation learning as important and leave integration of feature extraction with multimodal memory as future work.

Abstract

from arXiv · show

Video captioning which automatically translates video clips into natural language sentences is a very important task in computer vision. By virtue of recent deep learning technologies, e.g., convolutional neural networks (CNNs) and recurrent neural networks (RNNs), video captioning has made great progress. However, learning an effective mapping from visual sequence space to language space is still a challenging problem. In this paper, we propose a Multimodal Memory Model (M3) to describe videos, which builds a visual and textual shared memory to model the long-term visual-textual dependency and further guide global visual attention on described targets. Specifically, the proposed M3 attaches an external memory to store and retrieve both visual and textual contents by interacting with video and sentence with multiple read and write operations. First, text representation in the Long Short-Term Memory (LSTM) based text decoder is written into the memory, and the memory contents will be read out to guide an attention to select related visual targets. Then, the selected visual information is written into the memory, which will be further read out to the text decoder. To evaluate the proposed model, we perform experiments on two publicly benchmark datasets: MSVD and MSR-VTT. The experimental results demonstrate that our method outperforms the state-of-theart methods in terms of BLEU and METEOR.

1. Introduction

Video captioning bridges vision and language but struggles to model long-term visual-textual dependencies. M3 addresses this with shared multimodal memory that guides visual attention and supports caption generation, achieving state-of-the-art results on two benchmarks.

1. Introduction: Video captioning is important for bridging vision and language, with applications including human-robot interaction, video indexing, and accessibility for visually impaired people.
1. Introduction: Video captioning remains challenging because LSTM-based decoders may fail to model long-term visual-textual dependencies in long sequences.Existing methods also struggle to capture diverse video information over extended periods.
1. Introduction: M3 uses a shared visual-textual memory to model long-term dependencies and guide global visual attention toward described targets.The model combines a CNN video encoder, multimodal memory, and LSTM text decoder.
1. Introduction: The memory repeatedly stores and retrieves decoder and visual representations to connect attention-based video selection with next-word prediction.Decoder states are written first, memory contents guide visual attention, selected visual information is written back, and memory is read for decoding.
1. Introduction: M3 achieves state-of-the-art performance on the MSVD and MSR-VTT video-captioning benchmarks.The authors report effectiveness using BLEU and METEOR metrics.

2. Related Work

Prior video-captioning work includes template-based, graphical-model, attention, embedding, and recurrent approaches. Memory-modelling research motivates external memory as a way to extend information storage and access beyond conventional recurrent networks.

2. Related Work: Earlier video-captioning approaches detect attributes, use predefined sentence templates, or apply probabilistic graphical models to align video content with language.
2. Related Work: Deep-learning video-captioning methods improve visual-to-sentence mapping through temporal attention, visual-semantic embeddings, and hierarchical recurrent decoders.
2. Related Work: Neural Turing Machines and related memory models use external or continuous memory to store and access information over long time periods.These models target long-range operations such as copying, sorting, and associative recall.

3. Multimodal Memory Modelling for Video Captioning

The model combines CNN-based visual features, an LSTM text decoder, and shared multimodal memory to capture long-range visual-textual dependencies. Memory-mediated read and write operations guide temporal attention, update stored visual and textual information, and condition sentence generation.

CNN-based video encoder: The encoder samples frames or clips and combines pretrained 2D CNN appearance features with 3D CNN motion features into a video representation.The representation is V = {v1, v2, v3, . . . , vn}, where n is the number of sampled frames or clips.
LSTM-based text decoder: The decoder conditions language generation on both its recurrent state and information read from the multimodal memory.The LSTM language model uses multimodal information as an additional input during caption generation.
Multimodal memory: Shared multimodal memory addresses the limited capacity of LSTM cells for very long visual-textual sequences.The memory is inserted between the CNN-based visual model and LSTM-based language model for long-range interaction.
Memory interaction: Before predicting each word, the LSTM writes previous hidden representations into memory, which summarizes preceding textual information.Textual erase and add vectors update memory locations in a fine-grained manner.
Temporal attention: The updated memory guides soft attention over video frames or clips to select visual information relevant to the predicted word.Attention weights are normalized across frames, and their weighted appearance and motion features form the input video representation.
Memory interaction: Selected visual information is written back into memory through visual weighting, erase, and add operations for subsequent decoder use.This update complements textual memory writes and allows visual information to participate in later language modeling.

4. Experiments

Experiments evaluate M3 across MSVD and MSR-VTT with single and fused visual features, comparing it against state-of-the-art video-captioning methods. M3 consistently improves benchmark performance and produces more target-relevant descriptions and attention.

Experimental Settings: Evaluation uses MSVD and MSR-VTT with BLEU and METEOR, comparing single visual features, fused features, and multiple state-of-the-art baselines.MSVD contains 1,970 videos, while MSR-VTT contains 10,000 video clips and 200,000 sentences.
Experimental Results on MSVD: 51.17−40.3 improvement in BLEU@4 and 31.47−29.0 improvement in METEOR are reported for M3-google over SA-google using GoogleNet on MSVD.The comparison uses the same GoogleNet feature for both models.
Experimental Results on MSVD: Among single-feature M3 variants on MSVD, M3-inv3 performs best, followed by M3-res, M3-google, and M3-vgg19.The reported ranking is similar to the corresponding ImageNet image-classification accuracy ranking.
Experimental Results on MSVD: M3-VC and M3-IC outperform the other compared methods on both metrics with fused visual features on MSVD.M3-VC uses VGG-19 and C3D, while M3-IC uses Inception-V3 and C3D.
Experimental Results on MSVD: M3-google generates more precise object terms and attends globally to targets such as “dog,” whereas SA-google attends locally to non-targets such as “guitar.”The examples come from generated descriptions on the MSVD test set.
Experimental Results on MSR-VTT: M3 consistently outperforms the corresponding SA models on MSR-VTT across VGG-19, C3D, and fused visual features.The authors attribute the improvement to multimodal memory and report generally better results with C3D than VGG-19.

5. Conclusions and Future Work

The paper concludes that M3 uses shared visual-textual memory to model long-term dependencies and guide global visual attention, outperforming state-of-the-art methods on BLEU and METEOR. It identifies video representation as an important performance factor and proposes end-to-end integration as future work.

M3 builds a visual-textual shared memory that models long-term dependency and guides global visual attention, outperforming state-of-the-art methods on BLEU and METEOR.
Experiments on two benchmark datasets support the method’s reported improvements over state-of-the-art approaches.
Video representation is important for video-captioning performance, motivating future integration of video feature extraction networks with multimodal memory in an end-to-end system.