Source-linked AI summary
Video Captioning with Transferred Semantic Attributes
Yingwei Pan, Ting Yao, Houqiang Li, Tao Mei
TL;DR
Video captioning must translate visual content into natural-language descriptions while existing CNN-plus-RNN approaches underuse high-level semantics. LSTM-TSA learns and transfers complementary attributes from images and videos through a transfer unit, achieving the paper’s reported best published MSVD performance and superior results on two other datasets.
Problem
Video captioning seeks to generate natural-language descriptions from videos, while CNN-plus-RNN methods leave high-level semantic cues underexplored.
Method
LSTM-TSA incorporates semantic attributes learned from images and videos into CNN-plus-RNN sequence learning and fuses them with a transfer unit.
Results
LSTM-TSA achieves superior results across MSVD, M-VAD, and MPII-MD, including the best published MSVD performance reported by the paper.
Takeaways & Limitations
Jointly exploiting video representations and complementary semantic attributes improves video sentence generation within the evaluated datasets.
Takeaways & Limitations
The authors identify adding attention and extending the method to multiple-sentence or paragraph generation as future work.
Abstract
from arXiv · showhide
Automatically generating natural language descriptions of videos plays a fundamental challenge for computer vision community. Most recent progress in this problem has been achieved through employing 2-D and/or 3-D Convolutional Neural Networks (CNN) to encode video content and Recurrent Neural Networks (RNN) to decode a sentence. In this paper, we present Long Short-Term Memory with Transferred Semantic Attributes (LSTM-TSA)---a novel deep architecture that incorporates the transferred semantic attributes learnt from images and videos into the CNN plus RNN framework, by training them in an end-to-end manner. The design of LSTM-TSA is highly inspired by the facts that 1) semantic attributes play a significant contribution to captioning, and 2) images and videos carry complementary semantics and thus can reinforce each other for captioning. To boost video captioning, we propose a novel transfer unit to model the mutually correlated attributes learnt from images and videos. Extensive experiments are conducted on three public datasets, i.e., MSVD, M-VAD and MPII-MD. Our proposed LSTM-TSA achieves to-date the best published performance in sentence generation on MSVD: 52.8% and 74.0% in terms of BLEU@4 and CIDEr-D. Superior results when compared to state-of-the-art methods are also reported on M-VAD and MPII-MD.
1. Introduction
Video captioning commonly uses CNNs to encode videos and RNNs to generate sentences, but these approaches underuse high-level semantic cues. LSTM-TSA incorporates complementary image- and video-derived attributes into sequence learning and models their fusion for caption generation.
- Video captioning describes videos with natural language and remains challenging for computer vision and language processing.
- CNN-plus-RNN sequence models encode video representations and decode sentences, but leave high-level semantic cues underexplored.Semantic attributes have been effective in related vision-to-language tasks.
- LSTM-TSA incorporates semantic attributes into sequence learning for video captioning.The architecture is designed to exploit attributes together with video representations.
- Image attributes capture static objects and scenes, whereas video attributes convey temporal dynamics such as actions.The two sources therefore provide complementary semantic information for sentence generation.
- LSTM-TSA investigates how to fuse image- and video-derived attributes and exploit their mutual relationship for sentence generation.The paper frames this joint use of complementary attributes as an incompletely explored problem.
2. Related Work
Prior video-captioning work follows template-based or sequence-learning approaches, while attribute-based sequence learning has mainly focused on single domains. LSTM-TSA extends this line by learning attributes from image and video domains and dynamically fusing them.
- Video Captioning: Video captioning research has developed through template-based language methods and sequence-learning approaches such as RNNs.Template methods align visual concepts with sentence fragments and use predefined language templates.
- Video Captioning: Sequence-learning video captioning models include mean pooling, optical-flow inputs, semantic regularization, and temporal attention.These approaches differ in how they represent video content and exploit temporal structure.
- Sequence Learning by Using Attributes: Attribute-based sequence learning injects semantic properties from visual content into sentence generation and has been studied especially for image captioning.Attributes are described as properties with rich semantic cues that also support visual recognition.
- Sequence Learning by Using Attributes: LSTM-TSA is presented as the first effort to leverage semantic attributes in video captioning.It learns attributes from both image and video domains rather than relying solely on attributes learned within one domain.
3. Video Captioning with Transferred Semantic Attributes
The paper develops an end-to-end CNN-plus-LSTM video-captioning framework that injects semantic attributes learned from images and videos. Its transfer unit dynamically controls and fuses these complementary attribute sources during sentence generation.
- Semantic Attributes: Video MIL jointly processes sampled frames as one bag to reduce the semantic shift and noise caused by assigning video-level descriptions to individual frames.All sampled frames from a video are used together when learning video attributes.
- Framework: The framework learns semantic attributes from images and videos and incorporates both sources into an LSTM-based video-captioning model.The video representation is encoded from sampled frames or clips, while attribute representations provide additional semantic information during decoding.
- Transfer Unit: A transfer gate uses current word and previous LSTM-state context together with image and video attributes to select valuable semantic information.The gate produces a sigmoid-bounded weight vector that controls attribute impacts before their integration into the LSTM.
- Transfer Unit: Three transfer-unit variants differ in whether the gate controls image attributes, video attributes, or both through decoupled weights and a linear combination.The first two designs control one source, while the third simultaneously controls both sources.
4. Experiments
Experiments evaluate LSTM-TSA variants against established video-captioning methods on three datasets, using standard captioning metrics and ablations of attribute learning and transfer design. LSTM-TSAIV achieves the strongest reported results, while complementary image/video attributes and holistic video MIL improve generation.
- Experimental setup: Experiments compare LSTM-TSA variants with seven state-of-the-art video-captioning methods on MSVD, M-VAD, and MPII-MD.The evaluation uses BLEU@N, METEOR, and CIDEr-D where reported.
- MSVD results: 74.0% CIDEr-D makes LSTM-TSAIV the highest reported MSVD result, outperforming non-attention and attention-based methods.The reported relative improvements over TA, GRU-RCN, and h-RNN are 43.1%, 8.8%, and 12.5%, respectively.
- Attribute ablations: Attributes improve LSTM-based captioning, with video-trained attributes outperforming image-trained attributes and combined attributes improving over LSTM-TSAV.The results support complementary contributions from image and video domains.
- Movie-dataset results: 7.2% and 8.0% METEOR are achieved by LSTM-TSAIV on M-VAD and MPII-MD, exceeding HRNE and LSTM-E by 5.9% and 9.6% relative improvement.The passage attributes the comparisons to the best competitor on each dataset.
- Qualitative analysis: The qualitative examples show that LSTM-TSAIV predicts more accurate and descriptive words by jointly using video representations and image/video semantic attributes.Examples include “woman” and “lying,” “cleaning” and “floor,” and the added description “forest.”
- Attribute-learning analysis: Video MIL yields better LSTM-TSAV performance than applying image MIL independently to each video frame.The comparison indicates an advantage from modeling semantic information across all sampled frames holistically.
- Transfer-unit analysis: Transfer-unit variants are evaluated on MSVD, and weighting attributes from both sources produces a larger performance gain than direct element-wise summation.The comparison concerns LSTM-TSAIV configurations with and without transfer-gate-based control.
5. Discussions and Conclusions
The paper concludes that LSTM-TSA improves video captioning by jointly exploiting video representations and semantic attributes from images and videos. It identifies attention and multi-sentence or paragraph generation as future extensions.
- Conclusions: LSTM-TSA jointly explores video representations and semantic attributes for video captioning, using video MIL and a transfer unit to control attribute impacts.Experiments on three datasets validate the proposal and report improvements over other captioning techniques.
- Future work: Future work will incorporate attention into LSTM-TSA and investigate semantic attributes for generating multiple sentences or paragraphs.These are presented as two directions for further development.