Source-linked AI summary
CLIP4Caption: CLIP for Video Caption
Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, Xiu Li
TL;DR
Video captioning requires generating descriptions for diverse videos, while existing models may lack adequate visual representations because they neglect video-text gaps. CLIP4Caption pretrains a CLIP-enhanced video-text matching network, fine-tunes a pretrained video-language captioning architecture, and ensembles outputs; it achieves state-of-the-art MSR-VTT performance with up to a 10% CIDEr gain and ranks 2nd in the challenge test.
Problem
Existing video captioning models lack adequate visual representation because they neglect gaps between videos and texts.
Method
CLIP4Caption pretrains a CLIP-enhanced video-text matching network, fine-tunes a pretrained Uni-VL captioning architecture, and ensembles multiple captioning results.
Results
Up to 10% CIDEr gain was achieved on MSR-VTT, while the method ranked 2nd in the Video Understanding Challenge test dataset.
Takeaways & Limitations
Text-correlated video features and metric-based ensemble selection improve the reported video captioning results within the evaluated settings.
Abstract
from arXiv · showhide
Video captioning is a challenging task since it requires generating sentences describing various diverse and complex videos. Existing video captioning models lack adequate visual representation due to the neglect of the existence of gaps between videos and texts. To bridge this gap, in this paper, we propose a CLIP4Caption framework that improves video captioning based on a CLIP-enhanced video-text matching network (VTM). This framework is taking full advantage of the information from both vision and language and enforcing the model to learn strongly text-correlated video features for text generation. Besides, unlike most existing models using LSTM or GRU as the sentence decoder, we adopt a Transformer structured decoder network to effectively learn the long-range visual and language dependency. Additionally, we introduce a novel ensemble strategy for captioning tasks. Experimental results demonstrate the effectiveness of our method on two datasets: 1) on MSR-VTT dataset, our method achieved a new state-of-the-art result with a significant gain of up to 10% in CIDEr; 2) on the private test data, our method ranking 2nd place in the ACM MM multimedia grand challenge 2021: Pre-training for Video Understanding Challenge. It is noted that our model is only trained on the MSR-VTT dataset.
1 INTRODUCTION
Video captioning connects visual content with natural-language descriptions, but CNN-based encoders may underuse language information. CLIP4Caption addresses this with text-correlated video features, pretrained video-language weights, and ensemble captioning.
- Video captioning automates labor-intensive video description and supports subtitling, accessibility, interaction, and search.
- Existing sequence-learning approaches commonly use CNN-based video encoders that rely primarily on visual information.
- CLIP4Caption uses a CLIP-enhanced video-text matching network to learn strongly correlated video and text features for generation.
- The framework leverages pretrained Uni-VL video-language weights while simplifying its structure for video captioning.
- CLIP4Caption introduces an ensemble mechanism and reports improvements over multiple state-of-the-art video captioning methods on MSR-VTT.
2 METHODOLOGY
CLIP4Caption uses a two-stage pipeline: CLIP-enhanced video-text matching produces text-correlated visual features, which a Uni-VL-initialized Transformer captioner uses to generate descriptions. A metric-based ensemble then selects among captions from multiple models.
- 2 METHODOLOGY: The framework first pre-trains a video-text matching network, then fine-tunes a captioning model using its enhanced video features.The matching stage uses CLIP-based visual and BERT-based text representations, while fine-tuning performs caption generation.
- 2.1 Video-text matching pre-training: TSN sampling randomly selects one frame from each of K temporal video splits before CLIP-based encoding.This replaces fixed-position sampling and increases sampling randomness on the limited dataset.
- 2.1 Video-text matching pre-training: VTM trains self-supervised video-text matching by maximizing similarity for paired video-text examples across an N × N similarity matrix.The framework uses cosine similarity between pooled frame embeddings and text embeddings, with video-to-text and text-to-video losses.
- 2.2 Fine-tune on video captioning: The captioning model feeds strongly text-correlated video features through a one-layer Transformer Video Encoder and three-layer Transformer Decoder.Both components are initialized from Uni-VL, and training uses cross-entropy loss.
- 2.3 Ensemble strategy: The ensemble assigns generated captions metric-based importance scores and selects the highest-scoring caption as the final output.It can use BLEU4, CIDEr, SPICE, and other metrics; multiple metrics are normalized before aggregation.
3 RESULTS
CLIP4Caption is evaluated through video-text matching pre-training, MSR-VTT caption fine-tuning, and ensemble experiments on a challenge test set. The method improves captioning metrics over Uni-VL and ranks second in the Video Understanding Challenge.
- Experimental setup: The framework uses MSR-VTT for training, with Training-9K for pre-training results and Training-6K for caption fine-tuning.The dataset contains 10,000 videos and 20 descriptions per video; pre-training used 8 NVIDIA Tesla P40 GPUs with batch size 512 and 5 epochs.
- Pre-training result: VTM with three-split TSN sampling outperformed the original CLIP4Clip and was selected as the feature extractor for fine-tuning.The three-split configuration performed better in R@1 and R@5.
- Fine-tuning result: CLIP4Caption significantly improved all reported captioning metrics over Uni-VL and achieved up to a 10% gain in CIDEr on MSR-VTT.The reported metrics are BLEU4, ROUGE-L, METEOR, CIDEr, and SPICE.
- Ensemble strategy: The ensemble combines caption outputs from models with different Transformer configurations after removing ineffective validation results.Models were trained with different dataset splits and encoder-decoder layer combinations.
- Ensemble result: The ensemble significantly improved metrics over the best single model and ranked second on the Video Understanding Challenge test set.The ensemble used SPICE and BLEU-4 because other metrics performed poorly, and results improved as more model outputs were used.
4 CONCLUSION
CLIP4Caption improves video captioning by learning text-correlated visual features with video-language pre-training, initializing the captioning architecture with Uni-VL weights, and ensembling multiple models. It significantly outperforms the current state-of-the-art method and ranks second on the Video Understanding Challenge test dataset.
- 4 CONCLUSION: CLIP4Caption is a two-stage video captioning solution that pre-trains text-correlated video features before fine-tuning on MSR-VTT.The framework uses video and language pre-training models to improve visual representation for text generation.
- 4 CONCLUSION: The method initializes its encoder-decoder captioning architecture with Uni-VL pre-trained weights and adds a metric-based ensemble strategy.Multiple models’ captioning results are combined using captioning metrics.
- 4 CONCLUSION: CLIP4Caption significantly outperforms the current state-of-the-art method and ranks second on the Video Understanding Challenge test dataset.