Source-linked AI summary
Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, Mubarak Shah
TL;DR
Video recognition with CLIP faces a trade-off between supervised accuracy and zero-shot generalization, often requiring separate models. Vita-CLIP uses frozen-backbone multimodal prompting, achieving state-of-the-art zero-shot results across three datasets while remaining competitive in supervised recognition with one unified training scheme.
Problem
Adapting image-text pretrained models to video recognition must balance supervised performance with retained zero-shot generalization.
Method
Vita-CLIP freezes CLIP and learns global video-level, local frame-level, summary, and textual-context prompts for video recognition.
Results
Vita-CLIP achieves state-of-the-art zero-shot performance on HMDB51, UCF101, and Kinetics-600 while remaining competitive in supervised recognition using one training scheme.
Takeaways & Limitations
A unified frozen-backbone prompting model can balance zero-shot and supervised video recognition while training substantially fewer parameters.
Abstract
from arXiv · showhide
Adopting contrastive image-text pretrained models like CLIP towards video classification has gained attention due to its cost-effectiveness and competitive performance. However, recent works in this area face a trade-off. Finetuning the pretrained model to achieve strong supervised performance results in low zero-shot generalization. Similarly, freezing the backbone to retain zero-shot capability causes significant drop in supervised accuracy. Because of this, recent works in literature typically train separate models for supervised and zero-shot action recognition. In this work, we propose a multimodal prompt learning scheme that works to balance the supervised and zero-shot performance under a single unified training. Our prompting approach on the vision side caters for three aspects: 1) Global video-level prompts to model the data distribution; 2) Local frame-level prompts to provide per-frame discriminative conditioning; and 3) a summary prompt to extract a condensed video representation. Additionally, we define a prompting scheme on the text side to augment the textual context. Through this prompting scheme, we can achieve state-of-the-art zero-shot performance on Kinetics-600, HMDB51 and UCF101 while remaining competitive in the supervised setting. By keeping the pretrained backbone frozen, we optimize a much lower number of parameters and retain the existing general representation which helps achieve the strong zero-shot performance. Our codes/models are released at https://github.com/TalalWasim/Vita-CLIP.
1. Introduction
Vita-CLIP addresses the supervised–zero-shot trade-off in video recognition by adapting frozen image-text CLIP with multimodal prompts. Its unified prompting scheme models video information at global, frame, summary, and textual-context levels while improving zero-shot performance and remaining competitive under supervision.
- 1. Introduction: Existing CLIP-based video methods face a supervised–zero-shot trade-off, with prior state-of-the-art approaches training separate models for the two settings.Vita-CLIP instead uses the same training scheme for both evaluation settings.
- 1. Introduction: The method freezes the pretrained CLIP backbone and learns lightweight visual and textual modules to preserve generalization while adapting to videos.The design targets both the pretrained model’s general representations and missing video-domain information.
- 1. Introduction: Global video-level, local frame-level, and summary prompts capture distribution shifts, per-frame discrimination, and condensed video representations.The vision prompts explicitly incorporate temporal and video-level information absent from an image-pretrained model.
- 1. Introduction: Learnable text prompts augment limited class-label descriptions so language representations better match video recognition.The text-side scheme adapts textual context when datasets provide labels rather than per-sample descriptions.
- 1. Introduction: Vita-CLIP gains 4.0%, 3.0%, and 2.2% over X-CLIP on HMDB-51, UCF101, and Kinetics-600 zero-shot recognition, respectively.The method uses one unified model for supervised and zero-shot settings.
2. Related work
Related work establishes CLIP as a transferable vision-language foundation and surveys video recognition, prompt learning, and CLIP adaptation. Vita-CLIP differs by jointly learning multimodal prompts for spatiotemporal video recognition.
- Vision-Language Models: Vision-language models learn a shared image–text feature space from large-scale pairs and support zero-shot transfer to downstream tasks.The surveyed applications include segmentation, detection, point-cloud classification, and video classification.
- Video Recognition: Video recognition has progressed from handcrafted features to CNNs and Transformers that model spatial and temporal information.The literature includes 3D CNNs, separate 2D spatial and dynamic networks, and Transformer architectures.
- Prompt Learning: Prompt learning transfers pretrained-model knowledge through manually designed or learned task-specific instructions and vectors.Prior methods apply prompts to vision transformers or CLIP text encoders.
- Adapting VL Models for Videos: Prior CLIP video adaptations include full fine-tuning and text prompting with added temporal layers, but direct image-encoder use can lack early temporal information.Vita-CLIP instead learns multimodal prompts to steer both CLIP encoders for spatiotemporal learning.
3. Vita-CLIP: Methodology
Vita-CLIP adapts frozen image-text CLIP encoders to video recognition through lightweight multimodal prompts that preserve generalization while adding video-specific temporal and contextual modeling.
- 3. Vita-CLIP: Methodology: Vita-CLIP keeps the pretrained vision and text encoders frozen while introducing lightweight learnable modules for video adaptation.This preserves the pretrained representation while adapting the model to video tasks.
- 3.2.2 Text Encoder Prompt Learning: The text encoder replaces hand-crafted class-label prompts with trainable context vectors that augment textual representations for video categories.The text prompt consists of trainable vectors followed by the class label and is passed through the pretrained text encoder.
- 3.2. Video and text Prompt Learning: The architecture appends prompts to both encoders, combining video-level, frame-level, summary, and text-context conditioning in one multimodal scheme.The figure identifies the corresponding prompt types and their roles in the vision and text encoders.
- 3.2.1 Video Encoder Prompt Learning: The video encoder adds summary, global, and frame-level local prompts to exchange temporal information and adapt to the video data distribution.Summary tokens condense information across frames, global prompts learn distribution characteristics, and local prompts model frame-specific discriminative information.
- 3.2.1 Video Encoder Prompt Learning: Summary tokens are formed from frame classification tokens and appended to each frame sequence before frozen self-attention is applied.The summary tokens enable information exchange across frames within the vision encoder.
- 3.2.1 Video Encoder Prompt Learning: Global prompts are learnable vectors for modeling video-distribution characteristics, whereas local prompts are frame-specific vectors conditioned on each frame’s classification token.The number of local prompts matches the number of frames used during training.
4. Results and Analysis
Vita-CLIP balances supervised and zero-shot video recognition with one frozen-backbone model, achieving strong benchmark performance through multimodal prompting. Its global, local, and summary video prompts complement one another, while text prompting and prompt-count choices further shape results.
- 4.2. Supervised Experiments: On Something-Something-V2, Vita-CLIP surpasses the best vision-text method B6 by more than 10% but remains below cross-entropy-based methods.The paper attributes the vision-language gap to the fine-grained nature of SSv2 class descriptions.
- 4.4. Supervised vs. Zero-shot Trade-off: A single Vita-CLIP model balances supervised and zero-shot recognition, unlike X-CLIP's separate settings, using 8 frames and 30 training epochs.Vita-CLIP retains the generalized backbone representation while adapting to videos with prompt learning.
- 4.5. Ablations: Adding global prompts, local prompts, and a summary token raises Kinetics-400 top-1 accuracy from 77.83% to 79.16% and then 80.51%.The ablation reports the three prompting techniques as complementary; global prompt accuracy saturates around Mv = 8, and class-specific text context performs better than unified context.
- 4.5. Ablations: The prompting scheme focuses attention on localized discriminative parts and motion properties in sample videos.Attention rollout comparisons cover Cooking Chicken, Playing Guitar, Pull Ups, and Skipping Rope.
5. Conclusion
Vita-CLIP uses multimodal prompting to balance zero-shot and supervised video recognition within one unified training scheme. It achieves state-of-the-art zero-shot performance while remaining competitive supervised and training far fewer parameters.
- Vita-CLIP achieves state-of-the-art zero-shot performance on UCF101, HMDB51, and Kinetics-600 while remaining competitive supervised on Kinetics-400 and SSv2.
- The unified method balances zero-shot and supervised performance using the same training scheme instead of separate models.
- Vita-CLIP trains substantially fewer parameters while using multimodal prompting for video recognition.