Source-linked AI summary
ActionCLIP: A New Paradigm for Video Action Recognition
Mengmeng Wang, Jiazheng Xing, Yong Liu
TL;DR
Video action recognition traditionally maps labels to fixed categories, limiting transfer to unseen concepts. The paper instead uses video-text matching with semantic label supervision and introduces pre-train, prompt, and fine-tune; ActionCLIP reaches 83.8% top-1 accuracy on Kinetics-400 while supporting zero-shot and few-shot recognition.
Problem
Traditional models predict fixed predefined categories and require additional labeled data to transfer to new unseen concepts, while dataset labels are too succinct for rich language learning.
Method
The paper formulates recognition as multimodal video-text matching and reuses web-pretrained models through prompting followed by end-to-end fine-tuning on target datasets.
Results
ActionCLIP achieves 83.8% top-1 accuracy on Kinetics-400 and reports consistent outperformance of state-of-the-art methods across several public benchmarks.
Takeaways & Limitations
Semantic language supervision broadens action-recognition representations and supports zero-shot/few-shot transfer while reducing the cost of pre-training.
Abstract
from arXiv · showhide
The canonical approach to video action recognition dictates a neural model to do a classic and standard 1-of-N majority vote task. They are trained to predict a fixed set of predefined categories, limiting their transferable ability on new datasets with unseen concepts. In this paper, we provide a new perspective on action recognition by attaching importance to the semantic information of label texts rather than simply mapping them into numbers. Specifically, we model this task as a video-text matching problem within a multimodal learning framework, which strengthens the video representation with more semantic language supervision and enables our model to do zero-shot action recognition without any further labeled data or parameters requirements. Moreover, to handle the deficiency of label texts and make use of tremendous web data, we propose a new paradigm based on this multimodal learning framework for action recognition, which we dub "pre-train, prompt and fine-tune". This paradigm first learns powerful representations from pre-training on a large amount of web image-text or video-text data. Then it makes the action recognition task to act more like pre-training problems via prompt engineering. Finally, it end-to-end fine-tunes on target datasets to obtain strong performance. We give an instantiation of the new paradigm, ActionCLIP, which not only has superior and flexible zero-shot/few-shot transfer ability but also reaches a top performance on general action recognition task, achieving 83.8% top-1 accuracy on Kinetics-400 with a ViT-B/16 as the backbone. Code is available at https://github.com/sallymmx/ActionCLIP.git
1. Introduction
ActionCLIP reframes video action recognition as multimodal video-text matching, using semantic label information to improve representation and transfer. Its pre-train, prompt, and fine-tune paradigm reuses web-pretrained models while achieving strong benchmark performance.
- Traditional action recognition predicts fixed predefined categories, limiting transfer to unseen concepts because new datasets require additional labeled training data.
- The multimodal framework encodes videos and label texts separately, then pulls corresponding representations together through similarity learning.Inference becomes video-text matching rather than 1-of-N classification.
- The framework strengthens video representations with semantic language supervision and supports zero-shot transfer without further labeled data or parameters.
- The proposed paradigm reuses large-scale web-pretrained models through prompt design and target-dataset fine-tuning, reducing pre-training cost.It addresses succinct dataset labels and the storage, computation, and experiment costs of end-to-end web-data pre-training.
- Comprehensive experiments report consistent outperformance of state-of-the-art methods across several public benchmark datasets.
2. Related Works
Related work spans hand-crafted and increasingly engineered video architectures, while vision-text methods motivate treating action recognition as multimodal retrieval. ActionCLIP focuses on incorporating label-text semantics into general action recognition rather than only learning video features or text embeddings.
- 2.1. Video Action Recognition: Video action recognition progressed from hand-crafted spatio-temporal descriptors to two-stream, 3D CNN, compute-efficient, and transformer-based architectures.
- 2.1. Video Action Recognition: The paper proposes a pre-trained, prompt, and fine-tune paradigm based on video-text multimodal learning to foreground language modeling of label words.
- 2.2. Vision-text Multi-modality in Action Recognition: Prior vision-text approaches either learn video representations while adding a classifier or emphasize embedding design with simple text features, leaving general action recognition less addressed.
3. Method
ActionCLIP reframes action recognition as video-text matching rather than unimodal label classification, then combines multimodal learning with pre-training, prompting, and end-to-end fine-tuning. Its framework uses separate video and language encoders, cosine similarity, contrastive KL loss, and temporal visual prompts.
- 3.1. Multimodal Learning Framework: The framework replaces fixed-label classification with video-text matching, using semantic label information to compare videos with textual categories.Inference selects the label words with the highest similarity score.
- 3.1. Multimodal Learning Framework: Separate video and language encoders produce spatiotemporal video features and label-text features, which are aligned through symmetric cosine similarities.The encoded representations are compared in both video-to-text and text-to-video directions.
- 3.1. Multimodal Learning Framework: The method uses a video-text contrastive objective based on symmetric KL divergence because multiple videos in a batch may share one label.This avoids treating similarity learning as a strict 1-in-N classification problem.
- 3.2. The New Paradigm: The proposed paradigm combines web-data pre-training, textual and visual prompt engineering, and end-to-end fine-tuning on target datasets.Prompting addresses deficient label words while adapting pre-trained representations to action recognition.
- 3.3. New Paradigm Instantiation Details: ActionCLIP instantiates the paradigm with CLIP, manual textual prompts, and visual prompts that adapt image encoders to spatiotemporal video information.Visual prompt choices include pre-network, in-network, and post-network designs, with temporal modules such as MeanP, Conv1D, LSTM, and Transf.
- 3.3. New Paradigm Instantiation Details: The complete ActionCLIP network is fine-tuned end-to-end using the video-text contrastive training objective.The architecture includes two single-modal encoders, similarity calculation, and multiple prompt locations.
4. Experiments
Experiments show that ActionCLIP benefits from multimodal supervision, pre-training, suitable prompts, and end-to-end fine-tuning. It also transfers to data-poor settings and achieves strong accuracy while offering efficient configurations.
- Ablation Experiments: 2.91% top-1 accuracy gains result from exploiting label-text semantics in the multimodal framework.The comparison uses the same backbone, pre-trained weights, and temporal modeling strategy before the final linear layer.
- Ablation Experiments: 78.36% versus 40.10% shows that proper visual-encoder initialization is especially important for pre-training.Randomly initializing both encoders produces a 41.4% drop from the fully pre-trained model.
- Ablation Experiments: 2.74% and 5.38% performance decreases occur with Joint and Shift visual prompts, respectively, relative to MeanP.The authors attribute these decreases to catastrophic forgetting caused by changed input patterns or altered pre-trained features.
- Ablation Experiments: 6.15% lower top-1 accuracy occurs when both encoders are frozen, whereas end-to-end fine-tuning reaches 78.36%.Freezing either pre-trained encoder negatively influences accuracy, supporting dataset-specific end-to-end fine-tuning.
- Comparison with State-of-the-art Methods: 44.3 mAP is achieved on Charades, demonstrating effectiveness for multi-label video classification.The dataset contains longer-range activities and multiple actions per video.
5. Conclusion
The paper reframes action recognition as video-text multimodal learning and introduces “pre-train, prompt, and fine-tune” to reuse large-scale web-data models. ActionCLIP achieves strong performance on general and zero-shot/few-shot recognition.
- Action recognition is modeled as a video-text multimodal learning problem that exploits semantic information in label texts.
- The “pre-train, prompt, and fine-tune” paradigm reuses powerful large-scale web-data pre-trained models while greatly reducing pre-training cost.
- ActionCLIP delivers superior performance on general and zero-shot/few-shot action recognition.