Source-linked AI summary
Revisiting Classifier: Transferring Vision-Language Models for Video Recognition
Wenhao Wu, Zhun Sun, Wanli Ouyang
TL;DR
Video classification transfer commonly fine-tunes visual encoders while leaving the textual knowledge of vision-language models unused. The paper replaces the random classifier with pre-trained visual or textual knowledge, achieving strong general, few-shot, and zero-shot recognition with faster training. Its semantic transfer depends on meaningful category labels, limiting applicability when labels are purely numerical.
Problem
Downstream video recognition commonly overlooks the textual encoder of vision-language models despite their rich semantics and aligned visual-textual representations.
Method
The method replaces the randomly initialized linear classifier with a frozen predefined matrix initialized from visual statistics or textual embeddings of class labels.
Results
The approach improves general, few-shot, and zero-shot video recognition, including 87.8% top-1 accuracy on Kinetics-400 and large cross-dataset zero-shot gains.
Takeaways & Limitations
Classifier design can transfer semantic knowledge from vision-language pre-training while improving recognition performance and training efficiency.
Takeaways & Limitations
Performance depends on semantic category labels; numerical labels, such as re-identification identities, cannot transfer textual information.
Abstract
from arXiv · showhide
Transferring knowledge from task-agnostic pre-trained deep models for downstream tasks is an important topic in computer vision research. Along with the growth of computational capacity, we now have open-source vision-language pre-trained models in large scales of the model architecture and amount of data. In this study, we focus on transferring knowledge for video classification tasks. Conventional methods randomly initialize the linear classifier head for vision classification, but they leave the usage of the text encoder for downstream visual recognition tasks undiscovered. In this paper, we revise the role of the linear classifier and replace the classifier with the different knowledge from pre-trained model. We utilize the well-pretrained language model to generate good semantic target for efficient transferring learning. The empirical study shows that our method improves both the performance and the training speed of video classification, with a negligible change in the model. Our simple yet effective tuning paradigm achieves state-of-the-art performance and efficient training on various video recognition scenarios, i.e., zero-shot, few-shot, general recognition. In particular, our paradigm achieves the state-of-the-art accuracy of 87.8% on Kinetics-400, and also surpasses previous methods by 20~50% absolute top-1 accuracy under zero-shot, few-shot settings on five popular video datasets. Code and models can be found at https://github.com/whwu95/Text4Vis .
1 Introduction
The paper targets downstream video classification with vision-language models, where visual encoders are commonly fine-tuned while textual encoders remain underused. It revisits the classifier by transferring semantic and visual knowledge from pre-trained models.
- Vision-language models provide rich semantics and aligned image-text representations, but downstream tuning commonly overlooks the textual encoder.
- Textual class-label embeddings reveal inter-class semantic correlations, such as shared verbs, motivating their use in video classification.
- Learned classifier embeddings also develop inter-class correlations despite random initialization and optimization without label semantics.
- The proposed paradigm replaces the conventional classifier with knowledge derived from pre-trained visual and textual models.
- 87.8% top-1 accuracy is achieved on Kinetics-400, while zero-shot and few-shot accuracy improves by 20–50% over previous methods.
2 Methodology
The methodology compares standard visual transfer and vision-language tuning, then freezes a predefined classifier whose rows encode alternative knowledge sources. Textual label embeddings provide the central semantic initialization.
- Standard Vision Transferring Paradigm: Standard visual transfer jointly learns a visual encoder and randomly initialized linear classifier with cross-entropy on labeled downstream data.
- Vision-Language Learning Paradigm: Vision-language tuning uses contrastive cross-entropy over paired visual and textual samples, with textual features acting as a projection target.
- Our Proposed Paradigm: The proposed method replaces the learnable random classifier with a predefined matrix and freezes it during downstream fine-tuning.
- Our Proposed Paradigm: Four initializations vary inter-class correlation: random Gaussian, orthogonal, LDA-based visual statistics, and textual semantic embeddings.
- Textual Embedding Vectors: Textual initialization sets each classifier row from the textual encoder output for the corresponding tokenized class label, using CLIP or DistilBERT.
3 Related Works
Prior work established visual recognition backbones and vision-language pre-training, while video-text methods extended contrastive learning to video. This paper instead studies classifier correlations for efficient visual transfer.
- Visual Recognition: Convolutional networks and Vision Transformers form the principal backbone families for image and video recognition.
- Image-Language Pre-training: Vision-language pre-training aligns visual and textual representations using large-scale image-text contrastive learning, with later variants adding further objectives.
- Transferring CLIP Models for Video-Text Learning: Video-text methods extend CLIP-style pre-training for retrieval, matching, and recognition through contrastive tuning.
- Transferring CLIP Models for Video-Text Learning: This work differs by analyzing linear-classifier correlations and directly transferring visual and textual knowledge within standard video recognition.
4 Experiments: Video Recognition
Experiments evaluate the method across general, few-shot, zero-shot, efficiency, and ablation settings for video recognition. The method achieves strong benchmark performance, improves transfer under limited labels, and reduces computational cost while using frozen knowledge-based classifiers.
- Main Results: 87.8% top-1 accuracy on Kinetics-400 surpasses all compared JFT-pretrained methods, including Florence by 1.3%, despite using smaller model and data scales.The comparison uses publicly available CLIP checkpoints trained on WIT-400M.
- Main Results: 96.9% accuracy on ActivityNet, 98.2% mean class accuracy on UCF-101, and 81.3% on HMDB-51 demonstrate generalization across untrimmed and smaller-data benchmarks.The method is reported to outperform recent state-of-the-art methods on ActivityNet with a clear margin.
- Main Results: +52.6% over Swin and +21.9% over X-Florence on HMDB-51 at 2-shot show strong transferability in an extreme few-shot setting.The few-shot setup categorizes all dataset classes using K samples per category.
- Ablations: Frozen classifiers initialized with semantic or correlated knowledge outperform random and orthogonal alternatives, while the Vision-Text method improves over vision-only tuning, especially with limited data.The method initializes the classifier from textual class embeddings and freezes it during video-encoder training.
- Efficiency: 29× faster throughput and 44× fewer FLOPs than ViViT at the same accuracy show a substantial efficiency advantage.Throughput is measured with a single NVIDIA A100 GPU and batch size 16.
5 Limitation and Conclusion
The proposed paradigm improves visual-classification performance and convergence speed by using knowledge from a pretrained vision-language model's textual encoder. Its effectiveness is limited when category labels lack semantic representations, although it achieves strong performance across general, zero-shot, and few-shot video recognition.
- 5 Limitation and Conclusion: 87.8% top-1 accuracy is achieved on Kinetics-400, supporting state-of-the-art performance on challenging video datasets.
- 5 Limitation and Conclusion: The method is constrained by category-label representation, so numerical labels such as 0, 1, and 2 cannot transfer semantic information from textual encoders.Visual statistic knowledge, such as an LDA classifier, may still be helpful in this setting.
A.1 Training details
The appendix uses a shared training recipe for general video recognition across four datasets and repeats few-shot samples to match general-setting iteration counts.
- A.1 Training details: The same training recipe is used for general video recognition on Kinetics-400, ActivityNet, HMDB-51, and UCF-101.
- A.1 Training details: Few-shot training repeats samples so its iteration count matches the corresponding general-recognition setting.For Kinetics-400, the general setting uses approximately 900 iterations per epoch, which few-shot training preserves by repetition.
A.2 Evaluation protocols of zero-shot video recognition
Zero-shot evaluation uses Kinetics-400 pretrained models on other video datasets under both subset-based and full-dataset protocols. The subset protocol averages results over ten random repetitions, while the full-dataset protocol provides more realistic accuracy scores.
- A.2 Evaluation protocols of zero-shot video recognition: Zero-shot evaluation transfers Kinetics-400 pretrained models to UCF-101, HMDB-51, and ActivityNet.
- A.2 Evaluation protocols of zero-shot video recognition: The subset protocol evaluates randomly selected classes—50 for UCF-101, 25 for HMDB-51, and 100 for ActivityNet—and averages ten repetitions.These settings are denoted UCF*, HMDB*, and ANet*.
- A.2 Evaluation protocols of zero-shot video recognition: The full-dataset protocol evaluates directly on all classes to produce more realistic accuracy scores.
- A.2 Evaluation protocols of zero-shot video recognition: Kinetics-600 evaluation uses three provided splits, sampling 160 categories from 220 new categories per split and reporting the mean accuracy.
A.3 Statistics of video datasets
The appendix describes the scale and class coverage of the video datasets used for evaluation, spanning large-scale and smaller human-action benchmarks.
- A.3 Statistics of video datasets: Kinetics-400 contains 240k training videos, 20k validation videos, and 400 human-action categories.Each video is a 10-second action clip annotated from raw YouTube video.
- A.3 Statistics of video datasets: UCF-101 contains 13k videos across 101 human actions, while HMDB-51 contains approximately 7k videos across 51 action categories.
A.4 Large-scale datasets for pre-training
The paper reviews large-scale datasets used for pre-training video-recognition models, spanning image, video, and image-text data sources with widely varying scale and accessibility.
- ImageNet-1K/21K and JFT-300M provide large-scale image-classification data, with JFT-300M containing 300M images labeled across 18,291 categories.
- Pre-training resources range from 65M Instagram videos in IG-65M to 3B images in JFT-3B and 900M image-caption pairs in FLD-900M.
- Several datasets and their pre-trained weights are not open-source, including IG-65M, JFT-300M, FLD-900M, and JFT-3B.
- 400M image-text pairs from WIT-400M are used to train CLIP, whose pre-trained models support the paper’s experiments.
A.5 Visual encoder architectures
This section details the model architectures and training components used for visual transfer, including distributed contrastive learning and classifier initialization procedures.
- The visual and textual encoder architectures are specified through separate CLIP-ResNet and CLIP-ViT configuration tables.
- DDP is developed instead of DP because distributed data-parallel training avoids several single-machine and thread-related overheads.
- Distributed InfoNCE shards the similarity computation across GPUs, with each GPU comparing M vision features against NM text features.
- For LDA initialization, pre-extracted CLIP visual embeddings are used to initialize and freeze a classifier matrix before visual-encoder fine-tuning.
- LDA supplies discriminant coefficients for feature classification rather than reducing feature dimensionality.
A.8 Discussion on data overlaps
The paper discusses limited Kinetics data overlap and reports transfer performance on smaller video datasets, supported by supplementary visualizations and comparisons.
- Kinetics-700 contains less than 1% overlap in the analyzed comparison, with many overlaps consisting of black transition frames.
- Experiments on overlapping Kinetics-700 data report no performance improvement and an apparent 20% accuracy drop in one comparison.
- 98.2% and 81.3% mean class accuracy are achieved on UCF-101 and HMDB-51 after transferring models from Kinetics-400.
- Figure A.1 visualizes inter-class correlation maps for embeddings of class labels across 20 Kinetics-400 categories.
B.3 More ablations on Kinetics-400.
Additional Kinetics-400 ablations compare Vision-Text with vision-only training across label fractions, convergence speed, and text-input formulations.
- 5% additional improvement is obtained with the Vision-Text method using full data and the same training recipe, while convergence also improves.
- Vision-Text surpasses vision-only baselines across multiple label fractions, with a particularly pronounced advantage when only 10% of labels are available.
- Using class names as text input yields good results, while a single hard template only slightly improves over the class-name baseline.
- Multiple hard templates reduce Kinetics-400 performance by 0.64%, possibly because template variation introduces training noise.
- Learnable templates use four context tokens, and the results suggest that different templates have little impact on the model.