Frozen CLIP Models are Efficient Video Learners

Ziyi Lin; Shijie Geng; Renrui Zhang; Peng Gao; Gerard de Melo; Xiaogang Wang; Jifeng Dai; Yu Qiao; Hongsheng Li

Source-linked AI summary

Frozen CLIP Models are Efficient Video Learners

Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, Hongsheng Li

arXiv:2208.03550v1Aug 6, 2022cs.CV

TL;DR

Video recognition needs an alternative to computationally expensive end-to-end finetuning and the weaker direct use of pretrained image features. EVL freezes CLIP and trains a lightweight Transformer decoder with temporal modules for video recognition, achieving competitive performance with much shorter training time. The evaluation is presented under a specified ViT-B/16, eight-frame, three-view Kinetics-400 configuration unless otherwise stated.

Problem
End-to-end video finetuning is computationally and memory intensive, while directly using pretrained image features can produce subpar results.
Method
EVL trains a lightweight Transformer decoder on frozen CLIP features, using a learned query and local temporal modules for spatiotemporal fusion.
Results
EVL achieves competitive performance with much shorter training time than end-to-end finetuning.
Takeaways & Limitations
Frozen CLIP image features can support efficient video recognition while the decoder performs spatiotemporal fusion.
Takeaways & Limitations
Reported ablation-study defaults use a ViT-B/16 backbone, 8 input frames, and 3 testing views on Kinetics-400.

Abstract

from arXiv · show

Video recognition has been dominated by the end-to-end learning paradigm -- first initializing a video recognition model with weights of a pretrained image model and then conducting end-to-end training on videos. This enables the video network to benefit from the pretrained image model. However, this requires substantial computation and memory resources for finetuning on videos and the alternative of directly using pretrained image features without finetuning the image backbone leads to subpar results. Fortunately, recent advances in Contrastive Vision-Language Pre-training (CLIP) pave the way for a new route for visual recognition tasks. Pretrained on large open-vocabulary image-text pair data, these models learn powerful visual representations with rich semantics. In this paper, we present Efficient Video Learning (EVL) -- an efficient framework for directly training high-quality video recognition models with frozen CLIP features. Specifically, we employ a lightweight Transformer decoder and learn a query token to dynamically collect frame-level spatial features from the CLIP image encoder. Furthermore, we adopt a local temporal module in each decoder layer to discover temporal clues from adjacent frames and their attention maps. We show that despite being efficient to train with a frozen backbone, our models learn high quality video representations on a variety of video recognition datasets. Code is available at https://github.com/OpenGVLab/efficient-video-recognition.

1 Introduction

Video recognition commonly relies on expensive end-to-end finetuning of pretrained image models, motivating efficient transfer from frozen CLIP features. EVL trains a lightweight decoder for spatiotemporal fusion and achieves competitive performance with shorter training.

Results: 60 V100 GPU-hours train the 8-frame ViT-B/16 EVL model to 82.9% top-1 accuracy on Kinetics-400.Figure 1 presents this result as part of a new accuracy-versus-inference-FLOPS Pareto frontier.
Motivation: End-to-end video finetuning is costly because videos require multiple frames and larger models, increasing computation and memory consumption.It can also risk catastrophic forgetting when downstream videos are insufficiently informative.
Motivation: CLIP provides transferable visual representations aligned with rich language semantics through large-scale open-vocabulary image–text pretraining.This motivates using frozen CLIP image features for video recognition.
EVL: EVL trains a lightweight Transformer decoder on fixed CLIP features to dynamically gather frame-level spatial information and perform spatiotemporal fusion.A learned query token collects features from encoder layers, while temporal modules gather cues using convolution, positional embeddings, and cross-frame attention.
Results: EVL incurs much shorter training time than end-to-end finetuning while achieving competitive performance.The authors position this efficiency as broadening access to video recognition for researchers with average computational resources.

2 Related Work

Prior video-recognition work advances architectures and training strategies, but efficient transfer learning remains limited. EVL uses a Transformer-decoder design to adapt fixed image features for video understanding.

Video Recognition: Video recognition architectures have progressed from two-stream and 3D-CNN models toward spatiotemporal Transformers.Examples include fused CNN-Transformer designs and multi-stream Transformers operating at different resolutions.
Video Recognition: Existing video methods benefit from image pretraining, but end-to-end finetuning remains expensive because of its large memory footprint.This limits efficient scaling of pretrained image models for video recognition.
Efficient Transfer Learning: EVL’s architecture feeds intermediate feature maps from a pretrained image backbone into a Transformer decoder to gather spatiotemporal information.The supplied figure describes temporal modeling on raw frame features to retain spatiotemporal structure.
Efficient Transfer Learning: Efficient transfer strategies include adapters that keep pretrained weights fixed and prompts that add learnable tokens for task-specific adaptation.These approaches share EVL’s motivation, while EVL uses Transformer decoders as a more flexible efficient alternative.

3 Our Method

EVL transfers frozen CLIP image features to video recognition through a lightweight spatiotemporal Transformer decoder. It summarizes multi-frame features, captures local temporal information, and avoids updating the image encoder during training while adding limited computational overhead.

Overall Structure: EVL targets multi-frame summarization, motion capture, and efficient image-to-video transfer with a fixed CLIP backbone.The framework is designed to produce video-level predictions while limiting computation and memory use.
Overall Structure: A learned video-level classification query aggregates multi-layer spatiotemporal features extracted independently from each frame.Backbone feature volumes provide keys and values to decoder blocks, whose final output is projected to class predictions.
Training: Training stops back-propagation at the fixed image features, so no image-encoder weights are updated.This design reduces memory consumption and time per training iteration.
Learning Temporal Cues from Spatial Features: Local temporal modules combine adjacent-frame information through temporal convolution, positional embeddings, and cross-frame attention maps.Temporal convolution captures local feature variations, positional embeddings distinguish time positions, and attention maps expose feature correspondence between frames.
Learning Temporal Cues from Spatial Features: Temporal cross-attention maps learned from 2D image features still provide useful signals for video modeling.The final temporal features are blended with original spatial features through a residual combination.
Complexity Analysis: The decoder adds at most 20% FLOPS to the backbone even with one decoder block per encoder output and all temporal modules enabled.Only one query token is used, making the decoder substantially lighter than an encoder block.

4 Experiments

Experiments show that EVL achieves competitive video-recognition accuracy with substantially lower computation, training time, and memory than conventional approaches. Ablations and dataset analyses identify the benefits of high-resolution intermediate features, temporal modules, and frozen CLIP representations.

Main Results: EVL achieves competitive accuracy while requiring substantially less computation than similarly accurate regular video-recognition models.
Main Results: EVL achieves higher accuracy with fewer frames and fewer new parameters than prior CLIP-based video-recognition methods.
Main Results: Roughly 10× less training time comes from powerful pretraining, with the efficient transfer scheme providing a further reduction of about 8×.
Ablation Studies: Multiple decoder blocks improve accuracy by 1.0%, and feeding blocks multi-layer intermediate features adds another 0.8%.Deeper-layer features are more effective for video recognition.
Ablation Studies: Using high-resolution, unpooled feature maps is crucial, whereas summarizing temporal or spatial dimensions significantly reduces accuracy.The authors conjecture that task-specific re-attention helps prioritize features such as human-body information.
Analysis of Temporal Information: Temporal modules improve Something-Something-v2 accuracy by 13.8% but provide less than 0.5% gain on Kinetics-400.The analysis attributes this contrast to different temporal-information requirements across the benchmarks.
Analysis of Temporal Information: Something-Something-v2 benefits from all 12 decoder blocks, whereas Kinetics-400 requires only around 4 blocks.

5 Conclusion

EVL introduces an efficient video action-recognition pipeline that learns a transfer head over fixed image features. Freezing the backbone reduces training demands while multi-layer features and complementary open-world knowledge support strong video recognition.

EVL learns an efficient transfer-learning head on top of fixed transferable image features for video action recognition.
Freezing the image backbone vastly reduces training time and avoids heavy or prohibitive full-finetuning of large image models.
Multi-layer high-resolution intermediate feature maps largely compensate for accuracy loss caused by the frozen backbone.
Transferable image features learned in an open-world setting are highly complementary to features from labeled datasets.

A Implementation Details

The experiments use fixed dataset-specific training and testing protocols, with CLIP ViT backbones and Transformer decoders as the main model configuration. Full-finetuning baselines use reduced learning rates for pretrained backbone weights.

Kinetics-400: Kinetics-400 models use 224-pixel inputs, evenly strided frames, batch size 256, 50,000 AdamW steps, and a cosine learning-rate schedule.The 8-, 16-, and 32-frame variants use temporal strides of 16, 16, and 8, respectively.
Something-Something-v2: Something-Something-v2 uses TSN-style segment sampling, center-frame evaluation, three spatial crops, and 30,000 training steps.A random frame is selected from each segment during training.
Model Details: The default image backbone is CLIP-pretrained ViT-B/16, with decoder depth and backbone-layer inputs varied by dataset and model size.Kinetics-400 uses four decoder blocks, while Something-Something-v2 uses deeper decoders for motion modeling.
Full-finetuning Details: Full-finetuning experiments reduce the learning rate on pretrained backbone weights by 100x relative to other parameters.This configuration is used for TimeSformer and the authors’ own architecture in CLIP-related experiments.

B Results on UCF-101 and HMDB-51

EVL is evaluated on UCF-101 and HMDB-51 using checkpoints adapted from Kinetics-400. The reported results are competitive even against methods using additional modalities.

EVL is benchmarked on the additional UCF-101 and HMDB-51 action-recognition datasets.
The method reports competitive results against approaches that use additional modalities such as optical flow and pose.
Implementation Details: UCF-101 and HMDB-51 experiments fine-tune Kinetics-400 checkpoints with reduced learning rate and weight decay on pretrained weights.Testing uses two temporal views and three spatial views, with 32 frames per view at temporal stride 2.

C Model Ensemble Results

The paper reports ensemble results for EVL models combined with Uniformer-B on Kinetics-400 and Something-Something-v2. These ensembles are intended to assess the complementarity of the learned video features.

Complete ensemble results are reported on Kinetics-400 and Something-Something-v2 by combining an EVL model with Uniformer-B using 32 frames.

D Qualitative Results

Qualitative visualizations show that the video-level decoder focuses more on human-action-specific regions and captures interpretable motion patterns across frames. Additional results report ensemble accuracies and accuracy–compute trade-offs on benchmark datasets.

Model Ensemble Results: The reported Uniformer-B baseline accuracies used in the ensembles are 82.9% on Kinetics-400 and 71.2% on Something-Something-v2.These values are specified for the 32-frame Uniformer-B model.
Video-level Decoder Attention Maps: The video-level [CLS] token concentrates attention more on human-action-specific regions than the original CLIP [CLS] token.Highlighted regions include human bodies, facial parts, objects in hands, and moving objects.
Cross-frame Attention Maps: With only nonparametric modules added, the frozen-image-pretrained backbone spontaneously captures human-identifiable motion information.Visualized patterns include position change, shape change, and objects appearing or disappearing across adjacent frames.
Model Efficiency: Figure 5 compares model ensemble and single-model accuracy against GFLOPS on Kinetics-400.The figure presents accuracy–compute relationships for both model configurations.