Expanding Language-Image Pretrained Models for General Video Recognition

Bolin Ni; Houwen Peng; Minghao Chen; Songyang Zhang; Gaofeng Meng; Jianlong Fu; Shiming Xiang; Haibin Ling

Source-linked AI summary

Expanding Language-Image Pretrained Models for General Video Recognition

Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, Haibin Ling

arXiv:2208.02816v1Aug 4, 2022cs.CV

TL;DR

Adapting pretrained language-image models to video recognition avoids the cost of training a new language-video model while addressing temporal modeling and prompt generation. The paper adds cross-frame attention and video-specific prompting, achieving reported gains across supervised, zero-shot, and few-shot settings, including 87.1% top-1 accuracy with 12× fewer FLOPs than ViViT-H.

Problem
The paper asks how to adapt pretrained language-image models to video recognition, where temporal information and video-text adaptation remain insufficiently addressed without costly video-language pretraining.
Method
The method adds lightweight cross-frame communication and multi-frame integration for temporal modeling, plus learnable prompts combining semantic labels with video representations.
Results
The approach outperforms prior methods across fully supervised, zero-shot, and extremely limited few-shot settings, including 87.1% top-1 accuracy on Kinetics-400 with 12× fewer FLOPs than ViViT-H.
Takeaways & Limitations
Pretrained language-image models can be expanded directly to general video recognition across multiple learning configurations without pretraining a new video-language model from scratch.
Takeaways & Limitations
Multi-view inference has relatively high computational cost because its cost grows linearly with the number of views, although single-view performance can be comparable to 10 temporal views.

Abstract

from arXiv · show

Contrastive language-image pretraining has shown great success in learning visual-textual joint representation from web-scale data, demonstrating remarkable "zero-shot" generalization ability for various image tasks. However, how to effectively expand such new language-image pretraining methods to video domains is still an open problem. In this work, we present a simple yet effective approach that adapts the pretrained language-image models to video recognition directly, instead of pretraining a new model from scratch. More concretely, to capture the long-range dependencies of frames along the temporal dimension, we propose a cross-frame attention mechanism that explicitly exchanges information across frames. Such module is lightweight and can be plugged into pretrained language-image models seamlessly. Moreover, we propose a video-specific prompting scheme, which leverages video content information for generating discriminative textual prompts. Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited. Code and models are available at https://aka.ms/X-CLIP

1 Introduction

The paper adapts pretrained language-image models to general video recognition, addressing temporal modeling and video-specific text prompting without training a new video-language model from scratch. X-CLIP and X-Florence support fully supervised, zero-shot, and few-shot recognition across varied settings.

Motivation: Closed-set video recognition assumes predefined categories, limiting recognition of unseen or unfamiliar categories in applications such as automatic web-video tagging.The paper identifies recognizing categories unavailable during training as a central challenge.
Motivation: Directly training a language-video model is costly because it requires large-scale video-text data and extensive GPU resources.The authors motivate adapting existing image-level language-image models instead.
Method: The framework uses cross-frame communication and multi-frame integration transformers to exchange information across frames and produce video-level representations.Message tokens communicate frame semantics and dependencies before frame-level representations are integrated.
Method: A learnable video-specific prompting mechanism combines semantic labels with video representations to automatically generate more discriminative textual representations.It uses video content to provide contextual information for recognition rather than relying only on manually fixed prompts.
Results: 87.1%/88.3% top-1 accuracy is achieved by X-CLIP-L/14 on Kinetics-400/600, surpassing ViViT-H by 2.3%/2.5% with 12× fewer FLOPs.The approach also surpasses ActionCLIP by +7.6% and +14.9% in zero-shot experiments, and prevailing methods by +32.1% and +23.1% in extremely limited few-shot settings.
Contributions: The work contributes a general strategy for expanding large-scale language-image pretrained models to video recognition and potentially other video tasks.Experiments across learning configurations support its reported effectiveness and generalization.

2 Related Work

Prior work transfers visual-language pretraining to downstream tasks and video, but video adaptation remains constrained by computational cost and limited temporal interaction. This paper positions its approach as a more general, efficient alternative that supports multiple pretrained language-image models.

Visual-language Pretraining: Contrastive language-image pretraining learns visual or visual-language representations from web-scale image-text data and enables zero-shot transfer through natural language.Representative systems include CLIP, ALIGN, and Florence.
Visual-language Pretraining: VideoCLIP extends image-level pretraining using video-text pairs, but such pretraining is computationally expensive and requires substantial curated data.The related-work discussion contrasts this route with adapting pretrained image-language models.
Visual-language Pretraining: ActionCLIP uses a pretrain, prompt, and finetune framework, while another concurrent method optimizes random vectors to adapt CLIP to video tasks.The paper claims broader support for adapting models such as CLIP and Florence.
Video Recognition: Video recognition methods use 3D convolutions, factorized temporal modules, or transformers, with 3D convolutions noted for high computational cost.Transformer-based methods have recently achieved promising performance for temporal modeling.
Video Recognition: VTN, ViViT, and AVT encode frames separately before temporal aggregation, whereas this paper enables earlier cross-frame interaction through cross-frame attention.The paper characterizes separate frame encoding as a late-fusion strategy that does not fully use temporal cues.

3 Approach

The approach adapts pretrained language-image models to video recognition by combining cross-frame temporal modeling with video-specific text prompting. It aligns video representations with instance-level textual representations while retaining pretrained components and adding lightweight video modules.

3.1 Overview: The framework jointly trains video and text encoders to align video representations with corresponding text representations.A video-specific prompt generator uses both representations to produce instance-level textual features before cosine-similarity matching.
3.2 Video Encoder: The video encoder cascades a cross-frame communication transformer with a multi-frame integration transformer.The first exchanges information across frames and produces frame-level representations; the second integrates them into video features.
3.2 Video Encoder: The multi-frame integration transformer applies temporal position encoding, self-attention, feed-forward networks, and average pooling to produce the video representation.It processes the sequence of frame representations and outputs video-level features.
3.2 Video Encoder: Cross-frame attention exchanges information through message tokens using cross-frame fusion and intra-frame diffusion attention.Message tokens gather frame information, communicate global spatio-temporal dependencies, and diffuse them back into frame-token representations.
3.2 Video Encoder: The cross-frame communication transformer models global spatio-temporal information while reducing computational cost relative to other space-time attention mechanisms.Message tokens are generated online within each block and are discarded before the feed-forward layer.
3.3 Text Encoder: The video-specific prompting scheme uses video content to enhance text representations instead of relying on manually designed prompts.The method uses the label as the initial description and learns prompts conditioned on video information.

4 Experiments

Experiments evaluate X-CLIP and X-Florence across fully supervised, zero-shot, few-shot, ablation, pretraining, sampling, and inference settings. X-CLIP combines temporal frame communication, multi-frame integration, and video-specific prompting with strong accuracy and computational efficiency.

Zero-shot Experiments: +3.8% and +13.7% top-1 accuracy over previous best results are reported on HMDB-51 and UCF-101 in zero-shot recognition.On Kinetics-600, X-CLIP exceeds ER-ZSAR by +23.1%.
Few-shot Experiments: +32.1% and +23.1% top-1 accuracy over Swin-B are obtained on HMDB-51 and UCF-101 with K=2.The gains are attributed to exploiting semantic information in text representations under extremely limited labeled data.
Ablation and Analysis: Cross-frame communication improves accuracy by +1.2%, multi-frame integration adds +0.5%, and video-specific prompting adds +2.3% over CLIP-Mean.Together with multi-view inference, the method raises CLIP-Mean from 80.0% to 83.8%.
Ablation and Analysis: Sparse sampling performs best for X-CLIP, while single-view inference remains comparable to 10 temporal views at lower computational cost.The paper also reports that finetuning choices differ by supervision regime and that large-scale CLIP pretraining outperforms ImageNet pretraining.

5 Conclusion

The work adapts pretrained language-image models to video recognition using cross-frame attention and video-specific prompting, with effectiveness demonstrated across three learning scenarios.

The method directly adapts pretrained language-image models to video recognition rather than pretraining a new model from scratch.
Cross-frame attention explicitly exchanges information across frames to capture temporal information.
Video-specific prompting leverages video content to generate instance-level discriminative textual representations.
Extensive experiments under zero-shot, few-shot, and fully-supervised settings demonstrate the method's effectiveness.
The authors plan to extend the method beyond video classification to other video tasks.

Expanding Language-Image Pretrained Models for General Video Recognition ——–Supplementary Material——–

The supplementary material provides architectural details, hyperparameters, dataset protocols, and additional experiment analysis.

The supplement presents details of the proposed architectures and comparison methods.
It elaborates the experiments' hyperparameters and overviews four datasets with their evaluation protocols.
Additional experiment analysis is provided in the final supplementary section.

1 Architecture Details

The architectures combine pretrained visual encoders with temporal integration components, using different configurations for X-CLIP, Florence, and few-shot comparison models.

Architecture Details: The proposed-architecture section describes the proposed models and the architectures compared in few-shot experiments.
The proposed architectures: X-CLIP variants use ViT-B/32, ViT-B/16, or ViT-L/14 within the cross-frame communication transformer and a 1-layer multi-frame integration transformer.
The proposed architectures: For Florence, the cross-frame communication transformer is replaced by CoSwin-H, followed by a 4-layer multi-frame integration transformer.
Compared architectures: The few-shot comparison models include Video Swin, TSM, and TimeSformer implemented with MMAction2 default hyperparameters.

2 Hyperparameter Details

The supplementary hyperparameter section documents training settings, prompting templates, and the corresponding hyperparameter table.

Hyperparameter Details: This section presents elaborated training hyperparameters and the hand-crafted prompt templates used in the experiments.
Hyperparameter Details: Table 1 summarizes the training hyperparameters for all experiments.
Hyperparameter Details: The learning rate for randomly initialized parameters is 10× higher than the base learning rate.
Hand-craft Prompt Templates: The video-specific prompting scheme is compared with prompt ensembling using 16 hand-crafted templates, with one selected per training iteration.
Hand-craft Prompt Templates: Inference averages the results from all 16 prompt templates.

3 Datasets and Evaluation Protocols

The evaluation covers four video-recognition datasets and three training regimes: zero-shot, few-shot, and fully supervised. Dataset scale and protocol details vary across Kinetics-400/600, UCF-101, and HMDB-51.

Datasets Overview: Kinetics-400 contains approximately 240k training and 20k validation videos across 400 classes.
Datasets Overview: Kinetics-600 contains approximately 410k training and 29k validation videos across 600 classes.
Datasets Overview: UCF-101 contains 13,320 YouTube video clips spanning 101 realistic action categories and three train/test splits.
Datasets Overview: HMDB-51 contains around 7,000 videos across 51 classes and three training/test splits.
Evaluation Protocols: Fully supervised experiments use Kinetics-400/600 with sparse sampling, 8 or 16 frames, 224 crops, and 3 spatial crops across 4 temporal clips.
Evaluation Protocols: Few-shot experiments sample 2, 4, 8, or 16 videos per class on UCF-101 and HMDB-51, using the first test split and single-view 32-frame evaluation.
Evaluation Protocols: Zero-shot experiments use X-CLIP-B/16 with 32 frames and evaluate HMDB-51/UCF-101 across three splits plus 220 unseen Kinetics-600 categories.

4 Additional Experiments Analysis

Additional experiments compare image-to-video adaptation methods, embedding alignment, and the proposed temporal modules. The analyses include benchmark comparisons and a single-modality classification study.

Experimental Scope: The analysis compares image-to-video encoder adaptation methods, embedding alignment, and the cross-frame and multi-frame transformers.
Benchmark Comparison: X-CLIP records 70.0 zero-shot, 50.8 few-shot, and 82.3 fully-supervised performance with 145 FLOPs in the reported comparison.
Encoder Adaptation: CLIP-One samples one frame, whereas CLIP-Joint forwards all spatiotemporal tokens through the image encoder using joint space-time attention.
Single-Modality Analysis: Table 4 evaluates the proposed architecture in the single-modality framework.
Embedding Alignment: The ImageNet-pretrained visual encoder and CLIP-pretrained text encoder are not well aligned, motivating separate analysis of embedding alignment for zero-shot transfer.
Single-Modality Analysis: A single-modality Kinetics-400 study trains from scratch with ViT-B/32, comparing frame averaging against cross-frame attention plus a one-layer multi-frame integration transformer.