PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-tuned CLIP Models are Efficient Video Learners

Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, Fahad Shahbaz Khan

arXiv:2212.03640v3cs.CVcs.AI

TL;DR

Video adaptation of image-pretrained CLIP is difficult because video-text pretraining is costly and added temporal modules can require careful design while threatening generalization. The paper proposes simple video fine-tuning through ViFi-CLIP, with bridge and prompt for low-data regimes, and reports favorable performance across multiple video recognition settings.

  • Problem

    Adapting image-level CLIP representations to videos is challenging because large-scale aligned video-text pretraining is costly, while existing video-specific components require careful design and may compromise generalization.

  • Method

    ViFi-CLIP fine-tunes CLIP on videos using frame-level encoding and temporal average pooling, while bridge and prompt adds visual-language prompt learning after fine-tuning for low-data regimes.

  • Results

    ViFi-CLIP performs favorably or competitively across zero-shot, base-to-novel, few-shot, and fully supervised video settings, while prompting remains competitive with prior methods.

  • Takeaways & Limitations

    Simple fine-tuning can bridge the image-to-video domain gap, and video-fine-tuned representations can support downstream adaptation when full fine-tuning is unavailable.

  • Takeaways & Limitations

    Full fine-tuning of CLIP is not always feasible, especially in low-data regimes.

Abstract

from arXiv · show

Large-scale multi-modal training with image-text pairs imparts strong generalization to CLIP model. Since training on a similar scale for videos is infeasible, recent approaches focus on the effective transfer of image-based CLIP to the video domain. In this pursuit, new parametric modules are added to learn temporal information and inter-frame relationships which require meticulous design efforts. Furthermore, when the resulting models are learned on videos, they tend to overfit on the given task distribution and lack in generalization aspect. This begs the following question: How to effectively transfer image-level CLIP representations to videos? In this work, we show that a simple Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to bridge the domain gap from images to videos. Our qualitative analysis illustrates that the frame-level processing from CLIP image-encoder followed by feature pooling and similarity matching with corresponding text embeddings helps in implicitly modeling the temporal cues within ViFi-CLIP. Such fine-tuning helps the model to focus on scene dynamics, moving objects and inter-object relationships. For low-data regimes where full fine-tuning is not viable, we propose a `bridge and prompt' approach that first uses fine-tuning to bridge the domain gap and then learns prompts on language and vision side to adapt CLIP representations. We extensively evaluate this simple yet strong baseline on zero-shot, base-to-novel generalization, few-shot and fully supervised settings across five video benchmarks. Our code is available at https://github.com/muzairkhattak/ViFi-CLIP.

1. Introduction

The paper asks whether simple CLIP fine-tuning can transfer image-level representations to videos without sacrificing generalization. It proposes ViFi-CLIP and a two-stage bridge-and-prompt strategy for low-data settings.

  • Motivation: Image-text pretraining gives CLIP strong transfer, but video adaptation is constrained by scarce aligned video-text data and high computational cost.The paper motivates adapting pretrained image-language models without forgetting their generic multimodal representations.
  • Motivation: Existing video adaptations add temporal modules, prompts, or decoders that require careful modality-specific design and may perform inconsistently across supervision settings.The cited examples include cross-frame self-attention, textual or visual prompts, and dedicated video decoders.
  • Core finding: Fine-tuning CLIP encoders with temporal components can hinder generalization, whereas simple video fine-tuning performs competitively with more complex approaches.This empirical observation motivates testing whether additional video-specific parameters are necessary.
  • Mechanism: Late aggregation of frame-level representations before loss calculation enables temporal-cue exchange within video-fine-tuned CLIP.The mechanism is presented as an explanation for how regular CLIP acquires video capacity without dedicated temporal modules.
  • Contributions: For low-data regimes, bridge and prompt first fine-tunes vanilla CLIP on videos, then learns visual and language prompts while keeping the tuned CLIP frozen.This two-stage design addresses cases where full fine-tuning is not feasible.
  • Contributions: ViFi-CLIP adapts image-based CLIP to video tasks through simple fine-tuning that learns video-specific inductive biases.The paper evaluates this baseline across zero-shot, base-to-novel, few-shot, and fully supervised settings.

2. Related Work

Related work establishes CLIP and ALIGN as broadly transferable vision-language models, while video methods add temporal components to compensate for missing video-specific cues. The paper positions simple fine-tuning and prompting as alternatives whose comparative benefit remains uncertain.

  • Vision Language models: CLIP and ALIGN learn open-vocabulary multimodal representations from large-scale image-caption pairs using contrastive self-supervision.These representations transfer to few-shot, zero-shot, detection, and segmentation applications.
  • Vision Language models: Image-level pretraining lacks video-specific temporal cues, making adaptation of pretrained vision-language models to videos challenging.Recent methods address this gap with additional learnable temporal or prompting components.
  • Video adaptation: Video adaptations such as ActionCLIP, XCLIP, and Ju et al. combine CLIP representations with components for temporal modeling.These methods use CLIP’s generalized vision-language representations for video understanding.
  • Prompt Learning: Prompt learning transfers models by adding learnable input tokens, aiming to preserve generalization while adapting downstream representations.The related work discusses text, visual, and multimodal prompting approaches.
  • Video adaptation: Temporal modeling in Ju et al. uses text prompts and transformer layers but hinders CLIP generalization and struggles in zero-shot evaluation.This passage identifies a specific generalization concern associated with one video adaptation approach.

3. Problem Settings

The evaluation varies supervision from cross-dataset zero-shot transfer through base-to-novel, few-shot, and fully supervised learning. The paper also defines a frequency-based base-novel split and evaluates few-shot learning with multiple shot counts.

  • Zero-shot setting: In zero-shot evaluation, a model trained on source data transfers directly to a downstream target dataset with disjoint classes.The source passage introduces this setting before describing the remaining supervision regimes.
  • Base-to-novel generalization: Base-to-novel generalization trains on base classes and evaluates on both base and novel classes.The class sets are disjoint and together partition the source categories.
  • Base-to-novel generalization: The base-to-novel split divides categories into two equal halves, assigning the most frequent classes to the base set.Figure 2 shows the resulting splits for Kinetics-400 and SSv2.
  • Few-shot setting: Few-shot evaluation randomly samples K examples from every category for training, using K = 2, 4, 8, and 16 shots.The validation set is used for evaluation.
  • Fully supervised setting: Fully supervised evaluation trains on all training examples and evaluates on the corresponding test set.This is described as the conventional supervised setting.

4. Video Finetuned CLIP

ViFi-CLIP adapts image-based CLIP to videos through full fine-tuning and simple frame-level temporal pooling, avoiding additional video-specific components. Across zero-shot, base-to-novel, few-shot, and fully supervised settings, it achieves strong generalization and competitive performance.

  • Video Finetuned CLIP: ViFi-CLIP adapts CLIP to videos by fine-tuning both image and text encoders with minimal architectural changes.The approach is designed to bridge the image-video modality gap without adding modality-specific components.
  • Video Finetuned CLIP: The image encoder processes frames independently, average-pools their embeddings into a video representation, and matches it with prompted text embeddings.Temporal pooling implicitly incorporates temporal learning by aggregating multiple frame-level embeddings before similarity-based training.
  • ViFi-CLIP Generalizes Well!: In zero-shot evaluation, ViFi-CLIP improves over prior best methods by +6.7% on HMDB-51, +4.8% on UCF-101, and +4.5% on Kinetics-600.The model is trained on Kinetics-400 and evaluated directly on the other datasets.
  • ViFi-CLIP Generalizes Well!: ViFi-CLIP achieves the best overall harmonic mean across Kinetics-400, HMDB-51, UCF-101, and SSv2 in base-to-novel generalization.It provides a better trade-off between base and novel accuracy than approaches using additional video-specific components.
  • CLIP directly adapts to Video tasks: In few-shot settings, ViFi-CLIP outperforms compared methods across HMDB-51, UCF-101, and SSv2 for K = 2, 4, 6, and 8 shots.Reported gains over prior best methods include +9.3% on UCF-101 and 4.2% on HMDB-51.
  • CLIP directly adapts to Video tasks: ViFi-CLIP remains competitive in fully supervised evaluation while achieving 71.1 images/sec throughput and lower FLOPs and training parameters than other adapted methods.The efficiency is attributed to avoiding additional video-specific components.

5. Bridge and Prompt in low-data regimes

The bridge-and-prompt framework first fine-tunes CLIP on videos, then freezes it while learning vision-language prompts for efficient low-data adaptation. It improves few-shot and base-to-novel transfer while maintaining competitive efficiency and generalization.

  • 5. Bridge and Prompt in low-data regimes: ViFi-CLIP fine-tunes CLIP on videos to bridge the image-to-video modality gap before downstream adaptation.This stage is used because full fine-tuning may be infeasible in low-data regimes.
  • 5. Bridge and Prompt in low-data regimes: The second stage freezes the tuned CLIP and learns prompts to steer it toward downstream video tasks.The framework is explicitly designed to improve effectiveness and computational efficiency when training data are limited.
  • 5.1. Prompting is effective on fine-tuned CLIP: VL prompting improves base-class generalization while also improving novel-class performance and remains competitive with fine-tuning methods.All compared models are first pretrained on Kinetics-400.
  • 5. Bridge and Prompt in low-data regimes: Vision-language prompting learns prompt vectors in both the vision and language branches rather than only in the language branch.Prompts are also inserted at deeper layers to capture hierarchical contextual representations.
  • 5.1. Prompting is effective on fine-tuned CLIP: In few-shot transfer, VL prompting consistently outperforms A5 and remains competitive with fine-tuning approaches.The comparison follows pretraining on Kinetics-400 and evaluation on few-shot transfer and base-to-novel generalization.
  • 5.1. Prompting is effective on fine-tuned CLIP: For K = 2, VL prompting gains 2.5% over XCLIP on HMDB-51 and 2% over XCLIP on UCF-101.These are absolute gains reported for the extreme few-shot cases.
  • 5.2. Is prompting efficient w.r.t CLIP adaptation?: VL prompting offers higher throughput with comparable GFLOPs relative to prior CLIP adaptation approaches.A5 uses fewer FLOPs but achieves lower throughput because of additional temporal transformer blocks.

6. Conclusion

The conclusion identifies simple fine-tuning of both CLIP encoders as a strong approach for transferring image-based CLIP to videos. When fine-tuning is infeasible, bridge and prompt enables quick downstream adaptation using video-fine-tuned representations.

  • 6. Conclusion: Fine-tuning both the vision and text encoders on video data performs favourably on supervised and generalization tasks.The authors report scalability and advantages over sophisticated video-specific approaches in most settings.
  • 6. Conclusion: When fine-tuning is not possible, bridge and prompt uses video-fine-tuned representations to quickly adapt CLIP to downstream video applications.

Supplemental Material

The supplementary material provides additional task evaluations, implementation details, evaluation protocols, and qualitative results.

  • Supplemental Material: Appendix A covers other video-level tasks.
  • Supplemental Material: Appendix B provides additional implementation details.
  • Supplemental Material: Appendix D describes the evaluation protocols.
  • Supplemental Material: Appendix E presents additional qualitative results.

A. Other video-level tasks

The supplementary evaluation scales ViFiCLIP to video retrieval on MSRVTT (9K) under settings matched to A5 and reports improved performance.

  • A. Other video-level tasks: On MSRVTT (9K) video retrieval, ViFiCLIP uses 16 frames with ViT-B/16 and achieves improved performance over the comparison setting.The evaluation uses consistent settings with A5 and reports results in Table 10.

B. Implementation Details

The study evaluates ViFi-CLIP and its variants across multiple video benchmarks and supervision settings using specified optimization and preprocessing choices.

  • Preprocessing: All sampled frames are resized to 224×224, while text uses the handcrafted template ‘a photo of a <category>’.The maximum number of text tokens is 77.
  • Experimental scope: Experiments include zero-shot, base-to-novel, few-shot, and fully supervised settings across five action-recognition benchmarks.The benchmarks are Kinetics-400, Kinetics-600, HMDB-51, UCF-101, and Something-Something v2.
  • Optimization: Zero-shot training on Kinetics-400 uses 10 epochs, batch size 256, and learning rate 8e-6.Few-shot and base-to-novel settings use batch size 64 and learning rate 2e-6.
  • Baselines: Baseline implementations include A6, ActionCLIP, and XCLIP with their reported default optimal hyperparameters.Efficient prompting uses A6 for fully supervised evaluation and A5 for other settings; ActionCLIP uses Transf throughout.
  • Datasets: Kinetics-400 contains 400 action classes with about 240K training and 20K validation videos, whereas Kinetics-600 contains 600 categories with about 410K training and 29K validation videos.The clips last about 10 seconds and are sourced from YouTube videos.
  • Datasets: HMDB-51 contains 71K videos across 51 categories, and UCF-101 contains 13K videos across 101 categories.HMDB-51 uses 3570 training and 1530 validation samples in its standard split; UCF-101 uses 9537 training and 3783 evaluation videos.
  • Datasets: Something-Something v2 contains 174 fine-grained action categories involving humans and everyday objects.Its standard split contains 168,913 training and 24,777 validation videos, with top-1 validation accuracy reported.

D. Evaluation Protocols

Evaluation varies supervision, class novelty, sampling, and inference procedures to test transfer and generalization across video-recognition settings.

  • Common protocol: The protocol uses sparse frame sampling with either 16 or 32 frames across zero-shot, base-to-novel, few-shot, and fully supervised evaluations.The number of frames is specified by setting.
  • Cross-dataset transfer: Kinetics-600 visualizations use 160 classes that do not overlap with Kinetics-400, evaluating models trained on Kinetics-400.This protocol directly examines transfer to classes absent during training.
  • Preprocessing: Each sampled frame is spatially scaled on its shorter side to 256 before a center crop of 224.
  • Zero-shot setting: Zero-shot models trained on Kinetics-400 are evaluated on HMDB-51, UCF-101, and non-overlapping Kinetics-600 classes.Top-1 averages are reported for HMDB-51 and UCF-101, while Kinetics-600 also reports top-5 averages.
  • Base-to-novel setting: Base-to-novel evaluation trains on few-shot base classes and tests on unseen classes across Kinetics-400, HMDB-51, UCF-101, and SSv2.Three training splits are created for each dataset.
  • Few-shot setting: Few-shot evaluation samples 2, 4, 8, or 16 examples per category from HMDB-51, UCF-101, and SSv2.It uses 32 sparsely sampled frames and single-view inference.
  • Fully supervised setting: Fully supervised evaluation trains on Kinetics-400 and uses 16 frames with three spatial crops and four temporal clips.Performance is measured on the complete Kinetics-400 validation set using multi-view inference.

E. Additional qualitative results

Additional visual analyses show that ViFi-CLIP produces more separable embeddings and attends to temporal interactions, moving elements, and scene dynamics, including unusual examples.

  • Cluster metrics: Table 11 reports homogeneity, completeness, and V-measure for t-SNE clusters from vanilla CLIP, XCLIP, and ViFi-CLIP.
  • Embedding separability: ViFi-CLIP gains +6% on Kinetics-600 relative to XCLIP in the zero-shot comparison.The models are trained on Kinetics-400 and evaluated on non-overlapping Kinetics-600 classes.
  • Attention maps: Figure 8 compares ViFi-CLIP with vanilla CLIP on HMDB-51 and UCF-101 attention maps, illustrating differences in attended regions.
  • Attention maps: Fine-tuning shifts attention toward inter-object relationships, scene dynamics, moving parts, and objects relevant to the action.Examples include a moving ball, a child, fast-moving legs, and singer–guitar interaction.
  • Temporal cues: The visualizations support implicit modeling of temporal relations through simple fine-tuning of CLIP on video data.
  • Out-of-distribution examples: On synthetic and rare out-of-distribution videos, ViFi-CLIP attends to object interactions and combines appearance with temporal cues.Examples include skating shoes with a teddy-bear and a giraffe tight-roping.
Loading 2212.03640v3…