PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompting Visual-Language Models for Efficient Video Understanding

Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, Weidi Xie

arXiv:2112.04478v2cs.CVcs.CL

TL;DR

The paper addresses how to adapt powerful image-based visual-language models to resource-hungry video understanding efficiently. It learns continuous prompt vectors and lightweight temporal Transformers, achieving competitive or state-of-the-art results across video benchmarks and scenarios while training few parameters.

  • Problem

    Efficiently adapting image-based visual-language models to resource-hungry video tasks is challenging because video-text data is harder to collect and video processing requires more computation.

  • Method

    The method learns task-specific continuous prompt vectors while freezing the I-VL text encoder, and adds lightweight temporal Transformers over frame-wise visual features.

  • Results

    The approach achieves competitive or state-of-the-art closed-set performance and significantly outperforms existing methods on 7 public benchmarks in few-shot and zero-shot scenarios.

  • Takeaways & Limitations

    Prompt learning offers an efficient way to adapt I-VL models for action recognition, action localisation, and text-video retrieval across closed-set, few-shot, and zero-shot settings.

  • Takeaways & Limitations

    Zero-shot action localisation performance drops dramatically because class-agnostic proposals can be biased toward seen action distributions and second-stage classification errors remain.

Abstract

from arXiv · show

Image-based visual-language (I-VL) pre-training has shown great success for learning joint visual-textual representations from large-scale web data, revealing remarkable ability for zero-shot generalisation. This paper presents a simple but strong baseline to efficiently adapt the pre-trained I-VL model, and exploit its powerful ability for resource-hungry video understanding tasks, with minimal training. Specifically, we propose to optimise a few random vectors, termed as continuous prompt vectors, that convert video-related tasks into the same format as the pre-training objectives. In addition, to bridge the gap between static images and videos, temporal information is encoded with lightweight Transformers stacking on top of frame-wise visual features. Experimentally, we conduct extensive ablation studies to analyse the critical components. On 10 public benchmarks of action recognition, action localisation, and text-video retrieval, across closed-set, few-shot, and zero-shot scenarios, we achieve competitive or state-of-the-art performance to existing methods, despite optimising significantly fewer parameters.

1 Introduction

The paper asks how to adapt powerful image-based visual-language models to novel, resource-intensive video tasks without sacrificing broad generalisation. It proposes continuous prompt vectors and lightweight temporal modeling, achieving strong results across multiple settings with few trainable parameters.

  • Adapting I-VL models to novel vision tasks requires exploiting their general-purpose representations with minimal tuning.
  • Video understanding is resource-hungry because video-text data are harder to collect, video training requires more computation, and videos contain temporal dependencies.
  • Continuous prompt vectors are learnable virtual tokens that generate task-specific classifiers or embeddings while the pretrained text encoder remains frozen.
  • A lightweight temporal Transformer complements frame-wise visual features, while video tasks share a visual-textual similarity-matching formulation.
  • 10 public benchmarks cover action recognition, action localisation, and text-video retrieval across closed-set, few-shot, and zero-shot scenarios.
  • In few-shot and zero-shot scenarios, the method significantly outperforms previous methods on 7 benchmarks, sometimes by over 10% gains.

2 Related Work

Prior work established joint image-text learning, prompting, and specialized video understanding methods. This paper combines prompt-based transfer from CLIP with lightweight temporal modeling to provide a broad, efficient video baseline.

  • Joint Visual-Textual Learning: CLIP, ALIGN, and FILIP scale joint image-text representation learning using large-scale web data and noise contrastive learning.
  • Prompting: Continuous prompting automates prompt design, avoiding the expert knowledge required by handcrafted templates.
  • Video Action Recognition: Video action recognition research spans two-stream and single-stream RGB architectures, including data-efficient few-shot and zero-shot settings.
  • Video Action Localisation: Action localisation commonly uses either two-stage proposal classification or one-stage dense frame classification.
  • Concurrent Work: Unlike concurrent approaches that end-to-end finetune CLIP for individual video tasks, this paper favors efficient image-to-video adaptation through prompt learning.

3 Method

The method adapts frozen CLIP encoders to video tasks by learning continuous textual prompts and adding lightweight temporal modeling. It unifies classification, localisation, and retrieval through visual-textual similarity matching while supporting closed-set and zero-shot scenarios.

  • Model Adaptation: The framework adapts a pretrained I-VL model to action recognition, action localisation, and text-video retrieval using one shared backbone.
  • Pre-training: CLIP pretraining maximizes similarity for matched image-text pairs and minimizes it for mismatched pairs using symmetrical cross-entropy.
  • Inference: At inference, the text encoder generates open-vocabulary visual classifiers from category names and a prompt template.
  • Discussion: Handcrafted prompt sensitivity motivates automated prompt design for efficient adaptation with minimal training.
  • Temporal Modeling: Videos are represented as frame sequences, motivating temporal modeling on top of powerful image-based representations.
  • Model Adaptation by Learning Prompts: Prompt vectors are task-specific and shared across action categories, while both CLIP encoders remain frozen during training.
  • Action Localisation: Action localisation uses a two-stage pipeline that first detects class-agnostic proposals and then classifies the detected proposals.
  • Text-Video Retrieval: The unified objective maximizes similarity between normalized visual and textual embeddings, closing the optimization gap between CLIP pretraining and video understanding.

4 Experiments

Experiments evaluate prompt learning and lightweight temporal modeling across action recognition, action localisation, and retrieval, spanning closed-set, few-shot, and zero-shot settings. The method generally improves over handcrafted-prompt baselines while using relatively few trainable parameters, though zero-shot localisation remains substantially harder.

  • Experimental setup: Experiments cover three video tasks across ten standard datasets, including closed-set, few-shot, and zero-shot evaluation settings.The tasks are action recognition, action localisation, and text-video retrieval.
  • Action recognition ablations: 4.9% and 5.3% average gains on K-400 and K-700 follow from adding temporal modeling, while additional Transformer layers yield diminishing returns.The ablation identifies prompt learning and temporal modeling as important components, with later experiments using two Transformer layers for the performance–cost trade-off.
  • Action recognition ablations: 18.1% over Baseline-I and 4.4% over Baseline-II are achieved by A3 with only 16K trainable parameters in the reported ablation.A3 uses 16 prompt vectors and two temporal Transformer layers in the described best-practice configuration.
  • Few-shot and zero-shot recognition: Around 10% gains on HMDB-51 and K-400 are reported in few-shot action recognition, while C-way evaluation remains more challenging than 5-way classification.The method outperforms previous methods by a significant margin in the reported few-shot comparison; gains over Baseline-I on UCF-101 and HMDB-51 are around 15% in the C-way setting.
  • Few-shot and zero-shot recognition: 6.0% TOP1 accuracy gains over Baseline-I are reported for zero-shot recognition, although adding temporal modeling diminishes the gain on unseen action categories.The paper conjectures that the additional Transformer layer may specialise on the training set and harm generalisation.
  • Action localisation: Around 5.2% and 2.5% average mAP gains are reported for closed-set action localisation, whereas zero-shot localisation drops dramatically because proposal recall and classification errors both contribute.The zero-shot setting trains on disjoint action categories, with proposal bias toward seen distributions identified as one source of the drop.

5 Conclusion

The paper builds efficient video-understanding baselines from CLIP using lightweight prompt vectors and temporal Transformers, evaluating them across ten benchmarks and multiple supervision settings.

  • The method learns lightweight prompt vectors and temporal Transformers to adapt CLIP for efficient video understanding.
  • It evaluates action recognition, action localisation, and text-video retrieval on 10 public benchmarks.
  • In closed-set settings, the method achieves competitive performance despite training only a few free parameters.
  • In few-shot and zero-shot settings, it significantly outperforms existing methods on 7 public benchmarks.

A Implementation Details

The implementation freezes the pretrained CLIP encoders and uses fixed-dimensional features, while randomly initializing the learnable prompts and temporal Transformer.

  • Both pretrained CLIP image and text encoders are kept frozen during adaptation.
  • Prompt vectors and visual features use dimension D = 512, and the temperature hyper-parameter τ is 0.07.
  • Prompt vectors and the temporal Transformer are randomly initialized from a zero-mean Gaussian with standard deviation 0.01.
  • At inference, action recognition and text-video retrieval sample 16 frames five times and average the five results.

A.2 Action Localisation

For action localisation, the system uses off-the-shelf proposal detectors and a proposal-classification pipeline built from frame embeddings, temporal features, prediction heads, and Soft-NMS.

  • Class-agnostic action proposals are obtained with off-the-shelf proposal detectors.
  • The pipeline divides each video into equal-frame segments and extracts frame-wise embeddings with the CLIP image encoder and one Transformer layer.
  • A 6-layer feature pyramid and three parallel heads predict actionness, centerness, and boundaries.
  • Prediction results are assembled and Soft-NMS suppresses redundant proposals.

A.3 Text-Video Retrieval

The retrieval setup samples sparse long-range video frames, while the localisation tables distinguish closed-set from category-disjoint zero-shot evaluation settings.

  • Text-Video Retrieval: Text-video retrieval uses 16-frame inputs with random frame gaps of 10, 15, or 30.
  • Text-Video Retrieval: These retrieval inputs correspond to sampling at approximately 1–3 fps and use lower fps because retrieval can require long-term visual dependencies.
  • Action Localisation: Proposal detection compares closed-set evaluation with zero-shot settings that train on 75% or 25% of categories and test on the remaining 25% or 50%.
  • Action Localisation: Proposal classification uses the same-category setup for closed-set evaluation and disjoint category splits for zero-shot evaluation.

B Experiment and Analysis

The section presents additional results that analyse the method and investigate the semantics learned by its prompt vectors.

  • Additional results further analyse the proposed method.
  • The analysis explores semantic information learned by the prompt vectors.
  • The section extends the empirical examination beyond the earlier results.

B.1 Action Localisation

Action localisation uses proposal detection followed by proposal classification, with separate closed-set and zero-shot evaluations. Zero-shot performance drops because both proposal recall and classification contribute errors, while retrieval benefits from prompt vectors despite minimal adaptation.

  • Action Localisation: Action localisation follows a two-stage pipeline: proposal detection first, then proposal classification.The stages are evaluated separately in closed-set and zero-shot scenarios.
  • Proposal Detection: Zero-shot proposal performance decreases on both datasets, indicating proposals remain biased toward seen action categories.The drop is more significant on THUMOS14 because its videos contain denser action instances than ActivityNet1.3.
  • Proposal Classification: Zero-shot classification accuracy generally drops relative to closed-set evaluation.The 75%-category THUMOS14 setting is an exception because only five testing categories remain, making classification easier.
  • Summary: Zero-shot localisation degradation comes from both first-stage proposal-recall loss and second-stage classification errors.
  • Text-Video Retrieval: The retrieval experiments use 8 learnable prompt vectors and two Transformer layers for efficient adaptation.The prompt length is constrained because CLIP accepts at most 77 textual tokens while retrieval queries can be long.
  • Text-Video Retrieval: Learnable prompt vectors largely outperform the no-prompt CLIP baseline and sometimes approach retrieval methods designed specifically for that task.
  • Prompt Semantics: Nearest-vocabulary subwords usually lack meaningful semantics, consistent with continuous prompts extending beyond the discrete vocabulary.The CLIP vocabulary contains 49,408 subwords, which limits interpretation of the learned prompt semantics.

C Limitations

The method depends on image alt-text pre-training, which may introduce web-data bias and may inadequately capture fine-grained motion because temporal modeling is applied only to visual features.

  • Image alt-text pre-training may introduce bias from the web data.
  • Temporal modeling applied only on top of visual features may fail to model fine-grained motions.

D Dataset Splits

The experiments define few-shot, zero-shot, and category-split evaluations across several video benchmarks using repeated random sampling. Localisation splits vary the proportions of training and testing categories, with multi-label videos divided by category.

  • Few-Shot Splits: The few-shot 5-Shot-5-Way setting samples 5 action categories with 5 videos per category and runs 200 random trials.
  • Few-Shot Splits: The 5-Shot-C-Way setting samples 5 videos from every category and evaluates on the standard testing set across 10 random rounds.
  • Zero-Shot Action Recognition: Zero-shot K-700 evaluation trains on 400 disjoint action categories and tests on the remaining 300 categories.
  • Zero-Shot Action Localisation: When an untrimmed video contains both training and testing categories, it is divided into separate training-only and testing-only videos.
Loading 2112.04478v2…