Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Data

1 INTRODUCTION

• Fine-tuning CLIP for video can reduce generalization because video datasets are smaller than CLIP’s original training data, diminishing its zero-shot ability.The introduction contrasts adding temporal modeling with the risk of overfitting during fine-tuning.Source: §1 · p. 1
• The paper frames adaptation as continual learning without access to CLIP’s historical training data, while seeking generalization to unseen video actions and events.Unlike the usual goal of preserving performance on earlier tasks, the objective is transfer from image to video zero-shot learning.Source: §1 · pp. 1–2
• Open-VCLIP++ uses minimal CLIP modifications, Interpolated Weight Optimization to connect original and current weights, and language-model pseudo-captions to enrich video-training text.The authors say this regularization prevents drift from CLIP and that more descriptive captions improve the transition from image to video.Source: §1 · p. 2
• 88.1% UCF, 58.7% HMDB, and 81.2% Kinetics-600 zero-shot accuracy surpass alternative methods by 8.5%, 8.2%, and 12.3%, respectively.On MSR-VTT, Open-VCLIP++ reports 39.0% text-to-video recall and 37.8% video-to-text recall without additional manually labeled video-text pairs.Source: §1 · p. 2
• The paper extends a preliminary version with a literature review, pseudo-caption alignment to prevent forgetting, broader experiments, compatibility analyses, and image-classification tests of retained CLIP knowledge.The image-classification experiments are described as demonstrating maintenance of CLIP’s original image-task knowledge.Source: §1 · p. 2

Zero-Shot Video Action Recognition

• Earlier zero-shot action methods use attributes, object features, or word embeddings, while recent CLIP-based video approaches overlook forgetting during adaptation [15].The paper positions its goal as regularizing fine-tuning to build a robust open-vocabulary zero-shot video classifier.Source: §2 · p. 2

Continual Learning

• Continual-learning methods include replay buffers, network expansion, and weight regularization, but replay and EWC require historical data unavailable for this CLIP transfer setting [17] [18] [27] [28].The paper instead targets transfer of image-task zero-shot ability to video without historical data.Source: §2 · p. 2

Captioning with Large Language Models

• Prior LLM captioning uses fine-grained descriptions or interactive feedback [34] [36] [37], whereas this work also uses detailed video captions to enhance multimodal alignment.The related-work section describes caption enrichment as including objects, attributes, and frame-level captions in prior approaches.Source: §2 · p. 3

3 PRELIMINARY: VIDEO ACTION RECOGNITION USING CLIP

• Video action recognition fine-tunes an image encoder for temporal dynamics and aligns video representations with text prompts describing action labels [43].The objective is to maximize similarity between a video and its corresponding text label.Source: §3 · p. 3
• The video dataset and its action-label text set define the fine-tuning task, with visual and text encoders producing their respective representations.The passage denotes these as DB, TB, fθV B, and fθT B.Source: §3 · p. 3
• The text encoder is usually frozen during training, so fine-tuning primarily optimizes the visual encoder for video-domain adaptation [45].The superscript notation is omitted in subsequent paragraphs for brevity.Source: §3 · p. 3

4 OUR APPROACH

• Open-VCLIP++ combines temporal modeling, regularization for unseen-category recognition, and detailed video captions to align visual and textual features.These are the approach’s three stated components.Source: §4 · p. 3

4.1 Constructing VCLIP for Video Understanding

• VCLIP expands each self-attention layer’s temporal view to include adjacent frames’ keys and values, enabling patches to aggregate information across neighboring frames without extra parameters.The original layer draws information only from the corresponding frame; the modification broadens its key/value matrices to adjacent frames.Source: §4.1 · p. 3

4.2 Training Open-VCLIP

• The training strategy addresses the loss of zero-shot performance that can accompany fine-tuning CLIP for close-set video classification.The paper presents optimization as a way to adapt CLIP while retaining generalization to novel actions and events.Source: §4.2 · p. 3, §4.2.1 · p. 4

4.2.1 Problem Definition

• The problem definition approximates an ideal objective over a hypothetical universal video-text dataset with training on a comprehensive dataset spanning sufficiently broad video and text data.The universal dataset is defined as containing all possible videos and their corresponding text descriptions.Source: §4.2.1 · p. 4

4.2.2 An Approximation Equivalent to Continual Learning

• The authors frame video fine-tuning as continual learning without historical data, because CLIP’s original training examples are unavailable and video labels cover a restricted text space.They motivate preserving CLIP’s prior generalization while adapting it to the video domain.Source: §4.2.2 · p. 4, §4.2.3 · p. 4

4.2.3 Interpolated Weight Regularization

• Interpolated Weight Regularization trains on interpolated CLIP and video-model weights across sampled mixing coefficients to regularize performance on the video dataset.The sampled coefficient spans a range of interpolation weights, addressing the risk that a single patched model underfits or depends sensitively on its mixing parameter.Source: §4.2.3 · p. 4

4.2.4 Stochastic Weight Averaging

• Stochastic Weight Averaging averages model weights along training trajectories; the authors use this averaging with interpolated weights to seek a flat minimum and improve generalization to unseen classes.They state that averaging interpolated weights and interpolating averaged updated weights are interchangeable, so they maintain a moving average during training and interpolate at the end.Source: §4.2.4 · p. 5

4.3 Expanding Text Space for Better Alignment.

• BLIP-2 frame descriptions are aggregated by Llama-2 into coherent video captions, which are aligned with video features alongside action-label text.The captioning pipeline samples 16 frames per video; the caption loss adds detailed descriptions to the original label-based alignment.Source: §4.3 · pp. 5–6

• The evaluation compares Open-VCLIP++ with state-of-the-art video models and uses ablation studies to examine its characteristics.Source: §5 · p. 6

5.1.1 Datasets

• Kinetics-400 is used for training, with zero-shot testing on UCF-101, HMDB-51, and a subset of Kinetics-600.Kinetics-600 adds 220 categories beyond Kinetics-400, while UCF-101 and HMDB-51 provide the other evaluation datasets.Source: §5.1.1 · p. 6

5.1.2 Evaluation protocols

• UCF-101 and HMDB-51 use three evaluation protocols, while Kinetics-600 uses three splits of its 220 new classes.The protocols include repeated half-class testing, complete-dataset testing, and official-split testing; Table 1 covers UCF and HMDB protocols, Table 2 covers UCF and HMDB Protocol 3, and Table 3 covers Kinetics-600.Source: §5.1.2 · pp. 6–7, Table 1 · p. 7

5.1.3 Implementation Details

• Training fine-tunes on Kinetics-400 for 20 epochs after two warm-up epochs, using 8 GPUs with batches of 8 samples.Videos use 8 frames sampled at stride 16; testing uses 3 clips and 1 crop per video, and video captions are generated with BLIP-2 XXL and Llama-2 7B chat.Source: §5.1.3 · p. 7

5.2.1 Comparison to state-of-the-arts action recognition

• 88.1% versus 79.6% on UCF and 58.7% versus 49.8% on HMDB: ViT-L/14 Open-VCLIP++ outperforms TEXT4VIS.With ViT-B/16, it also exceeds ActionCLIP by 14.6% (84.1% versus 69.5%) and TEXT4VIS by 4.2% (54.7% versus 50.5%).Source: §5.2.1 · p. 7
• 11.9% on UCF and 11.0% on HMDB: Open-VCLIP++ exceeds X-CLIP under Evaluation Protocol 3 with ViT-B/16.The reported accuracies are 83.9% versus 72.0% on UCF and 55.6% versus 44.6% on HMDB.Source: §5.2.1 · p. 8

5.2.2 Results with different backbones

• ViT-L/14 exceeds ViT-B/16 in CLIP* zero-shot accuracy by 6.3% on UCF, 7.4% on HMDB, and 8.1% on Kinetics-600.The corresponding pairs are 80.5% versus 74.2%, 55.0% versus 47.6%, and 76.2% versus 68.1%.Source: §5.2.2 · p. 8, Table 4 · p. 8
• Standard fine-tuning does not always improve zero-shot performance: it lowers ViT-L/14 results on HMDB and lowers all reported Kinetics-600 results.Open-VCLIP++ instead shows consistent improvements over baselines across datasets and models, avoiding the Kinetics-600 drop reported for standard fine-tuning.Source: §5.2.2 · p. 8

5.3.1 The effectiveness of different components

• VCLIP with temporal modeling outperforms CLIP across the reported zero-shot datasets and achieves a better close-set/zero-shot trade-off at interpolation ratios.The curves compare zero-shot dataset accuracy with Kinetics-400 accuracy; VCLIP expands CLIP spatial attention to spatial-temporal attention without additional parameters.Source: §5.3.1 · pp. 8–9
• Adding captions to Open-VCLIP produces the best reported trade-off among the component variants.Open-VCLIP++ further expands the Kinetics-400 text space with captions generated using BLIP-2 and Llama-2.Source: §5.3.1 · pp. 8–9

5.3.2 IWR v.s. SWA

• Removing either IWR or SWA causes significant drops, while either component alone improves zero-shot performance over using no interpolation.The reported comparison suggests IWR and SWA are complementary.Source: §5.3.2 · p. 9, Figure 4 · p. 9

5.3.3 Weight regularization during fine-tuning

• IWR constrains weight changes with an ℓ2 penalty rather than estimating parameter importance from historical data as EWC does.Historical data is unavailable in this setting.Source: §5.3.3 · p. 9
• Open-VCLIP achieves a better close-set/zero-shot trade-off than ℓ2-regularized fine-tuning across the reported datasets.Changing the interpolated weight coefficients yields different trade-offs after one training run; changing the ℓ2 regularization coefficient requires retraining the full model.Source: §5.3.3 · p. 9, Figure 5 · p. 9

5.3.4 Comparison with parameter-efficient fine-tuning

• ST-Adapter fails to match Open-VCLIP++ in zero-shot action recognition, with marked degradation on Kinetics-600.The comparison incorporates CLIP's text encoder into ST-Adapter; the original approach omits it and adds a linear classifier, preventing zero-shot testing.Source: §5.3.4 · pp. 9–10, Table 6 · p. 10

5.3.5 Text-to-Video/Video-to-Text Retrieval Performance

• 3.4%, 6.1%, and 5.3%: Open-VCLIP++ exceeds CLIP in MSR-VTT text-to-video Recall@1, Recall@5, and Recall@10, respectively.Video-to-text recalls exceed CLIP by 8.2%, 8.6%, and 9.4% at Recall@1, Recall@5, and Recall@10, respectively.Source: §5.3.5 · p. 10, Table 5 · p. 10
• Open-VCLIP++ is reported to achieve comparable or better retrieval results while training only on Kinetics-400, unlike several methods using additional large datasets.It exceeds OMNIVL in text-to-video Recall@5 (59.8% versus 58.4%) and Recall@10 (69.5% versus 66.6%).Source: §5.3.5 · pp. 10–11, Table 5 · p. 10

5.3.6 The importance of alignment with pseudo captions

• 3.3%, 5.5%, and 3.9%: captions improve text-to-video Recall@1, Recall@5, and Recall@10, respectively.The reported pairs are 34.6% versus 31.3%, 59.8% versus 54.3%, and 69.5% versus 65.6%; the retrieval gains are more pronounced than action-recognition gains.Source: §5.3.6 · p. 11

5.3.7 Examples of generated captions

• In Figure 6, captions generated by BLIP-2 and Llama-2 include more video details than the original action labels, including environmental descriptions for ice climbing and a more detailed hummingbird-feeding description.Source: §5.3.7 · p. 11
• Generated captions are not always accurate: one describes cartwheeling as involving brief moments of a handstand.Source: §5.3.7 · p. 11

5.3.8 Results of Open-VCLIP++ on ImageNet

• Open-VCLIP++ achieves a decent ImageNet trade-off, demonstrating better retention of CLIP’s original image-task knowledge.Figure 7 compares close-set K400 performance with zero-shot ImageNet performance; the model can be tested on images by turning off extended attention view.Source: §5.3.8 · pp. 11–12, Figure 7 · p. 12

6 CONCLUSION

• Open-VCLIP++ outperforms state-of-the-art methods on zero-shot video action recognition and achieves the best trade-off between close-set and zero-shot recognition.Source: §6 · p. 12
• Lightweight temporal modeling modules equip CLIP to capture spatial and temporal relationships in videos.Source: §6 · p. 12
• A regularization strategy strives to preserve CLIP’s zero-shot abilities and support generalization.Source: §6 · p. 12
• The video-captioning strategy expands the text space to reduce the risk of overfitting.Source: §6 · p. 12
• Adversaries could craft membership-inference attacks to steal information from the model.Source: §6 · p. 12