Source-linked AI summary
MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge
Wei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Possegger, Mateusz Kozinski, Rameswar Panda, Rogerio Feris, Hilde Kuehne, Horst Bischof
TL;DR
Vision-language models underperform on zero-shot action recognition without video finetuning, while supervised adaptation relies on costly full annotations and limited action vocabularies. MAXI finetunes CLIP on unlabeled videos using per-video text bags and Multiple Instance Learning. It improves transfer to unseen zero-shot and few-shot benchmarks, including up to 14% absolute zero-shot improvement over the source model.
Problem
Vision-language models underperform on zero-shot video action recognition without finetuning, while supervised adaptation relies on costly full annotations and exposes models to limited action vocabularies.
Source: §1 · p. 2Method
MAXI finetunes CLIP on unlabeled videos using action-dictionary matches, expanded text and captions in per-video text bags, with Multiple Instance Learning.
Source: §3 · p. 3, §3.2 · p. 4Results
Takeaways & Limitations
Unlabeled video collections and language sources can support transfer to novel zero-shot and few-shot action-recognition tasks without ground-truth finetuning labels.
Source: §3 · p. 3, §5 · p. 9Unlabeled video collections are typically much smaller than CLIP's original domain and may be prone to overfitting.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Large scale Vision-Language (VL) models have shown tremendous success in aligning representations between visual and text modalities. This enables remarkable progress in zero-shot recognition, image generation & editing, and many other exciting tasks. However, VL models tend to over-represent objects while paying much less attention to verbs, and require additional tuning on video data for best zero-shot action recognition performance. While previous work relied on large-scale, fully-annotated data, in this work we propose an unsupervised approach. We adapt a VL model for zero-shot and few-shot action recognition using a collection of unlabeled videos and an unpaired action dictionary. Based on that, we leverage Large Language Models and VL models to build a text bag for each unlabeled video via matching, text expansion and captioning. We use those bags in a Multiple Instance Learning setup to adapt an image-text backbone to video data. Although finetuned on unlabeled video data, our resulting models demonstrate high transferability to numerous unseen zero-shot downstream tasks, improving the base VL model performance by up to 14\%, and even comparing favorably to fully-supervised baselines in both zero-shot and few-shot video recognition transfer. The code will be released later at \url{https://github.com/wlin-at/MAXI}.
1. Introduction
Vision-language models enable broad zero-shot transfer, but without video finetuning they underperform on action recognition and tend to represent objects more than actions. MAXI adapts them with unlabeled videos and language-derived text bags, improving transfer to unseen zero-shot and few-shot action benchmarks.
1. Introduction
- Without finetuning on video, VL models underperform on zero-shot action recognition and tend to represent objects more than verbs or verb phrases.Prior approaches finetune on fully annotated action datasets, which are time-consuming and costly to collect.Source: §1 · p. 2
- Supervised finetuning exposes models to a limited action vocabulary, such as K400's 400 actions, compared with over 8K possible single-verb actions in English.The passage also notes that English permits many more general actions.Source: §1 · p. 2
- MAXI builds a text bag for each unlabeled video from language sources and uses Multiple Instance Learning to finetune a vision-language model.The language sources include an action dictionary, a VL model, and an LLM.Source: Figure 1 · p. 1, §1 · p. 2
- MAXI improves zero-shot and few-shot transfer on seven unseen benchmarks, with up to 14% absolute zero-shot improvement over the source VL model and wins over fully supervised baselines on some tasks.The evaluations include baselines trained in a fully supervised manner on the same data.Source: §1 · p. 2
2. Related Work
Prior work adapts vision-language models for video action recognition, but typically relies on ground-truth labels or manually corrected descriptions. MAXI instead uses unlabeled videos and expands action text with language and vision-language sources.
2. Related Work
- Earlier zero-shot action recognition methods used attributes, action-name embeddings, or action descriptions; ER-ZSAR [7] and JigsawNet used manually corrected descriptions assigned to videos using ground-truth labels.The description correction requires human annotators.Source: §2 · pp. 2–3
- Video-adaptation approaches add spatio-temporal components or use frame-level processing; MAXI follows ViFi-CLIP's [38] architecture and finetuning paradigm.The cited work reports that frame-level processing with feature pooling outperforms more elaborate learnable spatio-temporal components.Source: §2 · p. 3
- Unlike related approaches that use ground-truth annotations and simple action names, MAXI finetunes without labels and expands action text with LLM descriptions and visual captions.The MAXI pipeline builds per-video text bags and trains with Multiple Instance Learning.Source: §2 · p. 3, Figure 2 · p. 3
3. Method
MAXI adapts CLIP to video using frame-level features, per-video text bags built from language sources, and Multiple Instance Learning. Its matching stage also distills the original CLIP model's knowledge to support generalization.
3. Method
- MAXI uses unlabeled videos and a potentially noisy action dictionary to finetune CLIP without ground-truth annotations for zero-shot transfer to novel video categories.Each video is paired with a noisy collection of prompts that may describe its contents.Source: §3 · pp. 3–4
3.1. CLIP on Video Data
- The video encoder processes sampled frames with CLIP and averages their features, enabling initialization from pretrained CLIP without randomly initialized parameters.Zero-shot classification selects the class prompt with the highest similarity to the video representation.Source: §3.1 · p. 4
3.2. Text Bag Construction
- For each video, MAXI matches dictionary prompts with CLIP and adds the top match to a text bag, then expands it with GPT-3 descriptions and BLIP frame captions.The expanded descriptions contribute verbs and verb phrases, including lemma and gerund forms; bag filtering keeps videos whose best-match score exceeds a threshold.Source: §3.2 · p. 4
- CLIP matching distills the original model's knowledge as a teacher, which the authors argue helps prevent overfitting to smaller unlabeled video collections and preserve pretrained generalizability.The passage ties this rationale to improved zero-shot transfer and few-shot adaptation on novel datasets.Source: §3.2 · p. 4
3.3. Multiple Instance Learning
- MIL-NCE softly associates each video with a bag of possible positive texts, randomly sampling a fixed number of texts from each bag during training.The proposed loss additionally encourages videos with the same best-matched dictionary text to be close.Source: §3.3 · p. 5
4. Experiments
The experiments evaluate MAXI on zero-shot and few-shot action recognition, using unlabeled K400 videos and language-derived text bags. Results cover seven transfer datasets and ablations of filtering, dictionary noise, bag content, learning strategy, and bag size.
4.1. Datasets
- The evaluation uses K400 for unlabeled finetuning and tests transfer on seven datasets, with both smaller and larger domain shifts from K400.The benchmarks include UCF101, HMDB51, MiniSSv2, K600, Charades, UAV, and MiT.Source: §4.1 · p. 5
4.2. Implementation Details
- The main setup full-finetunes CLIP’s ViT-B/16 encoders on K400 and uses 16 frames per video, a 90% text-bag filtering ratio, and bag size 16.Few-shot training uses 32 frames per video.Source: §4.2 · p. 5
4.3. Zero-Shot Action Recognition
- MAXI with added GPT-3 and BLIP verbs reaches state of the art across UCF, HMDB, and K600, beats ViFi-CLIP on MiT and MiniSSv2, and improves over CLIP on all four larger-shift datasets.The four larger-shift datasets are Charades, MiT, MiniSSv2, and UAV.Source: §4.3 · p. 6, Table 1 · p. 6, Table 2 · p. 6
4.4. Few-Shot Action Recognition
- Few-shot learning from a MAXI-pretrained backbone performs best in most settings, including some cases where it surpasses ViFi-CLIP’s fully supervised initialization.The advantage is especially marked in the 2-shot HMDB and UCF settings; the authors also report that MAXI mitigates degradation seen with supervised initialization in some settings.Source: §4.4 · p. 6, Table 3 · p. 7
4.5.1 Text bag filtering
- Filtering improves zero-shot transfer at p = 90% versus p = 100%, while more aggressive filtering eventually reduces performance as training data become limited.Matching accuracy rises as p decreases, but the authors report a trade-off between filtered-data quality and quantity.Source: §4.5.1 · p. 7, Table 4 · p. 7
4.5.2 Robustness against noisy action dictionary
- Finetuning with under- or over-specified noisy action dictionaries still boosts zero-shot transfer over CLIP, although the original K400 dictionary performs best on most downstream datasets.The noisy-dictionary evaluation uses p = 50%.Source: §4.5.2 · p. 7, Table 5 · p. 8
4.5.3 What words to include in the text bag?
- Adding language sources to the K400 action dictionary improves zero-shot transfer, with BLIP verbs slightly outperforming BLIP object nouns and combined BLIP and GPT-3 verbs performing best.The word-combination ablation uses p = 100% to retain the full noisy text bags.Source: §4.5.3 · p. 8, Table 6 · p. 8
4.5.4 How to learn from words in text bags?
- MIL-NCE achieves the best improvement among the tested text-bag learning strategies, and matching bags sharing the same best-matched text adds a boost over instance-level matching alone.The comparisons include fixed-class cross-entropy, NCE with a randomly sampled text, MIL-Max, and MIL-NCE variants.Source: §4.5.4 · p. 8, Table 7 · p. 8
5. Conclusion
MAXI adapts vision-language models for zero-shot action recognition using unlabeled videos and language sources. It builds a text bag for each video and uses Multiple Instance Learning, with evaluations reporting gains over the source model and fully supervised baselines.
5. Conclusion
- MAXI uses unpaired action dictionaries, Large Language Models, and vision-language models to construct a text bag for each unlabeled video.The paper describes MAXI as an approach for finetuning a vision-language model for improved zero-shot action recognition.Source: §5 · p. 8
- Multiple Instance Learning uses the unlabeled videos and their text bags to finetune the vision-language model.Source: §5 · p. 9
- Evaluation across unseen action benchmarks reports significant improvements over the source vision-language model and fully supervised baselines.Source: §5 · p. 9
Supplementary
The supplementary material adds dataset statistics and implementation details. It also reports further visualizations and experiments on noisy dictionaries, language sources, and a cross-frame attention module.
Supplementary
- The supplementary material provides additional dataset statistics and implementation details for MAXI.Source: Supplementary · p. 9
- Additional visualizations compare attention heatmaps across several approaches.Source: Supplementary · p. 9
- Further supplementary results cover noisy action dictionaries, training language sources, and a cross-frame attention temporal module.Source: Supplementary · p. 9
A. Dataset Statistics
The appendix describes the datasets used to assess transfer across action-recognition settings. They span varied video sources, class structures, domains, and evaluation protocols.
A. Dataset Statistics
- K400 contains about 240K YouTube training videos across 400 action classes and supplies the finetuning training set.Source: A. Dataset Statistics · p. 9
- UCF101 and HMDB51 are YouTube-based benchmarks, while K600 evaluation uses classes excluded from K400.UCF101 has 13K videos across 101 classes; HMDB51 has 7K videos across 51 classes. K600 evaluation uses 220 classes not included in K400.Source: A. Dataset Statistics · p. 9
- MiniSSv2 represents fine-grained, motion-based actions with a language style and visual domain that differ from K400.The described examples include detailed actions involving one object failing to fit into another and lifting a surface without sliding its contents.Source: A. Dataset Statistics · p. 9
- Charades tests multi-label recognition of long home-activity videos, averaging 6.8 action instances per video.Source: A. Dataset Statistics · p. 9
- UAV Human uses 155 action classes recorded from distinctive aerial viewpoints, with fixed subjects and backgrounds producing low object-scene bias and a large domain shift from K400 and CLIP.Moments-in-Time contains three-second YouTube clips across 305 action classes performed by humans and animals.Source: A. Dataset Statistics · p. 9
B. Implementation Details
The implementation combines CLIP-based action matching, GPT-3 text expansion, and frame captioning to build video text bags. It also specifies prompt construction, training configuration, and supplementary attention visualizations.
B. Implementation Details
- CLIP2 with a ViT-B/16 visual encoder matches videos to the action dictionary using an ensemble of 28 Kinetics prompt templates.The inference protocol uses a single prompt template.Source: B. Implementation Details · p. 9
- GPT-3 text-davinci-003 generates five simple verb-phrase descriptions from each video's best-matched dictionary action.Source: B. Implementation Details · p. 9
- The supplementary heatmaps compare CLIP, ViFi-CLIP, and MAXI, with warm and cold colors denoting higher and lower attention; MAXI focuses more on action-relevant body regions.Examples include hands for clapping, legs for kicking, and the mouth for chewing.Source: Figure 3 · p. 10, Figure 4 · p. 10, Figure 5 · p. 11
- BLIP with ViT-L/16 captions eight uniformly sampled video frames, and extracted verbs and verb phrases are augmented with lemma and gerund forms.Source: B. Implementation Details · p. 10
- Both visual and text encoders are fully finetuned with the ViFi-CLIP configuration, using multiscale crops, random flips, color jitter, and grayscale augmentation.MixUp and CutMix are not used.Source: B. Implementation Details · p. 10
- Each training iteration samples Nbag texts and forms Nbag prompts, pairing a shared “a photo of” action template with individual bag texts except the matched action.The prompts use the best-matched dictionary action in the shared template.Source: B. Implementation Details · pp. 10–11
C.1. Attention Heatmaps
Across in-dictionary, novel, and general actions, MAXI produces more focused attention on the body parts performing the action than CLIP and ViFi-CLIP. The heatmaps illustrate how language knowledge from video captions and GPT-3 phrases relates to these attention patterns.
In-dictionary action classes
- For in-dictionary actions, MAXI focuses attention on hands for clapping and legs for kicking, unlike CLIP’s object or background focus and ViFi-CLIP’s cluttered attention.GPT-3 clapping bags include hand-related phrases, while BLIP captions provide repeated verb occurrences.Source: §C.1 · p. 13
Novel action classes
- For novel actions absent from the K400 dictionary, MAXI focuses on the hand and arm for waving and the mouth for chewing, unlike CLIP and ViFi-CLIP.BLIP captions contain wave and chew in videos of related actions, supporting knowledge of actions missing from the dictionary.Source: §C.1 · p. 13
General actions
- For general verbs, MAXI concentrates attention on the action area in examples including catching, hitting, jumping, and running.Caption and GPT-3 verb bags contain diverse instances of general verbs, including unusual scenes such as jumping on stairs.Source: §C.1 · p. 13
C.2. Robustness Against Noisy Action Dictionary
MAXI retains robust zero-shot transfer when the action dictionary is expanded with noisy verbs, although performance changes are not consistent across downstream datasets.
C.2. Robustness Against Noisy Action Dictionary
- The expanded dictionaries add 800 or 1200 noisy verbs parsed from WebVid2.5M captions to the original K400 action dictionary.These additions produce 1200-class and 1600-class spaces.Source: §C.2 · p. 13
- With text bag filtering ratio p = 50%, finetuning using the resulting 1200-class and 1600-class spaces still yields robust zero-shot transfer to unseen datasets, even when 50% to 75% of dictionary words mismatch video data.Source: §C.2 · p. 13
- Transfer performance does not change consistently across datasets, which the authors relate to differences in language-domain shift from the training dictionary.They attribute robustness to collecting knowledge from multiple language sources and learning through Multiple Instance Learning.Source: §C.2 · p. 13
C.3. Examples of Language Sources
The appendix shows examples of language sources used to form verb bags and reports that adding cross-frame temporal attention did not improve zero-shot transfer.
C.3. Examples of Language Sources
- Figure 6 pairs unlabeled K400 video frames with BLIP captions, GPT-3 phrases, and derived BLIP and GPT-3 verb bags.The examples illustrate the language material associated with each video.Source: §C.3 · pp. 13–14
C.4. Parameter-Free Temporal Module
- The parameter-free temporal module computes attention for frame t using keys from frames t−1, t, and t+1.The cross-frame attention is applied to either the last two or last six visual-encoder transformer layers.Source: §C.4 · p. 14
- Adding cross-frame temporal attention does not improve zero-shot transfer compared with the variant without temporal attention.The authors suggest that finetuning from CLIP weights with a modified architecture may cause drift from CLIP’s source domain.Source: §C.4 · p. 14