Source-linked AI summary
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji
TL;DR
Few-shot video-language systems often lack a generative decoder, while finetuned video captioners do not flexibly handle unseen tasks. VidIL converts videos into hierarchical textual representations and prompts a language model with a few examples, achieving strong results across video-to-text tasks without video-dataset pretraining or finetuning.
Problem
Few-shot video-language learners are often encoder-only, while video captioners depend on video-data finetuning and cannot readily generate text for unseen tasks.
Method
VidIL uses image-language models to produce frame captions and visual phrases, organizes them temporally with optional ASR, and prompts a pretrained language model with in-context examples.
Results
VidIL outperforms video-language and image-language baselines on few-shot captioning and question answering and significantly outperforms supervised models on future event prediction without video-dataset pretraining or finetuning.
Takeaways & Limitations
A unified textual representation lets frozen language models support diverse few-shot video-to-text tasks and incorporate additional modalities such as ASR transcripts.
Takeaways & Limitations
Language-model outputs may have negative societal impacts, including bias against people of a certain gender, race, or sexuality.
Abstract
from arXiv · showhide
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction. Existing few-shot video-language learners focus exclusively on the encoder, resulting in the absence of a video-to-text decoder to handle generative tasks. Video captioners have been pretrained on large-scale video-language datasets, but they rely heavily on finetuning and lack the ability to generate text for unseen tasks in a few-shot setting. We propose VidIL, a few-shot Video-language Learner via Image and Language models, which demonstrates strong performance on few-shot video-to-text tasks without the necessity of pretraining or finetuning on any video datasets. We use the image-language models to translate the video content into frame captions, object, attribute, and event phrases, and compose them into a temporal structure template. We then instruct a language model, with a prompt containing a few in-context examples, to generate a target output from the composed content. The flexibility of prompting allows the model to capture any form of text input, such as automatic speech recognition (ASR) transcripts. Our experiments demonstrate the power of language models in understanding videos on a wide variety of video-language tasks, including video captioning, video question answering, video caption retrieval, and video future event prediction. Especially, on video future event prediction, our few-shot model significantly outperforms state-of-the-art supervised models trained on large-scale video datasets. Code and resources are publicly available for research purposes at https://github.com/MikeWangWZHL/VidIL .
1 Introduction
VidIL targets the gap between encoder-only few-shot video-language models and generative video-to-text tasks by combining image-language representations with frozen language-model prompting. It reports strong few-shot results without video-dataset pretraining or finetuning across captioning, question answering, and future event prediction.
- Most recent few-shot video-language pretraining frameworks are encoder-only and cannot generate text for captioning, question answering, or future prediction.
- VidIL uses image models to represent video content and instructs frozen large-scale language models to generate summaries, answers, or other task outputs.
- Videos require modeling both static visual features and temporal changes in objects, actions, and events across multiple granularities.
- Without video-dataset pretraining or finetuning, VidIL outperforms video-language and image-language state-of-the-art baselines on few-shot captioning and question answering.
- With only 10 labeled examples, VidIL significantly outperforms fully supervised models on video-language event prediction and also supports retrieval through pseudo-label generation.
2 Related Work
Prior work transfers image-language knowledge to videos and unifies modalities through text, but generation-oriented video tasks remain dependent on finetuning or annotated video-text data. VidIL extends textual image-language representations to few-shot, temporal video-to-text generation with a frozen language model.
- Image-language models have enabled zero-shot retrieval-style video tasks, while generation-oriented captioning still relies heavily on finetuning.
- Video-language pretraining is hindered by noisy and domain-specific datasets, motivating transfer of image-model knowledge to sparsely sampled video frames.
- Prior unified textual approaches connect modalities, but some require annotated video-text data to jointly train language models with video and audio tokenizers.
- VidIL introduces a temporal-aware hierarchical textual representation and uses prompting of a frozen language model for few-shot video-language tasks.
- Unlike zero-shot long-video world-state representation, VidIL adapts to different video and text distributions using few examples.
3 Method
VidIL decomposes videos into visual tokens, frame captions, and video-level text, then combines these representations with temporal-aware few-shot prompting. A pretrained language model generates task-specific outputs from the resulting prompt.
- 3 Method: VidIL separates spatial and temporal information through visual-token, frame, and video levels, all expressed as unified text.
- 3.1 Frame Level: Image Captioning: At the frame level, sparse samples are captioned and filtered by a pretrained image-language model such as BLIP.
- 3.3 Video Level: Temporal-Aware Few-shot Prompting: The video-level prompt aggregates visual tokens, frame captions, optional ASR, in-context examples, and a task-specific query for language-model generation.
- 3.2 Visual Token Level: Structure-Aware Visual Tokenization: At the visual-token level, retrieval-based tokenization extracts salient objects, events, and attributes from frames.
- 3.2 Visual Token Level: Structure-Aware Visual Tokenization: Events are constructed from Visual Genome synsets by selecting phrases containing at least one verb and one argument, then removing highly similar events.
- 3.3 Video Level: Temporal-Aware Few-shot Prompting: Temporal markers such as “First,” “Then,” and “Finally” encode ordering so the model can distinguish scenarios such as sunset and sunrise.
4 Experiments
Experiments evaluate VidIL across four few-shot video-language tasks and five benchmarks, comparing it with video- and image-language baselines. Results show strong cross-domain captioning, large gains in question answering and event prediction, benefits from pseudo-label retrieval training, and sensitivity to temporal representation and example selection.
- Experimental Setup: VidIL is evaluated on video captioning, video question answering, event prediction, and text-video retrieval across five benchmarks.The benchmarks are MSR-VTT, MSVD, VaTeX, YouCook2, and VLEP.
- Few-shot Video Captioning: Existing captioning baselines show dataset bias, whereas VidIL performs competitively on both open-domain and instructional videos and significantly improves average CIDEr across three benchmarks.UniVL favors instructional YouCook2 videos, while BLIP favors description-style MSR-VTT and VaTeX videos.
- Few-shot Video Captioning: Adding ASR information through the unified textual representation yields significantly better few-shot performance on some datasets, including YouCook2.Unlike many existing methods, the framework can incorporate a new modality without dedicated model redesign or retraining.
- Few-shot Video Question Answering: VidIL’s 5-shot model outperforms 8-shot Flamingo-3B and matches 4-shot Flamingo-80B without video pretraining or observed video data.The comparison uses video question answering benchmarks and reports average accuracy over randomly selected 5-shot examples.
- Few-shot Video-Language Event Prediction: With 10 labeled examples, VidIL outperforms the fully supervised MERLOT baseline on VLEP accuracy by approximately 4%.The task generates free-form future-event text that is mapped to one of two candidate answers for evaluation.
- Semi-supervised Text-Video Retrieval: Pseudo-label training substantially improves text-video retrieval, with MSR-VTT Recall @ 5 becoming comparable to BLIP fine-tuned on full ground-truth annotations.Pseudo-labels from other baselines are less effective and can even reduce performance.
- Ablation Studies: Using one frame instead of four causes a significant performance drop, while reversing visual-token and caption order decreases performance only marginally.The results indicate that multiple timestamps matter, whereas current benchmarks may insufficiently measure benefits from improved temporal ordering.
- Ablation Studies: In-context example selection improves performance and robustness at equal context length, but gains from adding more shots saturate around 20 to 30 examples.At 30 shots, selecting five examples outperforms directly placing all 30 examples in the prompt.
5 Conclusions, Limitations and Future Work
VidIL uses unified textual representations and temporal-order-aware prompting to support few-shot video-to-text tasks without video pretraining. The paper reports broad task performance while identifying limitations in fine-grained spatial reasoning and temporal tracking.
- Conclusions: VidIL represents frame features as unified text and uses a temporal-order-aware prompt with hierarchical video structure and optional speech transcripts.The representation includes multiple levels of frame features and can incorporate speech transcripts.
- Conclusions: Without video pretraining, VidIL outperforms vision-language models trained on large-scale video datasets across captioning, question answering, and future event prediction.The conclusion also names domain-specific captioning among the few-shot tasks.
- Limitations: Unified textual representation may lose low-level visual features needed for fine-grained spatial visual question answering.This is identified as a limitation of the representation choice.
- Future Work: Current video-language benchmarks rarely require explicit temporal tracking of frames and visual tokens.Future work targets script knowledge from long videos where temporal dynamics are more emphasized.
6 Broader Impact
Few-shot video-language learners may support applications such as content detection and accessibility. Their use of internet-scale language-model pretraining also raises concerns about biased outputs and potential societal harm.
- Benefits: Open-domain few-shot video-language learners could help detect violent or mature content and support people with vision impairment.These are presented as beneficial applications for society.
- Risks: Internet-scale text pretraining may produce unexpected outputs with potential bias against people based on gender, race, or sexuality.The paper calls for future work and community collaboration to mitigate these potential negative impacts.
- Mitigation: The paper calls for dedicated collaboration to alleviate the potential negative societal impact of large language models.This recommendation follows the stated concern about unexpected biased outputs.
A Additional Qualitative Examples
Additional captioning examples show that the framework captures important video semantics that baseline captions miss. The examples cover MSR-VTT, YouCook2, and VaTeX.
- Datasets: Qualitative captioning examples are provided for MSR-VTT, YouCook2, and VaTeX.The examples appear in Figures 6 and 7.
- Findings: The framework captures important video semantics that baseline-generated captions omit.The passage identifies objects, events, and attributes as examples of these semantics.
- Findings: The additional examples highlight objects, events, and attributes in bold green text.These semantic categories are the specific content emphasized in the qualitative comparisons.
B Few-shot Prompt Examples
The appendix presents full few-shot prompts for captioning, question answering, and event prediction, along with omitted in-context examples for a main-text figure.
- Video Captioning: Full few-shot prompts for video captioning are shown in Figures 8 and 9.These figures document the prompting examples used for captioning.
- Video Question Answering: The full prompt for video question answering is shown in Figure 10.The appendix provides the corresponding in-context prompt examples.
- Event Prediction: The full prompt for video-language event prediction is shown in Figure 11.Figure 11 contains the prompt examples for this task.
- Supplementary Examples: Figure 12 shows in-context examples omitted from Figure 3 in the main body.The appendix supplements the main-text presentation with those omitted examples.
C Additional Experimental Details
The supplementary material details datasets, visual-token vocabulary construction, prompt composition, and few-shot examples across captioning, question answering, and event prediction. It also documents implementation settings for baseline finetuning and semi-supervised retrieval.
- Datasets: MSR-VTT, MSVD, and VaTeX datasets use task-specific training and testing splits, with VaTeX results based on its latest v1.1 release.The VaTeX release includes a public testing set, unlike the previous version.
- Visual-token vocabulary: The visual-token vocabulary combines OpenImage classes with Visual Genome object, attribute, and event synsets, while event phrases require at least one verb and one argument.Highly similar events are removed using SentenceBERT embeddings.
- Visual-token aggregation: Video-level visual tokens are ranked using frame scores and appearance frequency, filtered by rank, and assigned temporal indicators from their frame indices.A dynamic template adapts the temporal markers to the number of retained tokens.
- Implementation details: Baseline captioning uses 50 epochs with warmup and low learning rates, while semi-supervised retrieval trains BLIP for one epoch with sampled video frames.The captioning baselines are UniVL, BLIP, and BLIPcap; retrieval uses four training frames and eight test frames.
- Few-shot prompting: Few-shot prompts combine an instruction, selected in-context examples, and a task query represented with aggregated visual tokens, frame captions, and optional dialogue.Examples cover MSR-VTT, YouCook2, VaTeX, MSVD, and VLEP tasks, including ASR input.