Source-linked AI summary
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid
TL;DR
Manual VideoQA annotation is costly and difficult to scale, motivating zero-shot approaches. FrozenBiLM connects visual inputs to a frozen bidirectional language model with lightweight trainable modules trained on Web-scraped multimodal data, then answers through masked language modeling. It improves zero-shot VideoQA across varied benchmarks and remains competitive in few-shot and fully supervised settings.
Problem
Manual question-answer annotation for videos is tedious and prevents scalable VideoQA training.
Method
FrozenBiLM combines a frozen bidirectional language model and visual backbone with lightweight trainable modules trained on Web-scraped multimodal data for masked-language-modeling-based VideoQA.
Results
FrozenBiLM largely improves state-of-the-art performance across zero-shot VideoQA benchmarks and is competitive in few-shot and fully supervised settings.
Takeaways & Limitations
Frozen bidirectional language models provide a stronger and cheaper alternative to frozen autoregressive language models for zero-shot VideoQA.
Takeaways & Limitations
The model is not directly applicable to complex multimodal text-generation tasks such as video captioning.
Abstract
from arXiv · showhide
Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answer. In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs. In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-modal data, and finally (iii) we perform zero-shot VideoQA inference through masked language modeling, where the masked text is the answer to a given question. Our proposed approach, FrozenBiLM, outperforms the state of the art in zero-shot VideoQA by a significant margin on a variety of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA, TGIF-FrameQA, How2QA and TVQA. It also demonstrates competitive performance in the few-shot and fully-supervised setting. Our code and models are publicly available at https://github.com/antoyang/FrozenBiLM.
1 Introduction
FrozenBiLM adapts frozen bidirectional language models to zero-shot VideoQA using lightweight trainable modules and Web-scraped multimodal data. The framework reports stronger zero-shot performance than prior autoregressive approaches while also supporting few-shot and fully supervised settings.
- Manual VideoQA question-answer annotation is costly, time consuming, and not scalable, motivating zero-shot approaches without visual question-answer annotations.
- The trainable modules are learned from Web-scraped multimodal data while the bidirectional language-model weights remain frozen.
- FrozenBiLM combines frozen bidirectional language models with visual inputs through lightweight trainable modules.
- FrozenBiLM enables zero-shot VideoQA through masked language modeling and compares favorably with previous autoregressive models.
- FrozenBiLM improves the state of the art in zero-shot VideoQA by a significant margin across varied benchmarks.
- The framework also demonstrates competitive fully supervised performance and strong results in the few-shot VideoQA setting.
2 Related Work
Prior zero-shot VideoQA methods use automatically mined video-text data instead of manual VideoQA annotations, while related visual-language work transfers pretrained language models to multimodal tasks. FrozenBiLM differs by using a frozen bidirectional model and masked language modeling without explicitly annotated language datasets or data-generation pipelines.
- Zero-shot VideoQA methods avoid visual question-answer annotations by training on automatically mined video clips with short text descriptions available at Web scale.
- Prior approaches generate VideoQA data or reformulate questions using language models and multimodal models.
- FrozenBiLM does not require explicitly annotated language datasets or data-generation pipelines for zero-shot VideoQA.
- Visual-language research has transferred pretrained autoregressive language models to image-text tasks, including approaches that freeze the language-model weights.
- Masked language modeling pretrains bidirectional transformers and has been adapted to multimodal transformer models using paired visual-linguistic data.
3 Method
FrozenBiLM combines frozen bidirectional language and visual encoders with lightweight trainable modules, then adapts them using web-scraped video-text data and visually conditioned masked language modeling. At inference, video question-answering tasks are formulated as masked-token prediction using prompts with visual and textual context.
- Architecture: FrozenBiLM unifies a frozen pretrained BiLM and visual encoder through a visual-to-text projection and lightweight adapters.The projection maps visual features into the joint visual-text embedding space, while adapters are inserted between frozen transformer blocks; normalization layers are also finetuned.
- Frozen bidirectional language model: The BiLM tokenizes text, contextualizes all tokens with a Transformer encoder, and predicts vocabulary distributions through an MLM classifier head.Its weights are pretrained with masked language modeling on large web text and remain frozen during multimodal adaptation.
- Visual conditioning: A frozen CLIP ViT-L/14 visual backbone independently encodes video frames into feature vectors that are linearly projected into text-token embedding space.The resulting video prompt is concatenated with text embeddings before the transformer encoder models joint visual-linguistic interactions.
- Cross-modal training: Training uses only web-scraped video-caption pairs, keeping the BiLM and visual backbone frozen while learning the projection, adapters, and normalization layers.The trainable parameters total about 5% of the BiLM parameters, and the objective randomly masks text tokens for prediction from surrounding text and video.
- Adapting to downstream tasks: At inference, open-ended, multiple-choice, and fill-in-the-blank tasks are converted into masked-language-modeling prompts containing video prompts and optional subtitles.Open-ended and fill-in-the-blank tasks predict answers from roughly 1K-word vocabularies, while multiple-choice tasks select the candidate with the highest Yes logit.
4 Experiments
Experiments evaluate FrozenBiLM through ablations, modality and scale analyses, comparisons with autoregressive models and zero-shot benchmarks, plus few-shot and supervised settings.
- Experimental setup: FrozenBiLM uses a frozen DeBERTa-V2-XLarge language model with a visual-to-text projection layer and adapters, trained for two epochs on WebVid10M.The model uses video, question and speech as input modalities.
- Ablation studies: Freezing pretrained language-model weights performs better than updating them during cross-modal training and makes training twice faster.Randomly initialized language weights perform poorly, while pretrained initialization is crucial.
- Ablation studies: +13.6% accuracy on iVQA and +22.1% on MSVD-QA result from adding vision over speech-only inputs.Speech additionally improves vision-equipped performance by +16.5% on How2QA and +29.5% on TVQA.
- Ablation studies: Performance increases monotonically as the fraction of WebVid10M used for multimodal training grows.The evaluated subsets are nested, with larger subsets including smaller ones.
- Ablation studies: The 900M-parameter DeBERTa-V2-XLarge outperforms the 300M-parameter BERT-Large, which outperforms the 100M-parameter BERT-Base.The comparison controls adapter use by omitting adapters for different language-model sizes.
- Zero-shot VideoQA: FrozenBiLM outperforms previous methods by a significant margin on all 8 zero-shot VideoQA datasets.It remains competitive without speech input, while qualitative comparisons show text-only and unfrozen variants make visually unrelated or commonsense-inconsistent predictions.
- Few-shot VideoQA: Using only 1% of VideoQA training data yields significant improvements over zero-shot performance, and freezing the BiLM benefits few-shot performance.The low number of trainable parameters makes the approach suited to low-data training.
5 Conclusion
FrozenBiLM extends frozen bidirectional language models to multimodal inputs using Web-scraped data and masked language modeling, improving zero-shot VideoQA across datasets. The framework is also competitive in supervised and few-shot settings, while leaving scaling and complex generation as open directions.
- FrozenBiLM extends frozen bidirectional language models to multimodal inputs by training additional modules on Web-scraped data and answering through masked language modeling.
- FrozenBiLM improves state-of-the-art zero-shot VideoQA across various datasets and performs competitively in fully supervised and newly introduced few-shot settings.
- The framework is presented as more efficient than its autoregressive variant, supported by extensive ablation studies.
- Open directions include scaling bidirectional language models to several billion parameters and training on larger YouTube datasets with speech transcripts or audio.
- FrozenBiLM cannot be applied out-of-the-box to complex multimodal text-generation tasks such as video captioning.
- The models may reflect biases from training videos, captions, text data, and visual-backbone images and captions.
A Qualitative examples for zero-shot VideoQA
Qualitative examples compare FrozenBiLM with Just Ask, an unfrozen variant, and a text-only variant. The zero-shot model can fail on complex temporal or spatial reasoning and can produce semantically inaccurate answers.
- Qualitative examples compare FrozenBiLM with Just Ask, UnFrozenBiLM, and a text-only variant across zero-shot VideoQA predictions.
- The zero-shot model underperforms supervised VideoQA models and fails on examples requiring complex temporal or spatial understanding.
- Observed errors include missing an object behind a person and confusion when a person browses through many phone tabs.
- The model also makes semantic errors, such as confusing a restaurant with a bakery or a chicken with another bird.
B Qualitative analysis of the frozen self-attention patterns in FrozenBiLM
FrozenBiLM’s frozen self-attention layers still enable interactions between visual and textual tokens during zero-shot VideoQA.
- Frozen self-attention layers enable visual-linguistic interactions despite remaining frozen after text-only pretraining.
- The analysis visualizes attention between video-prompt visual tokens and text-embedder textual tokens for an example from Figure 4.
C Experimental setup
The experimental setup covers multiple open-ended, fill-in-the-blank, and multiple-choice VideoQA datasets, with qualitative comparisons and specified architecture and training settings. A zero-shot image-VQA evaluation also reports strong performance for FrozenBiLM.
- Qualitative examples: The qualitative appendix presents successful and incorrect open-ended predictions, successful fill-in-the-blank predictions, and successful multiple-choice predictions.
- Qualitative examples: The comparisons include Just Ask, UnFrozenBiLM, FrozenBiLM without visual input, and FrozenBiLM.
- Datasets: The evaluation includes open-ended, fill-in-the-blank, and multiple-choice VideoQA datasets spanning instructional videos, movies, GIFs, long videos, and television shows.
- Architecture hyperparameters: The model samples T = 10 frames at temporally equal distances, uses 224 × 224 pixels per frame, and extracts 768-dimensional visual features.
- Training: Cross-modal training on WebVid10M uses batches of 128 video-caption pairs across 8 NVIDIA Tesla V100 GPUs, with variant-specific learning rates.
- Zero-shot image-VQA: 45.0% accuracy was achieved on VQAv2 validation, compared with 29.5% for Frozen, while FrozenBiLM was 7 times smaller.
D.3 Detailed zero-shot VideoQA results segmented per question category
The authors analyze zero-shot VideoQA performance across question categories and assess robustness across random seeds and supervision levels.
- FrozenBiLM shows large and consistent improvements over Just Ask across question categories, except number questions on MSRVTT-QA and MSVD-QA.
- Five random-seed runs show that seed variation does not change FrozenBiLM’s improvement over prior zero-shot VideoQA work.The comparison reports mean and standard deviation over five cross-modal training runs.
- FrozenBiLM outperforms its unfrozen language-model variant in few-shot settings when adapters are trained.The difference is larger when using 1% of downstream training data.
D.6 Ablation of the multi-token inference strategy
The authors compare FrozenBiLM’s simple multi-token answer scoring with parallel masked-span decoding and discuss why the alternative offers little benefit.
- FrozenBiLM averages the weights of answer tokens for multi-token answers, but this strategy does not preserve answer semantic structure.
- Parallel decoding with multiple mask tokens does not significantly improve zero-shot open-ended VideoQA accuracy over FrozenBiLM’s existing inference strategy.The comparison is reported in Table 13.
- The limited decoding gain may reflect that these datasets contain mostly short answers, including 99% one-token answers in the MSRVTT-QA test set.
D.7 Additional ablation studies in the zero-shot setting
Additional zero-shot ablations examine frame count, adapter size, visual backbone, and prompt design, identifying useful design choices and prompt dependencies.
- Visual and adapter ablations: Using 10 frames significantly improves zero-shot VideoQA performance over using a single frame.
- Prompt ablation: Replacing “Question”, “Answer” and “Subtitles” with “Q”, “A” and “S” does not affect zero-shot VideoQA accuracy.
- Prompt ablation: Removing prompt tokens that link the textual inputs causes a significant drop in zero-shot VideoQA performance.The affected tokens include “Question”, “Answer”, “Subtitles” and “is it”.
D.8 Cross-modal training and adapters are crucial for fully-supervised performance
The fully supervised ablations evaluate cross-modal training and adapter training, showing that both contribute substantially to FrozenBiLM’s downstream performance.
- The fully supervised ablations compare variants without adapters and without cross-modal training against the full model.
- Training adapters has considerable importance for fully supervised VideoQA performance.This comparison is reported in Table 17.