Source-linked AI summary
Self-Chained Image-Language Model for Video Localization and Question Answering
Shoubin Yu, Jaemin Cho, Prateek Yadav, Mohit Bansal
TL;DR
Uniform frame sampling can miss query-relevant video cues, while query-aware temporal localization requires costly annotations. SeViLA uses one parameter-efficiently fine-tuned BLIP-2 to localize language-aware keyframes, answer questions from them, and refine localization through Answerer-generated pseudo-labels; it outperforms strong baselines across five benchmarks and reaches state of the art in specified fine-tuning and zero-shot settings.
Problem
Image-language models used for video QA typically concatenate uniformly sampled frames without explicit language-aware temporal modeling, while query-aware localization annotations are expensive.
Method
SeViLA parameter-efficiently fine-tunes BLIP-2 into a Localizer and Answerer linked by forward keyframe selection and reverse pseudo-label self-refinement.
Results
SeViLA outperforms strong baselines across five video QA and event prediction benchmarks and achieves state of the art in reported fine-tuning and zero-shot settings.
Takeaways & Limitations
Language-aware keyframe selection improves video-language task performance over uniform sampling in both zero-shot and fine-tuning settings.
Takeaways & Limitations
The frame-level Localizer may handle complex, fine-grained temporal events poorly, motivating structured prediction beyond frame-level localization.
Abstract
from arXiv · showhide
Recent studies have shown promising results on utilizing large pre-trained image-language models for video question answering. While these image-language models can efficiently bootstrap the representation learning of video-language models, they typically concatenate uniformly sampled video frames as visual inputs without explicit language-aware, temporal modeling. When only a portion of a video input is relevant to the language query, such uniform frame sampling can often lead to missing important visual cues. Although humans often find a video moment to focus on and rewind the moment to answer questions, training a query-aware video moment localizer often requires expensive annotations and high computational costs. To address this issue, we propose Self-Chained Video Localization-Answering (SeViLA), a novel framework that leverages a single image-language model (BLIP-2) to tackle both temporal keyframe localization and QA on videos. SeViLA framework consists of two modules: Localizer and Answerer, where both are parameter-efficiently fine-tuned from BLIP-2. We propose two ways of chaining these modules for cascaded inference and self-refinement. First, in the forward chain, the Localizer finds multiple language-aware keyframes in a video, which the Answerer uses to predict the answer. Second, in the reverse chain, the Answerer generates keyframe pseudo-labels to refine the Localizer, alleviating the need for expensive video moment localization annotations. Our SeViLA framework outperforms several strong baselines on 5 challenging video QA and event prediction benchmarks, and achieves the state-of-the-art in both fine-tuning (NExT-QA, STAR) and zero-shot (NExT-QA, STAR, How2QA, VLEP) settings. We also analyze the impact of Localizer, comparisons of Localizer with other temporal localization models, pre-training/self-refinement of Localizer, and varying the number of keyframes.
1 Introduction
SeViLA addresses the limits of uniformly sampled video frames by using one image-language model for language-aware keyframe localization and video question answering, with self-refinement between the two modules.
- Uniformly sampled frames often lack explicit language-aware temporal modeling, which can miss visual cues relevant to a query.
- SeViLA uses a single image-language model to handle temporal localization and question answering while avoiding expensive temporal grounding annotations.
- The forward chain passes Localizer-selected keyframes to the Answerer, while the reverse chain uses Answerer feedback to refine the Localizer.
- SeViLA evaluates five challenging video question answering and event prediction benchmarks and reports state-of-the-art performance in fine-tuning and zero-shot settings.
- The framework contributes a single-image-language-model design, pseudo-label self-refinement, strong benchmark performance, and analysis of key design choices.
2 Related Work
Related work spans image-language pre-training, image-to-video transfer learning, and language-aware keyframe localization as approaches to efficient video-language understanding.
- Image-Language Pre-trained Models: Image-language models have advanced faster than video-language models in model and pre-training data scale because image data are more accessible and structurally simpler.
- Image-to-Video Transfer Learning: Image-to-video transfer methods leverage limited video frames to improve video-language learning efficiency.
- Language-aware Keyframe Localization: Language-aware localization methods select frames or video clips for downstream question answering using answer labels, separate models, iterative training, or retrieval.
3 Method: SeViLA
SeViLA adapts BLIP-2 into a Localizer and Answerer, chaining language-aware keyframe selection with answer generation and pseudo-label refinement.
- 3.1 Preliminaries: BLIP-2: SeViLA uses BLIP-2 as a backbone, with a frozen image encoder, frozen language model, and trainable Q-Former that bridges visual features and language generation.
- 3.2 Self-Chained Video Localization-Answering: The Localizer scores uniformly sampled frames against question, options, and a localization prompt, then selects the top-k language-aware keyframes.
- 3.2 Self-Chained Video Localization-Answering: The Answerer processes Localizer keyframes with a separate Q-Former and concatenates their features with language contexts to generate a video-level answer.
- 3.3 Training Answerer and Localizer via Self-Chaining: In the forward chain, Localizer keyframes support Answerer inference and fine-tuning; in the reverse chain, Answerer-derived pseudo-labels train the Localizer without costly frame-level annotations.
- 3.3 Training Answerer and Localizer via Self-Chaining: Localizer pre-training transfers supervision from QVHighlights video moment retrieval by assigning frame labels from annotated temporal spans.
4 Experiments
Experiments evaluate SeViLA across video question answering, event prediction, and moment retrieval, showing gains from temporal modeling, language-aware keyframe selection, and self-refinement.
- Experimental setup: SeViLA is evaluated on four video QA benchmarks, one event-prediction dataset, and the QVHighlights moment-retrieval benchmark.The experiments include both fine-tuning and zero-shot comparisons, ablations, and stand-alone Localizer evaluation.
- Fine-tuning comparison: 69.0% vs. 54.8%: BLIP-2concat outperforms BLIP-2voting on STAR-Sequence, indicating the importance of temporal modeling.BLIP-2voting processes frames independently, whereas BLIP-2concat concatenates visual features for temporal processing.
- Fine-tuning comparison: 5.3%: SeViLA† averages this advantage over the top video-LM, InternVideo, across all evaluated tasks.SeViLA† also improves over uniformly sampled BLIP-2concat on NExT-QA (+1.2%), STAR (+0.7%), How2QA (+1.5%), and VLEP (+0.4%).
- Fine-tuning comparison: SeViLA achieves new state-of-the-art fine-tuning performance on NExT-QA, STAR, TVQA, and VLEP after self-refining its Localizer with pseudo-labels.Relative to SeViLA†, refinement increases performance on NExT-QA (0.4%), STAR (+2.2%), and TVQA (+1.9%).
- Zero-shot comparison: BLIP-2voting outperforms InternVideo on NExT-QA (+13.6%), How2QA (+7.6%), and VLEP (+5.1%) despite lacking inter-frame temporal modeling.On How2QA, it also surpasses FrozenBiLM by 11.4%, highlighting the potential of image-LMs for video-language tasks.
- Zero-shot comparison: Keyframe selection improves zero-shot performance over uniform sampling on NExT-QA (+1.2%), STAR (+2.4%), How2QA (+1.5%), TVQA (+1.6%), and VLEP (+0.4%).On STAR, SeViLA† also outperforms zero-shot Flamingo with 80B parameters by 4.9%.
- Ablation studies: Sparse frames outperform dense frames, while QVHighlights pre-training and self-refinement each provide additional Localizer benefits.Using keyframes improves the zero-shot Answerer by an average of 2.1% after pseudo-label refinement, and fine-tuning gains average 0.7% with keyframes plus 1.5% from refinement.
- Localizer evaluation: The Localizer produces binary frame-level predictions that are aggregated into video-level temporal spans for QVHighlights moment retrieval.The evaluation extracts frames at 0.5 fps before merging frame predictions into temporal span predictions.
5 Conclusion and Future Work
SeViLA combines language-aware temporal localization and question answering through forward chaining, while reverse-chain pseudo-labeling refines the Localizer without expensive keyframe annotations. The framework achieves competitive or better performance across five video QA and event prediction benchmarks.
- SeViLA adapts an image-language model into a Localizer for temporal localization and an Answerer for question answering on keyframes.The Localizer output feeds the Answerer in the forward chain.
- Answerer-generated pseudo-labels refine the Localizer through a backward chain, reducing reliance on expensive language-aware keyframe annotations.
- SeViLA achieves competitive or better performance than state-of-the-art baselines on five video question answering and event prediction benchmarks.
- The paper provides comprehensive analysis of the proposed two-stage self-chaining and encourages further work on temporal localization in video understanding.
Appendix
The appendix supplements the paper with comparisons, experimental details, additional experiments, limitations, broader impacts, and licensing information.
- The appendix compares image-language and video-language models by model size and pre-training data size.
- It documents benchmark and task definitions, metrics, baseline implementations, and SeViLA implementation details.
- Additional experiments examine Localizer frame count, iterative self-refinement, pre-training settings, alternative image-language models, computational cost, prompts, and qualitative visualizations.
- The appendix discusses limitations and broader impact and provides license information for the datasets, code, and models used.
6 Comparison Between Image-LMs and Video-LMs
The appendix compares image-language and video-language models in scale and pre-training data. Recent video-language models generally operate at smaller model and data scales.
- Recent video-language models generally have smaller model and pre-training data scales than image-language models.Figure 5 visualizes the comparison across both dimensions.
- The scale gap is attributed to the higher cost of video-level annotations compared with image-level annotations.
7 Experimental Setup
The experimental setup covers benchmarks, metrics, baselines, SeViLA architecture and training, prompt selection, Localizer pre-training, and frame-level span aggregation.
- Benchmarks: Experiments evaluate video QA and event prediction on NExT-QA, STAR, How2QA, TVQA, and VLEP, plus moment retrieval on QVHighlights.The listed benchmarks cover causal, temporal, descriptive, situated-reasoning, future-event, and moment-retrieval settings.
- Metrics: Video QA and event prediction use answer accuracy, while moment retrieval uses mAP across IoU thresholds and Recall@1 at IoU thresholds 0.5 and 0.7.
- Baselines: Baseline comparisons include InternVideo, BLIP-2 voting and concatenation, CLIP, Moment-DETR, ATP, and Differentiable Top-K under specified frame-sampling procedures.
- Architecture: SeViLA uses BLIP-2 with frozen visual and language components, fine-tuning only the Q-former and post-Q-former layers, totaling 106M trained parameters.The trained parameters comprise 2.5% of BLIP-2's total parameters.
- Training: The framework trains Localizer and Answerer components on downstream datasets, using pseudo-labels for Localizer self-refinement and answer labels for Answerer fine-tuning.
- Prompts: Prompt engineering selects among multiple QA and localization prompts according to downstream zero-shot performance.
- Localizer: Localizer pre-training transfers QVHighlights temporal spans into frame-level keyframe labels using frame timestamps and query prompts.
- Localizer: Frame-level Localizer predictions are aggregated into temporal spans using a span threshold that splits spans after excessive continuing negative predictions.The reported span threshold is 6.
8 Experiments
The experiments examine self-refinement, pre-training, model variants, computational cost, prompt sensitivity, and visualizations of SeViLA’s Localizer and Answerer.
- Single-frame versus multi-frame Localizer: A four-frame Localizer performs worse than a single-frame Localizer in both zero-shot and fine-tuning settings.The authors attribute this to BLIP-2 lacking video-data pre-training and leave improved multi-frame localization to future work.
- Self-refinement: Iterative self-refinement with two iterations slightly improves performance over a single self-refinement.The Answerer supplies pseudo-labels to train the Localizer, which then provides language-aware frames for Answerer fine-tuning.
- Localizer pre-training: Weakly supervised ASR pre-training improves Localizer performance and closes the gap with manually annotated QVHighlights pre-training.
- Alternative image-language model: Zero-shot MiniGPT4 Answerer accuracy on NExT-QA is 52.7%, with a further 0.7% boost from the MiniGPT4 Localizer.
- Computational cost: Adding the Localizer has a very small additional memory footprint because it shares most parameters with the Answerer.
- Prompt design: The model is insensitive to changes in localization prompts in the reported zero-shot Video QA experiment.
- Visualization: Compared with uniform sampling, the Localizer identifies frames matching human annotations across keyframe amounts and supports correct answers.
9 Limitations and Broader Impacts
SeViLA’s frame-level Localizer supports language-aware keyframe localization across many real-world video tasks but may struggle with fine-grained temporal events. Because it relies on a large internet-trained image-language model, it may also produce inappropriate or socially biased responses.
- Limitations: Frame-level keyframe localization may handle complex fine-grained events poorly, such as distinguishing opening a door from closing it.Future work could explore structured temporal localization beyond the frame level.
- Broader impacts: SeViLA may occasionally generate unexpected or inappropriate responses reflecting biases related to gender, race, or sexuality.The authors call for further studies to evaluate and mitigate bias and toxic output in large image-language models.
10 License
The authors plan to publicly release the code and models and use standard community licenses for the datasets, code, and models used in the paper.
- License: The code and models will be made publicly accessible under standard community licenses.The paper directs readers to specific license links for further information.
- License: QVHighlights is used under the CC BY-NC-SA 4.0 license.
- License: Huggingface Transformers uses the Apache license, while Torchvision uses the BSD 3-Clause license.