Source-linked AI summary
PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, Jiashi Feng
TL;DR
Video-language pre-training is costly in computation and annotation, motivating efficient adaptation of image MLLMs to video. PLLaVA uses pooling and post-training weight fusion to stabilize multi-frame learning and preserve image-domain knowledge, achieving state-of-the-art results across video question-answering and captioning benchmarks.
Problem
Adapting image MLLMs to video is attractive because video-language pre-training requires substantial computational resources and annotated data.
Method
PLLaVA combines temporal feature pooling with post-training fusion of image-pretrained and video-finetuned weights.
Results
PLLaVA achieves new state-of-the-art performance across video understanding benchmarks, including video question-answering and dense captioning.
Takeaways & Limitations
PLLaVA provides a simple, efficient approach for scaling image MLLMs to video while producing detailed captions and supporting larger language models.
Abstract
from arXiv · showhide
Vision-language pre-training has significantly elevated performance across a wide range of image-language applications. Yet, the pre-training process for video-related tasks demands exceptionally large computational and data resources, which hinders the progress of video-language models. This paper investigates a straight-forward, highly efficient, and resource-light approach to adapting an existing image-language pre-trained model for dense video understanding. Our preliminary experiments reveal that directly fine-tuning pre-trained image-language models with multiple frames as inputs on video datasets leads to performance saturation or even a drop. Our further investigation reveals that it is largely attributed to the bias of learned high-norm visual features. Motivated by this finding, we propose a simple but effective pooling strategy to smooth the feature distribution along the temporal dimension and thus reduce the dominant impacts from the extreme features. The new model is termed Pooling LLaVA, or PLLaVA in short. PLLaVA achieves new state-of-the-art performance on modern benchmark datasets for both video question-answer and captioning tasks. Notably, on the recent popular VideoChatGPT benchmark, PLLaVA achieves a score of 3.48 out of 5 on average of five evaluated dimensions, exceeding the previous SOTA results from GPT4V (IG-VLM) by 9%. On the latest multi-choice benchmark MVBench, PLLaVA achieves 58.1% accuracy on average across 20 sub-tasks, 14.5% higher than GPT4V (IG-VLM). Code is available at https://pllava.github.io/
1 Introduction
Adapting image MLLMs to video can be resource-efficient but exposes prompt vulnerability, short or missing responses, and forgetting of image-domain capabilities. PLLaVA addresses these issues with pooling and post-training weight fusion, achieving state-of-the-art video understanding performance.
- Adapting image MLLMs to video avoids the high computational and annotation costs of training video-language models from scratch.
- Directly fine-tuning image MLLMs with multiple frames can reduce performance and make generation vulnerable to prompt changes.Out-of-distribution prompts may produce very short descriptions or no response.
- High-norm visual feature tokens can suppress other tokens during self-attention, producing shorter, lower-quality descriptions and instability under changed prompts.Adding more frames can mitigate suppression but substantially increases memory consumption.
- Pooling smooths frame features while preserving the benefits of a larger temporal receptive field without relying solely on more input frames.The method targets a balance between spatial resolution, temporal coverage, and computation cost.
- Post-training optimization merges image-pretraining and video-fine-tuning weights to reduce forgetting without creating additional high-quality datasets.The merged model supports a 34B LLM.
- PLLaVA achieves new state-of-the-art results across video understanding benchmarks, especially for dense video captioning.The authors also use PLLaVA to recaption the top 1M Panda-70M videos with dense bilingual captions.
2 Related Works
Related work develops video-language models through learnable interfaces, computational adaptations, temporal modeling, and pipelines that convert video into text for LLM processing.
- Video MLLMs commonly connect video inputs to LLMs through projection networks, inter-modality attention, or modality perceivers.These interfaces transform spatial-temporal video content into sequences processed by LLMs.
- BLIP reduces video-processing cost by freezing the vision encoder and learning only a newly added Q-Former.Video-ChatGPT later introduced video instruction tuning and a high-quality instructional dataset.
- Long-video methods address high computational and memory demands with temporal modeling techniques designed to capture spatial details and temporal dynamics efficiently.
- Pipelined approaches translate video into textual descriptions before an LLM performs final reasoning or question answering.VideoChat-Text and LLoVi use detailed video descriptions to support comprehension over temporal sequences or long durations.
3 Method & Analysis
PLLaVA analyzes why direct multi-frame adaptation of image MLLMs becomes prompt-sensitive, saturates with more data, and degrades with larger models, then combines adaptive pooling with post-training model merging. Its framework reduces visual features before concatenating them with text embeddings and uses LoRA-based video adaptation, while experiments identify spatial pooling and lower merge ratios as favorable choices.
- Failure Cases Analysis: Direct n-frame adaptation can become vulnerable to prompt changes, with OOD responses shortening or disappearing as training continues.Under IND prompts, descriptions remain decent but tend to shorten; under OOD prompts, quality declines sharply, including no response after longer training.
- Failure Cases Analysis: Dominant high-norm vision tokens emerge as training samples increase and are plausibly correlated with degraded OOD generation.The analyses associate wider norm distributions and shorter, lower-quality descriptions with the n-frame setting.
- Failure Cases Analysis: The n-frame method saturates or degrades as training data and training time increase, especially under OOD prompts, whereas PLLaVA keeps improving until late training.Figure 4 reports stagnation for n-frame under IND prompts and substantial degradation under OOD prompts after training samples exceed 0.48M.
- Model Scaling Degradation: Video-ChatGPT also fails to benefit from additional video data, while scaling model size generally produces little improvement and can worsen performance.The reported degradation is associated with low-quality video-text data and shorter generations from larger MLLMs.
- PLLaVA: PLLaVA processes ViT-L and projector features with parameter-free adaptive average pooling, flattens them with question embeddings, and adapts the LLM using LoRA.The pooled representation reduces temporal and spatial dimensions before generation; trainable weights include the multimodal projector and LLM LoRA.
- PLLaVA: Grid-search analysis favors spatial pooling, associates temporal pooling with decreased performance, and finds lower post-training merge ratios improve generation.Post-training optimization blends the video-trained LLM with the original image-model LLM by varying α during inference.
4 Experiments
PLLaVA adapts image MLLMs to video using pooling, with experiments examining pooling design, robustness, scaling, and benchmark performance. Across VideoQA, video captioning, and MVBench, the model achieves strong results while remaining parameter-efficient.
- Implementation: PLLaVA uses average pooling on visual features before LLM generation, with 16 sampled frames and a target shape of 16 × 12 × 12 × d.
- Pooling design: 50% spatial downsampling does not degrade performance, whereas further spatial reduction causes a significant drop.
- Pooling design: Temporal pooling is performance-sensitive: all tested curves improve with lower downsampling rates, so temporal pooling consistently reduces performance.
- Pooling impact: Pooling produces more consistent visual-feature norms and text responses across training iterations and prompt changes than the 4-frame method.
- VideoQA: PLLaVA 34B outperforms existing methods on Accuracy and Score across MSVD, MSRVTT, ActivityNet, and TGIF VideoQA benchmarks.Its improvement margins over GPT-4V are 3.6, 4.9, 3.9, and 15.3 on the four benchmarks, respectively.
- Video captioning: PLLaVA achieves state-of-the-art average VCG scores, with 7B, 13B, and 34B versions exceeding same-size counterparts by 2.9%, 7.1%, and 12.6%.The 34B model also exceeds the previous SOTA on correctness, detail orientation, and context understanding by 5.8%, 6.7%, and 9.2%.
- MVBench: PLLaVA surpasses VideoChat2 by 13.7% on average across MVBench’s 20 tasks, performing strongly on 17 tasks but needing improvement on counterfactual inference and object shuffle.
- Qualitative analysis: Qualitative studies report that PLLaVA 34B captures more video details than IG-VLM and better represents caption details and motion than the Open-Sora GPT-4 pipeline.
5 Conclusion
The paper concludes that PLLaVA is a simple, parameter-efficient approach for extending image-language models to videos. Its detailed captions also support multimodal understanding and generation applications.
- PLLaVA makes scaling training to more data and larger language models easier, with a more controllable strategy for over-training and performance saturation.
- PLLaVA’s detailed captions contribute to multimodal understanding and generation, including video recaptioning with motion information.
- Case studies: In a saxophone-video example, PLLaVA describes clothing, background people, setting, and possible performance context beyond IG-VLM’s shorter description.
- Case studies: A Dutch windmill example contains a static scene with a body of water, moving sails, a building, a fence, and partly cloudy weather.