PaperlayerView exact revision on arXiv

Source-linked AI summary

PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning

Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, Jiashi Feng

arXiv:2404.16994v2cs.CV

TL;DR

Video-text fine-tuning is costly in computation and annotations, motivating efficient adaptation of image-pretrained models. PLLaVA combines pooling with post-training model merging and achieves state-of-the-art results across video-understanding benchmarks, including a 13.7% average margin over VideoChat2 on MVBench.

  • Problem

    How to adapt image-pretrained MLLMs for video understanding remains important because video-text fine-tuning carries high computation and annotation costs.

    Source: §1 · p. 2

  • Method

    PLLaVA uses pooling to balance temporal context and spatial information, alongside post-training model merging to reduce forgetting during multimodal fine-tuning.

    Source: §1 · pp. 2–3

  • Results

    PLLaVA achieves state-of-the-art results across video-understanding benchmarks, surpassing VideoChat2 by 13.7% on average across MVBench’s 20 tasks.

    Source: §1 · p. 3, §4.3 · p. 10

  • Takeaways & Limitations

    PLLaVA makes training with more data and larger language models more controllable with respect to over-training and performance saturation.

    Source: §5 · p. 12

Abstract

from arXiv · show

Vision-language pre-training has significantly elevated performance across a wide range of image-language applications. Yet, the pre-training process for video-related tasks demands exceptionally large computational and data resources, which hinders the progress of video-language models. This paper investigates a straight-forward, highly efficient, and resource-light approach to adapting an existing image-language pre-trained model for dense video understanding. Our preliminary experiments reveal that directly fine-tuning pre-trained image-language models with multiple frames as inputs on video datasets leads to performance saturation or even a drop. Our further investigation reveals that it is largely attributed to the bias of learned high-norm visual features. Motivated by this finding, we propose a simple but effective pooling strategy to smooth the feature distribution along the temporal dimension and thus reduce the dominant impacts from the extreme features. The new model is termed Pooling LLaVA, or PLLaVA in short. PLLaVA achieves new state-of-the-art performance on modern benchmark datasets for both video question-answer and captioning tasks. Notably, on the recent popular VideoChatGPT benchmark, PLLaVA achieves a score of 3.48 out of 5 on average of five evaluated dimensions, exceeding the previous SOTA results from GPT4V (IG-VLM) by 9%. On the latest multi-choice benchmark MVBench, PLLaVA achieves 58.1% accuracy on average across 20 sub-tasks, 14.5% higher than GPT4V (IG-VLM). Code is available at https://pllava.github.io/

1 Introduction

The introduction argues that adapting image-pretrained MLLMs to video can avoid the compute and annotation costs of video-specific pretraining, but direct multi-frame fine-tuning has important failure modes. It proposes temporal pooling and post-training weight fusion to address caption degradation and forgetting, and reports state-of-the-art performance across video-understanding benchmarks.

1 Introduction

  • Directly fine-tuning image MLLMs on video does not always improve performance, increases vulnerability to prompt changes, and gains no video-understanding benefit from a larger language model.The authors also motivate adapting image-pretrained MLLMs as a more pragmatic alternative to the high compute and annotation costs of video-specific training.Source: §1 · p. 2
  • Dominant high-norm visual tokens can suppress other features, shortening captions and causing prompt-template collapse; adding frames can mitigate suppression but increases memory use.The authors attribute these effects to limited information in the image encoder and softmax self-attention; Figure 2 is described as showing shorter outputs with more training and out-of-distribution prompts.Source: §1 · p. 2, Figure 2 · p. 3
  • Pooling is proposed to balance training efficiency and caption accuracy while preserving spatial information and the benefits of a larger temporal receptive field.The introduction contrasts this goal with direct spatial-temporal averaging, which loses spatial information and does not optimally scale with training data.Source: §1 · pp. 2–3
  • Post-training weight fusion combines image-pretrained and video-fine-tuned weights to reduce forgetting, enabling a 34B-LLM video model without creating high-quality video datasets.The authors attribute degradation in stronger LLMs partly to short question-answer video data, which can weaken learned object and spatial-relation descriptions.Source: §1 · p. 3
  • Top 1M Panda-70M videos are recaptioned with dense, accurate bilingual captions, alongside claimed state-of-the-art results across video-understanding benchmarks, especially dense video captioning.The paper describes these as contributions of Pool-LLaVA.Source: §1 · p. 3

2 Related Works

Related work adapts language models to video through learned interfaces, temporal compression, and pipelines that convert video into text or executable programs. These approaches balance richer video understanding against computational and memory costs.

Video Multimodal Large Language Models

  • Video MLLMs commonly use projection networks [33] [25] [21], inter-modality attention [19] [20], or modality perceivers [50] [37] [14] to convert video into tokens for LLMs.These interfaces connect video spatial-temporal dynamics with LLM processing capabilities [40].Source: §2 · p. 3
  • Parameter-efficient learning schemes [11, 23, 7] reduce cost; BLIP [18] freezes the vision encoder and learns a Q-Former, while Video-ChatGPT [33] introduced video instruction tuning and a benchmark dataset.BLIP demonstrated zero-shot Video Question Answering capabilities, while Video-ChatGPT established video-based text-generation benchmarks.Source: §2 · pp. 3–4
  • Later models refine alignment and multimodal processing through cross-attention, bootstrapping, shared image-video projections, and audio integration.VideoChat [19] uses cross-attention; VideoChat2 [20] uses multi-stage bootstrapping; VILA [26] proposes advanced training recipes; Video-LLaVA [25] shares projections across images and videos; CAT [46] adds audio.Source: §2 · p. 4

Long-video modeling

  • Long-video methods reduce token, computation, or memory demands using frame merging, dynamic token merging, dual-token compression, temporal condensation, or keyframe selection.Examples include MovieChat [37], Chat-UniVi [14], LLaMA-VID [21], Vista-LLaMA, and approaches selecting keyframes; VTimeLLM introduces a question-answering dataset focused on video boundaries.Source: §2 · p. 4

Pipelined Video Understanding

  • Pipelined systems either turn video into textual descriptions for LLM reasoning or use LLM-generated programs to analyze visual content.VideoChat-Text [19] and LLoVi [49] use video descriptions, while ViperGPT [39] and ProViQ [6] generate programs or Python scripts for video queries.Source: §2 · p. 4

3 Method & Analysis

Section 3 analyzes why adapting image MLLMs to video can fail, then presents PLLaVA’s pooling approach and a post-training optimization to address data- and model-scaling degradation.

3.1 Failure Cases Analysis for Applying Image MLLMs

  • The n-frame baseline separately encodes selected video frames and concatenates their features as input to the image MLLM.It is intended to let the LLM interpret temporal information across the encoded frames.Source: §3.1 · p. 5
  • n-frame models become prompt-sensitive and suffer data-scaling failures, with OOD performance degrading beyond 0.48M training samples.Under OOD prompts, longer training can shorten generations or eliminate responses; high-norm dominant tokens emerge with more training data and may be correlated with this degradation. VideoChatGPT also deteriorates when additional video data is introduced.Source: §3.1 · pp. 5–6, Figure 4 · p. 6, Table 1 · p. 6

3.2 Model Scaling Degradation

  • 7B, 13B, and 34B IG-VLM models show almost no performance difference, while pooled LLaVA-Next 34B performs worse than 13B.The authors attribute IG-VLM’s scaling limits to grid-view resolution constraints and their own models’ shorter generations with larger LLMs to low-quality video-text data undermining language ability.Source: §3.2 · pp. 6–7, Figure 5 · p. 6

3.3 PLLaVA

  • PLLaVA uses parameter-free adaptive average pooling to reduce video-feature dimensions before feeding them to the LLM.The approach addresses the need to retain temporal information while limiting long-video input costs; grid search favors spatial pooling, while temporal pooling decreases performance.Source: §3.3 · p. 7, Figure 6 · p. 7

3.4 Post Optimization

  • Post Optimization blends the video-trained LLM, including its LoRA weights, with the original image MLLM LLM; lower α yields significantly better generative performance.The mix ratio is adjusted by varying α during inference.Source: §3.4 · p. 8

4 Experiments

PLLaVA adapts image MLLMs for video using feature pooling, and its reported evaluations show strong performance across video question answering, captioning, and multi-choice understanding. The experiments also examine pooling choices, prompt robustness, and image–video weight fusion.

4.1 Experiment Setting

  • Training uses 783k instructional-tuning samples, and evaluation covers open-ended VideoQA, VCG Score, and 20-task MVBench.PLLaVA is built on LLaVA Next 7B, 13B, and 34B; it uses 16 uniformly sampled frames and average pooling to a 16 × 12 × 12 × d shape, with results reported after 6,250 training steps.Source: §4.1 · p. 8

4.2 Impact of Pooling Operation Design

  • Spatial downsampling to 12×12 preserves performance after 50% reduction, while further reduction causes a significant drop; temporal pooling consistently lowers performance.The experiments compare pooling shapes and find better performance at lower temporal downsampling rates; pooling also stabilizes response lengths and prompt robustness relative to the 4-frame method by smoothing dominant high-norm tokens.Source: §4.2 · pp. 9–10, Figure 7 · p. 9

4.3 Quatitative Results

  • PLLaVA 34B outperforms GPT-4V by 3.6, 4.9, 3.9, and 15.3 on MSVD, MSRVTT, ActivityNet, and TGIF, respectively.It also outperforms existing methods on both Accuracy and Score across these benchmarks; PLLaVA 7B and 13B exceed all baselines on Score.Source: Table 2 · p. 10, §4.3 · p. 10
  • PLLaVA’s 7B, 13B, and 34B versions exceed their same-size counterparts’ average VCG score by 2.9%, 7.1%, and 12.6%, respectively.PLLaVA 34B exceeds the previous SOTA by 5.8% on CI, 6.7% on DO, and 9.2% on CU; it exceeds IG-VLM LLaVA 34B by 6% on TU.Source: §4.3 · p. 10
  • PLLaVA surpasses VideoChat2 by 13.7% on average across MVBench’s 20 tasks and performs well on 17 of them.The passage notes remaining weaknesses on counterfactual inference and object shuffle, which require reasoning and imagination.Source: §4.3 · pp. 10–11, Table 3 · p. 11

4.4 Analysis

  • Spatial neighbors have higher token similarity than temporal neighbors, supporting the authors’ explanation that temporal pooling can distort frame features.The analysis attributes the difference to spatial pooling being local to individual frames, while temporal pooling alters original frame features; the authors note LLMs can model temporal relations without preprocessing temporal aggregation.Source: §4.4 · p. 11, Figure 8 · p. 11
  • MVBench performs best near LoRA alpha 20 and VCG Score near alpha 4, while combining image and video weights outperforms either extreme.The authors relate the different peaks to MVBench’s multiple-choice format versus VCG Score’s longer generations.Source: §4.4 · pp. 11–12, Figure 9 · p. 12

4.5 Case Studies; 4.6 Dense Recaption

  • In case studies, PLLaVA 34B describes more video details than IG-VLM and identifies badminton rather than volleyball; its recaption example captures detail and motion better than the Open-Sora GPT-4 pipeline.The authors suggest pooling reduces information loss compared with concatenating frames into a grid, and report contributing a 1K-video Inter4K caption dataset.Source: §4.5 · p. 12, §4.6 · p. 12

5 Conclusion

The conclusion presents PLLaVA as a simple, effective approach to extending image-language models to video. It highlights more controllable scaling and detailed captioning as contributions to multimodal research.

5 Conclusion

  • PLLaVA is a simple, highly effective method for extending image-language models to videos.The authors describe it as an initial investigation into this extension.Source: §5 · p. 12
  • PLLaVA enables scaling training with more data and larger language models while making over-training and performance saturation more controllable.Source: §5 · p. 12
  • Its ability to produce detailed captions contributes to multimodal understanding and generation.Source: §5 · p. 12

Related papers

Loading 2404.16994v2…