Source-linked AI summary
LLaVA-Video: Video Instruction Tuning With Synthetic Data
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, Chunyuan Li
TL;DR
Curating high-quality video-language instruction data is difficult because existing datasets often contain static, simplified videos and sparse frame sampling that misses detailed changes. The paper addresses this with the synthetic LLaVA-Video-178K dataset and LLaVA-Video models, which achieve strong performance across video benchmarks, including results comparable to Gemini-1.5-Flash for the 72B model.
Problem
High-quality video-language instruction data is difficult to obtain because existing datasets often use static or simplified videos and sparse frame sampling that misses detailed movements and changes.
Method
The paper constructs LLaVA-Video-178K from dynamic videos with dense annotations and trains LLaVA-Video using this dataset alongside existing visual instruction data and a SlowFast video representation.
Results
LLaVA-Video achieves strong performance across video benchmarks; its 72B model performs as well as Gemini-1.5-Flash, while the 7B model outperforms LLaVA-OV-7B on seven of ten datasets.
Takeaways & Limitations
The results support synthetic video instruction data as an effective basis for developing open video large multimodal models.
Takeaways & Limitations
The dataset’s diverse video sources may introduce source biases, and annotators’ perspectives may skew the question-answer pairs.
Abstract
from arXiv · showhide
The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we propose an alternative approach by creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.
1 Introduction
LLaVA-Video addresses limitations in video instruction data by introducing a large synthetic dataset and a video model trained with it alongside visual instruction data. The approach emphasizes richer temporal content, diverse instruction tasks, and efficient frame use.
- Motivation: Existing video instruction datasets often contain static videos, simplified plots, and sparse frame annotations that miss detailed movements and changes.ShareGPT4Video, for example, sometimes samples only 2 frames from a 30-second video, which can lead to hallucinations in detailed descriptions.
- Contribution: LLaVA-Video-178K contains 178,510 videos ranging from 0 to 3 minutes, with detailed captions, open-ended questions, and multiple-choice questions.The annotations were developed through GPT-4o and human efforts.
- Method: LLaVA-Video SlowFast distributes visual tokens across frames to support increased frame sampling under limited GPU memory.The model development was motivated by findings that frame count significantly affects performance for this dataset.
- Contribution: The project presents LLaVA-Video, a family of video-language models trained using the proposed instruction-following data and existing visual instruction tuning data.The stated contributions also include releasing the instruction data, codebase, model checkpoints, and a visual chat demo.
2 Related Work
The related-work discussion frames video instruction tuning as constrained by expensive manual annotation and describes synthetic-data approaches that differ in how they construct temporally informed annotations.
- Instruction-following data: The paper aims to move beyond simple captions by supporting detailed descriptions, open-ended video question-answering, and multiple-choice video question-answering.This goal motivates the construction of a broader video-language instruction-following dataset.
- Video sources: Figure 1 relates ten selected video sources to existing video-language datasets and lists the filtering logic used to construct the video pool.The filters include view-based sorting, scene-count, duration, scene-duration ratio, resolution, and category sampling criteria.
- Existing datasets: Manual video-language annotation covers captioning and question-answering tasks but is expensive, limiting dataset size.The cited prior datasets include manually annotated video captions and video question-answering data.
- Synthetic data: Unlike Video ReCap, the paper generates clip-wise descriptions with historical context so prior events remain connected to the current event.The related-work passage presents this as a distinction between the two synthetic video instruction-tuning pipelines.
3 Video Instruction-Following Data Synthesis
The dataset synthesis pipeline targets diverse, dynamically selected videos and generates detailed, temporally informed annotations across captions and question-answering tasks. The resulting collection contains 178K videos and 1.3M instruction-following samples spanning captions, open-ended QA, and multiple-choice QA.
- 3.1 Video source: The authors prioritize richness and diversity in video content and language annotations, drawing from ten video sources that support over 40 benchmarks.Videos are selected for significant temporal dynamics to broaden the video pool.
- 3.1 Video source: PySceneDetect identifies dynamic videos by counting scenes, while a separate filter excludes videos that mainly contain slides.The number of scenes is used as an indicator of video dynamism.
- 3.2 Video Detail Description: GPT-4o generates descriptions from frames sampled at 1 FPS through a sequential three-level pipeline designed for videos of any length.Sequential processing addresses GPT-4o input-size constraints.
- 3.2 Video Detail Description: Level-1 descriptions summarize 10-second segments using current frames and historical context, while level-2 summaries update every 30 seconds from recent level-1 descriptions.Historical context includes recent unsummarized level-1 descriptions and the latest level-2 summary.
- 3.3 Video Question Answering: The pipeline organizes video questions into 16 categories to support complex interactions and assess perceptual and reasoning skills.GPT-4o generates at most one question-answer pair per category from detailed descriptions, task definitions, examples, and the current video description.
- 3.3 Video Question Answering: Generated question-answer pairs are filtered by removing duplicates and discarding answers beginning with unsupported-response phrases such as “does not specify” or “does not show.”The filtering strategy uses sentence-transformer representations for duplicate removal.
- 3.4 Dataset Statistics: 178K videos yield 1.3M instruction-following samples: 178K captions, 960K open-ended QAs, and 196K multiple-choice QAs.The collection is described as balanced and comprehensive across datasets and question types.
- 3.4 Dataset Statistics: Compared with prior synthetic datasets, the collection emphasizes dynamic untrimmed videos, 1 FPS sampling, and diverse caption, open-ended, and closed-form QA tasks.The authors contrast this design with datasets containing mostly static videos, simpler plots, or much lower sampling rates.
4 Experiments
The experiments evaluate LLaVA-Video across video benchmarks and training-data configurations, finding strong overall performance and substantial benefits from LLaVA-Video-178K. They also study video representation and dataset composition under standardized evaluation settings.
- Video Representation: LLaVA-Video SlowFast balances frame count and visual-token count by assigning different pooling rates to slow and fast frame groups.Its configuration is parameterized as V = (T, M, s, p), within limited context-window and GPU-memory budgets.
- Evaluation Setup: LLaVA-Video was evaluated on 11 video benchmarks spanning captioning, open-ended QA, and multiple-choice QA.The evaluation includes benchmarks for long-video understanding and other video tasks.
- Training Configuration: The training mixture combines LLaVA-Video-178K with existing video, QA, and image instruction data, totaling 1.6 million video-language samples.The model is fine-tuned from the LLaVA-OneVision single-image checkpoint.
- Overall Results: The 72B model performs as well as Gemini-1.5-Flash, while the 7B model outperforms LLaVA-OV-7B on seven of ten datasets.The reported comparisons use benchmark results summarized in Table 2.
- Dataset Ablation: Adding LLaVA-Video-178K improved both in-domain and out-of-domain performance, including a 31.9-point increase on NExT-QA and a 9.1-point rise on VideoMME.The ablation progressively adds datasets to an LLaVA-OneVision single-image baseline.
5 Conclusion
The paper introduces LLaVA-Video-178K, a synthetic dataset for video-language instruction-following, and LLaVA-Video models trained with it and existing visual instruction data. The dataset and models support dense video representation and achieve strong performance across video benchmarks.
- LLaVA-Video-178K is a high-quality synthetic dataset covering detailed captioning, open-ended QA, and multiple-choice QA.
- Training with LLaVA-Video-178K and existing visual instruction data produced the LLaVA-Video model family.
- LLaVA-Video uses a video representation designed to use GPU resources effectively and include more frames during training.
- Experiments report strong performance for LLaVA-Video models across a wide range of video benchmarks.
6 Limitations
The paper identifies source and annotation biases as limitations, while describing video representation as a trade-off between frame coverage, visual-token count, and computational constraints. Its SlowFast design assigns different pooling rates to frame groups, but visual-token reduction may affect information preservation.
- Videos sourced from various platforms may introduce source-specific biases, while annotator perspectives may skew question-answer pairs.
- LLaVA-Video represents videos with up to T frames and M tokens per frame, making token allocation central to its representation design.
- Video LMMs must manage the computational cost of representing every frame while preserving enough visual tokens for video comprehension.
- The SlowFast representation divides frames into slow and fast groups according to strike rate s and applies different pooling rates.
- The special case s = 1 reduces SlowFast to the original simple representation.
B.1 Video Detail Description
The paper studies detailed video description generation and shows that historical context helps maintain character identity across time. It also presents the associated captioning figure and the question types and examples used in the dataset.
- Including historical context helps identify characters across different times as the same individual.
- Figure 9 compares video caption generation with and without historical context.
- The dataset documentation lists question types, their proportions, generation prompts, and an example combining a video, detailed description, open-ended question, and multiple-choice question.
B.3 Dataset Comparison
LLaVA-Video-178K is compared with other video-language datasets across captioning and question-answering tasks. Unlike the other datasets, it includes captions, open-ended questions, and multiple-choice questions.
- The comparison covers video captioning and video question-answering datasets organized into four groups by text-annotation method.
- LLaVA-Video-178K uniquely combines captions, open-ended questions, and multiple-choice questions.
C Beyond Singularity: Extensive Sampling Matters
The experiments examine how frame count and visual-token allocation affect video understanding. More training frames generally improve performance, while balanced allocation and restrained inference sampling avoid degradation.
- The experiments evaluate frame count and visual tokens per frame in a video-only setting using 0-to-30-second training videos.
- Dynamic 30-second videos may require more than 100 frames because sparse sampling can miss content occurring at 15 FPS.
- 110 frames improves performance across all datasets compared with 32 frames, with 110 frames set as the upper limit to avoid GPU overload.
- Earlier frame-count saturation may reflect static training videos, whereas dynamic videos and detailed annotations support continued benefits from extensive sampling.
- Increasing inference frames modestly helps, but excessive inference-frame counts can reduce performance.
- More frames with fewer tokens per frame can outperform fewer frames with more tokens, even with fewer total visual tokens.The reported totals are 18,590 versus 21,632 visual tokens.
D Capabilities
LLaVA-Video demonstrates video-understanding capabilities beyond benchmark performance, including real-world knowledge, optical-illusion recognition, specialized-domain understanding, unusual-action interpretation, and physical-law reasoning.
- LLaVA-Video exhibits capabilities in understanding videos using real-world knowledge beyond benchmark performance.
- Special Domain: LLaVA-Video understands specialized video domains, including sketches and video-game fights.
- Optical Illusion: LLaVA-Video recognizes that a green dragon is a 2D cutout producing an optical illusion rather than a real 3D object.
- Unusual Action: LLaVA-Video identifies physical therapy for pets as an atypical action beyond ordinary activities.
- Physical Laws: LLaVA-Video explains that a yellowish substance remains suspended because the video depicts zero gravity.