Source-linked AI summary
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, Jiaqi Wang
TL;DR
Existing video-caption data is often brief, while scalable methods struggle with temporal relations, detail, and arbitrary frame counts. The paper introduces DiffSW, ShareGPT4Video, and ShareCaptioner-Video, and reports gains across video understanding and generation, including SOTA performance on three benchmarks.
Problem
Existing brief video captions limit detailed video understanding and controllable generation, while large-scale high-quality annotation remains difficult.
Method
The paper uses differential sliding-window GPT4V captioning with semantic filtering and keyframe extraction, then develops ShareGPT4Video and ShareCaptioner-Video.
Results
ShareGPT4Video-8B reaches SOTA performance on three comprehensive video benchmarks, while the captioning resources support video understanding and generation experiments.
Takeaways & Limitations
The resulting resources provide dense temporal captions at scale, including 40K ShareGPT4Video pairs and 4.8M captions generated by ShareCaptioner-Video.
Takeaways & Limitations
The pipeline cannot incorporate audio information simultaneously because GPT4V lacks audio input.
Abstract
from arXiv · showhide
We present the ShareGPT4Video series, aiming to facilitate the video understanding of large video-language models (LVLMs) and the video generation of text-to-video models (T2VMs) via dense and precise captions. The series comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with various lengths and sources, developed through carefully designed data filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that reached SOTA performance on three advancing video benchmarks. To achieve this, taking aside the non-scalable costly human annotators, we find using GPT4V to caption video with a naive multi-frame or frame-concatenation input strategy leads to less detailed and sometimes temporal-confused results. We argue the challenge of designing a high-quality video captioning strategy lies in three aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame detailed content description. 3) Frame-number scalability for arbitrary-length videos. To this end, we meticulously designed a differential video captioning strategy, which is stable, scalable, and efficient for generating captions for videos with arbitrary resolution, aspect ratios, and length. Based on it, we construct ShareGPT4Video, which contains 40K high-quality videos spanning a wide range of categories, and the resulting captions encompass rich world knowledge, object attributes, camera movements, and crucially, detailed and precise temporal descriptions of events. Based on ShareGPT4Video, we further develop ShareCaptioner-Video, a superior captioner capable of efficiently generating high-quality captions for arbitrary videos...
1 Introduction
The paper addresses the shortage of detailed video captions by introducing a scalable differential captioning strategy and constructing resources for video understanding and generation.
- Brief existing captions limit detailed video understanding and controllable video generation, while comparable detailed-caption efforts remain scarce for videos.
- Large-scale high-quality video annotation is difficult because human captioning is costly and time-consuming, while available models and APIs have limited video capability.
- Naive multi-frame GPT4V captioning can confuse temporal relations and degrade as frame count increases, while frame concatenation loses detail.
- The authors identify three requirements: precise inter-frame temporal understanding, detailed intra-frame description, and scalability to arbitrary video lengths.
- DiffSW converts all-frames captioning into differential description by captioning the first frame and then comparing chronological two-frame sliding windows.
- ShareGPT4Video contains 40K diverse video-caption pairs with world knowledge, object attributes, camera movements, and precise temporal event descriptions.
- ShareCaptioner-Video scales annotation to 4.8M aesthetically appealing videos totaling about 3000 hours, while ShareGPT4Video-8B reaches SOTA on three video benchmarks.
2 ShareGPT4Video Dataset
ShareGPT4Video combines diverse video selection, semantic filtering and sampling, and differential GPT4V captioning to produce temporally rich annotations.
- The dataset pipeline selects diverse sources, filters semantically similar content, extracts semantic-aware keyframes, and applies differential sliding-window GPT4V captioning.
- Collection primarily focuses on videos shorter than two minutes to balance video duration against the number of available videos.
- Traditional keyframe methods may miss crucial changes and transitions, motivating sampling that reduces temporal redundancy while preserving semantic coherence.
- The dataset spans categories such as wildlife, cooking, sports, scenery, egocentric activities, and autonomous driving, with videos from 2 seconds to 2 minutes.
- Captions primarily contain 200 to 400 words, providing rich temporal information for video understanding and generation.
- Semantic-aware extraction samples frames at fixed 2-second intervals, retains frames below a semantic-similarity threshold, and always includes the last frame.
- DiffSW feeds GPT4V the current and previous keyframes plus the previous differential caption, using prompts to focus on changes such as posture, position, and camera angle.
- Hierarchical prompt design stabilizes output format and improves caption quality.
3 ShareCaptioner-Video
ShareCaptioner-Video is fine-tuned for flexible captioning workflows and is used to annotate millions of aesthetically appealing videos at scale.
- ShareCaptioner-Video is fine-tuned on collected video-caption data and organized into four capabilities: fast captioning, sliding captioning, clip summarizing, and prompt re-captioning.
- Fast Captioning: Fast captioning concatenates keyframes into a vertical image grid for rapid direct captioning, particularly on short videos.
- Sliding Captioning: Sliding captioning uses adjacent keyframes and the previous differential caption to describe events in a streaming format suitable for long videos.
- Clip Summarizing: Clip summarizing converts existing differential descriptions into a video caption without re-processing the frames.
- Prompt Re-Captioning: Prompt re-captioning reformulates user prompts into Sora-style formats aligned with the training captions for text-to-video inference.
- The model is fine-tuned end-to-end for one epoch with a batch size of 1024 and a learning rate warming to 2.5 × 10^-5.
- ShareCaptioner-Video annotates 4.8 million video clips totaling approximately 3000 hours using its sliding captioning mode.
4 Experiments
Experiments evaluate high-quality video captions for LVLM understanding and text-to-video generation. ShareGPT4Video-8B leads three video benchmarks, while detailed captions improve temporal comprehension and generation control.
- Video understanding: 61.5% average accuracy on TempCompass, an 11.6% increase over VideoLLaVA-7B.ShareGPT4Video-8B outperforms competitors in 19 of 20 TempCompass dimensions despite using 7B parameters.
- Video understanding: 2.7% and 8.2% average-accuracy gains over prior state of the art on VideoBench and MVBench, respectively.The model achieves the strongest performance across all three comprehensive benchmarks.
- Caption quality ablation: Detailed captions produce larger LVLM gains than short captions, which can even degrade performance on some benchmarks.The ablation attributes especially significant gains in temporal-sequence comprehension to high-quality caption data.
- Video captioning: ShareCaptioner-Video performs on par with GPT4V in human preference voting.The comparison evaluates caption quality using volunteer assessments of captions generated by both systems.
- Video generation: High-quality detailed captions improve adherence to complex prompts, including semantic content and camera movement.With brief captions, the text-to-video model struggles to follow complex generation prompts and produces subpar results.
5 Limitations and Social Impacts
The pipeline does not incorporate audio information, and the authors have not manually verified every generated caption for socially biased content. Video selections may also contain human faces, so users must follow the original source licenses.
- Limitations: The captioning pipeline cannot simultaneously incorporate audio information, which the authors identify as beneficial in daily human conversational scenarios.They plan to add audio once GPT4o supports audio input.
- Social Impacts: Generated captions were not manually verified for socially biased content because a large language model produced the large-scale captions.
- Social Impacts: The selected public-dataset videos may contain human faces, so users must adhere to the original video sources’ licenses when using them.
6 Conclusion
The study addresses the lack of high-quality video-caption data by developing ShareGPT4Video and ShareCaptioner-Video, whose captions support video understanding and generation. The authors report 40K detailed GPT4V captions and 4.8M ShareCaptioner-Video captions, with experiments validating their effectiveness.
- Conclusion: The study develops ShareGPT4Video and ShareCaptioner-Video to address the lack of high-quality video-caption data for LVLMs and T2VMs.
- Conclusion: 40K detailed captions are generated with GPT4V, while ShareCaptioner-Video produces 4.8M high-quality captions.
- Conclusion: The captions include world knowledge, object attributes, camera movements, and detailed temporal descriptions of events.
- Conclusion: Extensive experiments validate the dataset and captioner’s effectiveness for video understanding and generation tasks.
A Appendix
The appendix documents experimental setups, prompt design, caption-quality comparisons, dataset statistics, and additional results and analysis.
- A Appendix: The appendix provides more results and analysis alongside a summary of its contents.
- A Appendix: Appendix sections cover experimental setups, prompt design and templates, caption-quality comparisons, ShareGPT4Video statistics, and detailed pseudocode.
A.1 Experimental Details
ShareGPT4Video-8B is trained with AdamW and specified learning rates, while T2VM training uses staged image-video pretraining and reduced batch sizes.
- A.1 Experimental Details: T2VM training uses Latte-XL with T5-XXL, beginning with lower-frame pretraining and joint image-video training before reducing both batch sizes to 2.The first stage runs for 50k training steps.
A.2 Hierarchical Prompt Design.
Hierarchical prompts structure video-captioning instructions so multimodal and language models perform distinct roles with fewer omissions or confusions.
- Hierarchical prompts organize captioning instructions into Character, Skills, Constraints, and Structured Input components.These components define the model’s role, required capabilities, prohibited behaviors, and scenario-specific inputs.
- The differential-caption skills require describing inter-frame actions, environmental changes, object attributes, and camera movements.
- The design separates differential-caption and summary prompts for their respective stages of video captioning.
A.3 Prompt Template
The prompt templates assign specialized roles to analyze frame-to-frame changes and summarize consecutive-frame descriptions into a complete video caption.
- Differential Caption Prompt Template: The differential caption prompt casts the model as an expert video frame analyst producing clear, sequential descriptions.
- Differential Caption Prompt Template: The differential prompt focuses on changes in actions, behaviors, environments, object states and attributes, and camera movements between adjacent frames.
- Summary Prompt Template: The summary prompt takes descriptions of consecutive frames and generates a description for the entire video.
A.4 Comparison of Caption Quality
The paper compares caption sources and presents ShareGPT4Video’s data scale, caption-generation process, and qualitative emphasis on temporal correctness and detail.
- Caption Quality: The qualitative comparison contrasts concise captions that minimize errors with captions that provide insufficient information for information-dense videos.
- Caption Quality: DiffSW generates a full video caption with correct temporal understanding and comprehensive detail description.
- Caption Quality: GPT4V with multiple timestamped frames can misunderstand temporal order and produce an incorrect caption.
- Caption Quality: Concatenating all frames into a large image can omit video details.