PaperlayerView exact revision on arXiv

Source-linked AI summary

ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, Jiaqi Wang

arXiv:2406.04325v1cs.CV

TL;DR

Video datasets often pair rich content with brief captions, while detailed captions have received less attention for video understanding and generation. ShareGPT4Video uses a differential captioning strategy to develop caption datasets and models; experiments report gains across understanding and generation, with ShareGPT4Video-8B achieving the best performance on three benchmarks.

  • Problem

    Existing video data often pair rich content with brief captions, while detailed captions have received less attention for video understanding and generation.

    Source: §1 · p. 2

  • Method

    The authors use a differential captioning strategy to build ShareGPT4Video and develop ShareCaptioner-Video, which generates captions for videos across varied resolutions, aspect ratios, and durations.

    Source: §1 · p. 3

  • Results

    Across video-generation and video-understanding experiments, the authors report strong generation and consistent LVLM gains, with ShareGPT4Video-8B achieving the best performance on all three benchmarks and 61.5% average accuracy on TempCompass.

    Source: §1 · p. 3, §4.1 · p. 7

  • Takeaways & Limitations

    The authors conclude that their dataset and captioner enhance video understanding and generation tasks.

    Source: §6 · p. 9

    The captioning pipeline cannot incorporate audio alongside visual and textual information, although audio is beneficial in conversational scenarios involving daily activities.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

We present the ShareGPT4Video series, aiming to facilitate the video understanding of large video-language models (LVLMs) and the video generation of text-to-video models (T2VMs) via dense and precise captions. The series comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with various lengths and sources, developed through carefully designed data filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that reached SOTA performance on three advancing video benchmarks. To achieve this, taking aside the non-scalable costly human annotators, we find using GPT4V to caption video with a naive multi-frame or frame-concatenation input strategy leads to less detailed and sometimes temporal-confused results. We argue the challenge of designing a high-quality video captioning strategy lies in three aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame detailed content description. 3) Frame-number scalability for arbitrary-length videos. To this end, we meticulously designed a differential video captioning strategy, which is stable, scalable, and efficient for generating captions for videos with arbitrary resolution, aspect ratios, and length. Based on it, we construct ShareGPT4Video, which contains 40K high-quality videos spanning a wide range of categories, and the resulting captions encompass rich world knowledge, object attributes, camera movements, and crucially, detailed and precise temporal descriptions of events. Based on ShareGPT4Video, we further develop ShareCaptioner-Video, a superior captioner capable of efficiently generating high-quality captions for arbitrary videos...

1 Introduction

The paper addresses the shortage of detailed video captions by introducing a scalable captioning strategy, a 40K-video dataset, and a captioner used to annotate 4.8M videos. Experiments demonstrate their value for video understanding and generation.

Motivation

  • Video captioning is difficult to scale, and naive GPT4V multi-frame inputs can confuse temporal relations or lose detail as frame counts increase.The paper identifies three requirements: precise inter-frame temporal understanding, detailed intra-frame descriptions, and scalability to arbitrary video lengths.Source: §1 · pp. 2–3

Differential Sliding-Window Captioning

  • DiffSW captions the first frame, then uses a two-frame sliding window for chronological change descriptions before GPT4 synthesizes the video caption.GPT4V receives the previous frame, its differential caption, and the current frame; the constant frame count supports scalability, while differential captions can be reused to caption subclips.Source: §1 · p. 3

ShareGPT4Video

  • ShareGPT4Video contains 40K high-quality video-caption pairs, with rich world knowledge, object attributes, camera movements, and precise temporal descriptions.The pipeline combines semantic-based data filtering, semantic-aware key-frame extraction, DiffSW, hierarchical prompts, and manual quality inspection.Source: §1 · p. 3

ShareCaptioner-Video

  • ShareCaptioner-Video annotates 4.8M aesthetically appealing videos, totaling about 3,000 hours, with high-quality captions at minor cost.It is designed for videos with a wide range of resolutions, aspect ratios, and durations.Source: §1 · p. 3

Experimental results

  • ShareGPT4Video yields consistent LVLM benchmark gains with little training-data replacement; its DiT model generates controlled 10-second high-resolution videos, and ShareGPT4Video-8B achieves SOTA on three benchmarks.The paper reports gains across multiple benchmarks for several LVLMs and SOTA performance on three comprehensive video benchmarks.Source: §1 · p. 3

2 ShareGPT4Video Dataset

ShareGPT4Video is constructed through diverse video-source selection, semantic filtering, semantic-aware key-frame extraction, and differential captioning designed to produce detailed temporal descriptions. The pipeline balances content diversity, semantic coherence, and caption quality.

2.1 Data Collection

  • The collection combines complex, diverse Panda-70M [12] clips with aesthetically appealing videos from user-uploaded websites, supplemented by Ego4D [18] and BDD100K for ego-centric and driving scenes.The source mix serves video understanding and generation while spanning varied real-world content, including wildlife, cooking, sports, scenery, and human activities.Source: §2.1 · p. 4
  • Videos longer than two minutes are excluded, and semantic filtering admits a candidate only when its caption embedding is sufficiently dissimilar to those already selected, mitigating content homogeneity.Panda-Student [12] generates one-sentence candidate captions, which are encoded with Bert-Base-Uncased [15] and compared using CLS-token similarity against the selected pool.Source: §2.1 · p. 4

2.2 Video Processing

  • Semantic-aware key-frame extraction removes semantically similar adjacent frames while retaining frames that capture event evolution, addressing traditional methods’ risk of missing crucial changes and transitions.Frames are sampled at two-second intervals; CLIP-Large similarities determine whether to add a frame, while the first and last frames are always included.Source: §2.2 · pp. 4–5
  • The dataset spans videos from 2 seconds to 2 minutes, with captions primarily ranging from 200 to 400 words and providing rich temporal information.Its content includes wildlife, cooking, sports, scenery, ego-centric activities, and auto-driving scenarios.Source: Figure 3 · p. 5

2.3 Captioning Pipeline

  • Differential sliding-window captioning uses current and previous key-frames to generate temporally detailed captions, then summarizes timestamped captions with GPT-4-Turbo-04-09.A Differential Prompt focuses GPT-4V on changes such as posture, position, and camera angle; previous differential captions add temporal context and reduce hallucinations. A Hierarchical Prompt Design stabilizes output format and quality.Source: Figure 2 · p. 4, §2.3 · p. 5

3 ShareCaptioner-Video

ShareCaptioner-Video fine-tunes IXC2-4KHD on video-caption data and supports four captioning modes. Its sliding-captioning mode is used to caption 4.8 million aesthetic video clips.

3.1 Model design

  • Fast captioning concatenates a video's keyframes into a vertically elongated image and trains the model to caption that image for rapid generation on short videos.The model uses an image-grid format for direct video captioning.Source: §3.1 · p. 6, Figure 4 · p. 6
  • Sliding captioning takes two adjacent keyframes and the previous differential caption as input, describing events between them for streaming captions on long videos.This uses a differential sliding-window format similar to the captioning pipeline in Section 2.3.Source: §3.1 · p. 6, Figure 4 · p. 6
  • Clip summarizing uses differential descriptions as input and outputs a video caption without re-processing frames.It supports clips from ShareGPT4Video or videos already processed with differential sliding-window captioning.Source: §3.1 · p. 6, Figure 4 · p. 6
  • Prompt re-captioning trains on GPT-4-generated Sora-style prompts as input and dense captions as targets, rephrasing prompts to align T2VM inference with training formats.This supports users who prefer specific video-generation areas.Source: §3.1 · p. 6, Figure 4 · p. 6

3.2 Scaling-up Captions

  • The sliding-captioning mode generated captions for 4.8 million clips totaling approximately 3,000 hours from MixKit, Pexels, and Pixabay.The clips were selected for aesthetic appeal; captioning required approximately 4,000 H100 GPU hours.Source: §3.2 · p. 6

4 Experiments

ShareGPT4Video-8B achieves the strongest reported results across three video benchmarks, while ShareCaptioner-Video performs on par with GPT4V and detailed captions improve text-to-video generation quality. Experiments also show that caption quality and an unlocked vision encoder affect LVLM modality alignment.

4.1 Video Understanding

  • Average benchmark performance gains are 1.1 for VideoLLaVA-7B, 2.0 for LLaMA-VID-7B, and 2.3 for LLaMA-VID-13B after integrating high-quality captions.For ShareGPT4Video-8B, the study combines 153K VQA examples with 28K caption examples and samples 16 frames into a 4x4 grid.Source: §4.1 · p. 7
  • Short captions may add little or degrade some benchmark results, whereas detailed captions improve temporal-sequence understanding and unlocking the vision encoder further improves modality alignment.The comparison uses VQA data with short or detailed captions, and reports gains for detailed captions when comparing the first, second, and fourth rows of Table 2.Source: §4.1 · p. 7, Table 2 · p. 7
  • 61.5% average accuracy on TempCompass puts ShareGPT4Video-8B 11.6% above VideoLLaVA-7B; it also surpasses prior SOTA by 2.7% on VideoBench and 8.2% on MVBench.The paper reports the TempCompass comparison despite competitors using larger training data or more parameters; ShareGPT4Video-8B is best across all three benchmarks.Source: §4.1 · p. 7, Table 3 · p. 8, Table 4 · p. 8

4.2 Video Captioning

  • Human preference voting finds ShareCaptioner-Video performs on par with GPT4V.The paper also presents qualitative comparisons in Figure 9.Source: §4.2 · pp. 7–8

4.3 Video Generation

  • High-quality detailed captions help the T2VM follow complex prompts and control semantic content and camera movement, while brief captions lead to subpar results.The setup generates captions for 4.5M videos with 65 frames and 0.3M videos with 221 frames, and compares against a baseline trained on the same quantity of video-short-caption pairs.Source: §4.3 · p. 8

5 Limitations and Social Impacts

The captioning pipeline cannot incorporate audio, and captions were not individually checked for social bias. Selected videos may contain human faces, so users must follow the original video-source licenses.

Limitations

  • The captioning pipeline cannot incorporate audio, which can benefit conversational scenarios involving daily human activities.The authors plan to add audio information when GPT4o supports audio input, to further improve caption quality.Source: §5 · pp. 8–9

Social impacts

  • Captions were not manually checked for social bias, selected videos may contain human faces, and users must follow the original video-source licenses.The captions themselves have no use restrictions, but the original video licenses still apply to the videos.Source: §5 · p. 9

6 Conclusion

ShareGPT4Video addresses the shortage of high-quality video-caption data with a dataset and captioning model for LVLM and T2VM research.

6 Conclusion

  • The study develops ShareGPT4Video and ShareCaptioner-Video to address the lack of high-quality video-caption data for LVLMs and T2VMs.The authors report that experiments validate their effectiveness in enhancing video understanding and generation tasks.Source: §6 · p. 9
  • The work produces 40K GPT4V-generated detailed captions and 4.8M high-quality captions from ShareCaptioner-Video.The captions include world knowledge, object attributes, camera movements, and detailed temporal descriptions of events.Source: §6 · p. 9
  • The authors present ShareGPT4Video and ShareCaptioner-Video as resources for advancing research in the LVLM and T2VM communities.Source: §6 · p. 9

A Appendix

The appendix provides additional results and analysis, covering experimental setups, prompt design, caption-quality comparisons, dataset statistics, filtering and key-frame extraction, and related work.

A.4 Caption-quality comparison

  • The appendix compares ShareCaptioner-Video with other methods on qualitative and quantitative caption quality.Source: A Appendix · p. 10

A.1 Experimental setups

A.2 Prompt design

  • It explains the prompt design philosophy and showcases prompts for each stage.Source: A Appendix · p. 10

A.3 Hierarchical prompt template

A.5 Dataset statistics

A.6 Filtering and key-frame extraction

  • It provides detailed pseudocode for semantic-based data filtering and semantic-aware key-frame extraction.Source: A Appendix · p. 10

A.7 Related work

A.1 Experimental Details

The section specifies training configurations and compute requirements for ShareGPT4Video-8B and the Latte-XL-based text-to-video model.

A.1 Experimental Details

  • ShareGPT4Video-8B fine-tunes the full model with batch size 128 and AdamW, using learning rates of 2e-6 for the vision encoder, 2e-5 for the MLP projector, and 2e-4 for the LLM using LoRA [21].Training takes about 5 hours on 8 A100 GPUs.Source: §A.1 · p. 10
  • Latte-XL [41] with a T5-XXL [51] text encoder is trained in two stages at 512×512 resolution using AdamW and a constant 2e-5 learning rate.Stage one uses fewer frames, joint image-video training, and 50k steps; stage two reduces both batch sizes to 2. Compute requirements are around 6K H100 GPU hours for stage one and 36K Ascend GPU hours for stage two; Bf16 is used because fp16 caused NaN loss.Source: §A.1 · p. 10

A.2 Hierarchical Prompt Design.

Hierarchical prompts help multimodal and language models perform their captioning roles through four components: Character, Skills, Constraints, and Structured Input. For differential captioning, they specify the model’s role, temporal-analysis skills, output rules, and frame-based inputs.

A.2 Hierarchical Prompt Design

  • The prompt design has four components: Character defines the model’s role and work environment, Skills specifies required abilities, Constraints sets undesired behaviors and output rules, and Structured Input adapts inputs to the scenario.The design aims to help multimodal and language models perform their roles effectively during captioning.Source: §A.2 · p. 10
  • For differential captioning, Skills directs the model to describe inter-frame actions, environmental and background changes, target appearance changes, and camera movements.Source: §A.2 · p. 10
  • For differential captions, Constraints require precise descriptions without itemized lists, while Structured Input includes frame indexes, timestamps, the previous frame, and its differential caption.Source: §A.2 · p. 10

A.3 Prompt Template

The section presents separate prompt templates for differential captioning and summary captioning.

A.3 Prompt Template

A.4 Comparison of Caption Quality … A.7 Related Work

The appendix reports that ShareCaptioner-Video produces captions comparable to GPT4V, then documents the caption dataset, filtering and key-frame procedures, and related work on video-language and text-to-video models. It frames high-quality video-caption data as an underexplored factor in aligning modalities and advancing text-to-video generation.

A.4 Comparison of Caption Quality

  • ShareCaptioner-Video’s captions have comparable information levels to GPT4V’s and perform on par in a volunteer evaluation of omission and fabrication, distortion, and temporal mismatch.The evaluation used 100 caption pairs and 10 volunteers, with up to 3 points per pair.Source: §A.4 · p. 12, Table 7 · p. 12, Table 6 · p. 12
  • Panda-Student produces short captions that can omit information-dense video content, while Video ChatCaptioner’s longer captions can hallucinate; ShareCaptioner-Video is described as comparable to GPT-4V.The qualitative comparison is illustrated with captions of a video showing the Eiffel Tower from an interior perspective.Source: §A.4 · pp. 12–13, Figure 9 · p. 13

A.5 Data Statistics

  • Table 8 summarizes 4.8M high-quality video-caption pairs generated by ShareCaptioner-Video.The appendix also includes a word cloud of captions in the Share4Video dataset.Source: Table 8 · p. 16, Figure 13 · p. 16

A.6 Pseudo Code

  • The appendix provides pseudo code for semantic-based data filtering and semantic-aware key-frame extraction.The supplied captions identify these procedures but do not describe their steps.Source: Figure 14 · p. 17, Figure 15 · p. 17

A.7 Related Work

  • Prior large video-language models encode video into vision tokens for LLMs, but the cited approaches have not investigated high-quality captions’ role in video-language alignment.The discussion describes VideoChat2’s video transformer and Q-Former, VideoLLaVA’s pre-aligned encoder and MLP, and LLaMA-VID’s per-frame token compression.Source: §A.7 · p. 18
  • Although high-quality video-caption data is pivotal to text-to-video models, it has received insufficient attention, motivating this work’s high-quality video-caption dataset.The discussion cites Sora as an example of a text-to-video model whose success relies in part on extensive high-quality video-caption data.Source: §A.7 · p. 18

Related papers

Loading 2406.04325v1…