PaperlayerView exact revision on arXiv

Source-linked AI summary

CI-VID: A Coherent Interleaved Text-Video Dataset

Yiming Ju, Jijin Hu, Zhengxiong Luo, Haoge Deng, hanyu Zhao, Li Du, Chengwei Wu, Donglin Hao, Xinlong Wang, Tengfei Pan

arXiv:2507.01938v1cs.CV

TL;DR

Isolated text–video datasets do not support learning coherent multi-clip sequences or generation conditioned jointly on visual and textual inputs. CI-VID provides interleaved video clips and captions for clips and their transitions, and its benchmark reports that fine-tuned models significantly outperform baselines in coherent, consistent sequence generation.

  • Problem

    Most public video datasets consist of isolated text–video pairs, which do not model inter-clip relationships or support joint visual-and-text conditioning for multi-scene generation.

    Source: §1 · p. 1

  • Method

    CI-VID provides coherent multi-clip video sequences with captions for individual clips and adjacent-clip transitions, alongside a multidimensional generation benchmark.

    Source: §1 · p. 2

  • Results

    Fine-tuned models significantly outperform baselines in producing coherent, story-driven video sequences with smooth transitions and consistent content.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CI-VID supports story-driven video generation with smooth visual transitions and strong temporal coherence.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Text-to-video (T2V) generation has recently attracted considerable attention, resulting in the development of numerous high-quality datasets that have propelled progress in this area. However, existing public datasets are primarily composed of isolated text-video (T-V) pairs and thus fail to support the modeling of coherent multi-clip video sequences. To address this limitation, we introduce CI-VID, a dataset that moves beyond isolated text-to-video (T2V) generation toward text-and-video-to-video (TV2V) generation, enabling models to produce coherent, multi-scene video sequences. CI-VID contains over 340,000 samples, each featuring a coherent sequence of video clips with text captions that capture both the individual content of each clip and the transitions between them, enabling visually and textually grounded generation. To further validate the effectiveness of CI-VID, we design a comprehensive, multi-dimensional benchmark incorporating human evaluation, VLM-based assessment, and similarity-based metrics. Experimental results demonstrate that models trained on CI-VID exhibit significant improvements in both accuracy and content consistency when generating video sequences. This facilitates the creation of story-driven content with smooth visual transitions and strong temporal coherence, underscoring the quality and practical utility of the CI-VID dataset We release the CI-VID dataset and the accompanying code for data construction and evaluation at: https://github.com/ymju-BAAI/CI-VID

1. Introduction

CI-VID addresses the limits of isolated text–video pairs by introducing interleaved text-and-video sequences for coherent multi-scene generation. Its benchmark reports that fine-tuning on CI-VID significantly outperforms baselines in sequence coherence and content consistency.

1. Introduction

  • Isolated text–video pairs lack inter-clip relationships, leaving models ill-suited to consistent cross-scene generation and joint conditioning on visual and textual inputs.Prior pair-trained models struggle with consistent characters, visual style, and smooth transitions across scenes; visual-only extrapolation can also be repetitive and lack semantic control.Source: §1 · p. 1
  • The dataset’s sequences pair consistent style and entities with visual variation, while its structured captions capture continuity and distinctions between adjacent clips.These annotations support generation guided jointly by video and text inputs.Source: §1 · p. 2
  • Fine-tuned models significantly outperform baselines in generating coherent, story-driven video sequences with smooth transitions and consistent content.The benchmark evaluates generation using human, VLM-based, and similarity-based measures.Source: §1 · p. 2
  • The paper argues for training beyond T-to-V mapping and introduces a multidimensional benchmark for coherent video sequence generation.The benchmark combines human evaluation, VLM-based assessment, and similarity-based metrics.Source: §1 · p. 2
  • CI-VID introduces a dataset of coherent multi-clip videos with captions for individual clips and the transitions between adjacent clips.Each sample interleaves video clips, individual captions, and joint captions describing continuity and change.Source: §1 · p. 2, Figure 1 · p. 2

2. Related Work

Related work covers the evolution of text-to-video datasets and open interleaved image-text datasets, noting that video datasets generally lack inter-clip relationships and interleaved video-generation data remain unexplored.

2.1. Text-to-Video Datasets

  • CI-VID’s construction pipeline segments source videos into clip sequences and creates captions for individual clips and adjacent clip pairs.Source: Figure 2 · p. 3
  • Text-to-video datasets improved caption quality, diversity, and video length, but remain largely limited to one-to-one text–video pairs without connections among clips.Examples include WebVid-10M, Panda-70M, InternVid, Vript, MiraData, and ShareGPT4Video.Source: §2.1 · p. 3

2.2. Interleaved Datasets

  • Open interleaved datasets such as MMC4, OBELICS, and CoMM address image-text sequences, but interleaved datasets remain unexplored for video generation.Flamingo and KOSMOS-1 are cited as evidence of benefits from interleaved image-text training, though their datasets are not public.Source: §2.2 · p. 3

3. CI-VID Dataset

CI-VID is built from complete videos into coherent clip sequences, then paired with individual and joint captions describing clip content and inter-clip relations. Its reported scale and sequence statistics cover both short and longer clip sequences.

3. CI-VID Dataset

  • CI-VID construction has three stages: source video collection, video clip sequence construction, and caption generation.The dataset uses complete source videos rather than pre-segmented clips.Source: §3 · pp. 3–4, §3.1 · p. 4

3.1. Source Video Collection and Processing

  • The team manually selected 4,068 YouTube channels and collected 592,429 raw videos; filtering removed over 80% of clips.Filters included shot segmentation, clip duration, motion strength, and on-screen text coverage.Source: §3.1 · p. 4

3.2. Video Clip Sequence Construction

  • Sequences are designed to balance related clips with visual variation, since simply extracting consecutive clips does not meet both requirements.The pipeline uses similarity-based and entity-based segmentation.Source: §3.2 · p. 4

3.2.1. Similarity-Based Segmentation

  • Similarity-based segmentation starts a new sequence below 0.6, ignores clips above 0.8, and retains clips between those thresholds.Adjacent clips must also be within three pre-filtering indices and ten seconds; single-clip sequences are discarded.Source: §3.2.1 · p. 4

3.2.2. Entity-Based Segmentation

  • Entity-based segmentation checks for a shared main entity across clips, removes clips that fail examination, and uses GPT-4o to revalidate sequences.Sequences are discarded if fewer than 70% of clips pass examination or if the main entities are not the same person.Source: §3.2.2 · p. 5

3.3. Caption Generation

  • Individual captions use sequentially sampled frames for detail, while joint captions use grid images to describe continuities and changes between adjacent clips.Each sequence receives one individual caption per clip and one joint caption per adjacent clip pair.Source: §3.3 · p. 5

3.4. Analysis of CI-VID

  • 341,550 samples average 3.1 clips each; over 98% of clips are at least 1080p, and individual and joint captions average over 200 words.The caption average is second only to MIRA.Source: §3.4 · p. 6
  • Most source videos contribute fewer than ten samples; although most samples contain 2–3 clips, over 30% contain four or more.CI-VID themes include food, entertainment, education, and sports.Source: §3.4 · p. 6, Figure 3 · p. 6

4. Experiment

Experiments evaluate whether CI-VID fine-tuning improves coherent multi-clip generation through human, VLM-based, and similarity-based assessments. The fine-tuned model outperforms the baseline across these evaluations, while qualitative comparisons show stronger cross-clip continuity.

4.1. Experiment Settings

  • The experiment uses a 0.6-billion-parameter sequential video model with temporal and spatial encoders and a decoder, initialized from NOVA-0.6B.Visual and text tokens are interleaved in the input sequence, while diffusion-loss supervision is applied only to visual tokens.Source: §4.1 · pp. 6–7

4.2. Experimental Results

  • The evaluation uses 1,000 prompts, each comprising six interconnected scenes that form a coherent narrative.An example prompt is shown in Table 3.Source: §4.2 · p. 7, Table 3 · p. 7

4.2.1. Qualitative Results

  • Fine-tuning improves stylistic and character consistency, prompt adherence, and camera transitions across clips compared with the base model.The comparison is illustrated in Figure 5.Source: Figure 5 · p. 7, §4.2.1 · p. 8

4.2.2. Quantitative Evaluation and Results

  • CI-VID fine-tuning significantly outperforms the base model on consistency, narrativity, and factual correctness in human evaluation.The comparison results are reported as win/tie/loss percentages.Source: §4.2.2 · p. 8, Table 4 · p. 8
  • The CI-VID model clearly outperforms the baseline on VLM dimensions 1–4, slightly on dimension 6, and performs similarly on dimension 5.The evaluation scores generated sequences and pairwise clips across six dimensions.Source: §4.2.2 · p. 8, Table 5 · p. 8
  • The model achieves superior performance on all three similarity metrics at both holistic and object-specific levels.The metrics are CLIP similarity, inverse LPIPS, and SSIM; higher is better.Source: Table 6 · p. 8, §4.2.2 · p. 9

5. Conclusion

The conclusion presents CI-VID as a dataset for coherent multi-scene generation and pairs it with a multidimensional benchmark. Experiments on that benchmark support the dataset’s effectiveness and utility.

5. Conclusion

  • CI-VID moves beyond isolated T2V toward text-and-video-to-video generation of coherent multi-scene sequences.The conclusion also describes a benchmark evaluating this task from human and automatic perspectives.Source: §5 · p. 9
  • The paper’s benchmark combines human and automatic evaluation of coherent video-sequence generation.Source: §5 · p. 9
  • Experimental results on the benchmark validate the effectiveness and utility of CI-VID.Source: §5 · p. 9

A. Experiment Details

The appendix describes diffusion training and inference for the experiment. Training uses noisy samples and a noise estimator; inference progressively denoises samples initialized with noise.

A. Experiment Details

  • The diffusion loss trains a noise estimator on noisy inputs conditioned on time and z_n.The estimator is implemented as a stack of MLP blocks.Source: A. Experiment Details · p. 10
  • The noisy input x_t^n is generated from the original sample x_n using a noise schedule indexed by time step t.Source: A. Experiment Details · p. 10
  • At inference, samples initialized with Gaussian noise are progressively denoised through sequential steps, with σ_t denoting the noise level at step t.Source: A. Experiment Details · p. 10

B. CI-VID Examples

The appendix presents CI-VID video-sequence examples. The figure captions describe each row as one clip from a sequence extracted from the same original video.

B. CI-VID Examples

Loading 2507.01938v1…