PaperlayerView exact revision on arXiv

Source-linked AI summary

GenVidBench: A 6-Million Benchmark for AI-Generated Video Detection

Zhenliang Ni, Qiangyu Yan, Mouxiao Huang, Tianning Yuan, Yehui Tang, Hailin Hu, Xinghao Chen, Yunhe Wang

arXiv:2501.11340v3cs.CV

TL;DR

Large-scale, high-quality datasets for AI-generated video detection are lacking as generated videos become harder to distinguish from real ones. GenVidBench provides a 6.78-million-video benchmark with cross-source and cross-generator design, semantic labels, and evaluations of video classification models. The experiments characterize detection as challenging, and the paper reports that GenVidBench is more challenging than existing datasets.

  • Problem

    Large-scale, high-quality datasets specifically for AI-generated video detection are lacking as generated videos become harder to distinguish from real ones.

    Source: Introduction · p. 1

  • Method

    GenVidBench combines cross-source and cross-generator video pairs with semantic labels and evaluations of video classification models.

    Source: Introduction · pp. 1–2, Overview of GenVidBench · p. 3

  • Results

    GenVidBench is reported as more challenging than existing datasets, with experiments underscoring the difficulty of cross-source and cross-generator detection.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The benchmark and its experimental results provide a foundation for developing and evaluating AI-generated video detectors.

    Source: Introduction · p. 2, Conclusion · p. 7

Abstract

from arXiv · show

The rapid advancement of video generation models has made it increasingly challenging to distinguish AI-generated videos from real ones. This issue underscores the urgent need for effective AI-generated video detectors to prevent the dissemination of false information via such videos. However, the development of high-performance AI-generated video detectors is currently impeded by the lack of large-scale, high-quality datasets specifically designed for generative video detection. To this end, we introduce GenVidBench, a challenging AI-generated video detection dataset with several key advantages: 1) Large-scale video collection: The dataset contains 6.78 million videos and is currently the largest dataset for AI-generated video detection. 2) Cross-Source and Cross-Generator: The cross-source generation reduces the interference of video content on the detection. The cross-generator ensures diversity in video attributes between the training and test sets, preventing them from being overly similar. 3) State-of-the-Art Video Generators: The dataset includes videos from 11 state-of-the-art AI video generators, ensuring that it covers the latest advancements in the field of video generation. These generators ensure that the datasets are not only large in scale but also diverse, aiding in the development of generalized and effective detection models. Additionally, we present extensive experimental results with advanced video classification models. With GenVidBench, researchers can efficiently develop and evaluate AI-generated video detection models.. Datasets and code are available at https://genvidbench.github.io.

Introduction

The introduction presents GenVidBench as a large, challenging benchmark for AI-generated-video detection, designed to reduce content and generator overlap between training and testing. It also provides semantic labels and experimental baselines.

Introduction

  • GenVidBench is introduced as a 6-million-scale AI-generated video detection dataset with cross-source and cross-generator design.The dataset pairs videos derived from the same prompts or images and assigns different generators to the training and test sets.Source: Introduction · pp. 1–2, Table 1 · p. 2
  • Paired videos use the same text prompt or image, while the training and test pairs use different generators to make detection challenging.The design aims to reduce content-based cues and prevent videos from the same generator sharing identical attributes across the splits.Source: Introduction · pp. 1–2
  • GenVidBench supplies semantic labels for a portion of videos across object category, action, and location dimensions.The paper reports that object category affects generated-video characteristics and that category differentiation supports selecting scenes of interest.Source: Figure 1 · p. 1, Introduction · p. 2
  • Experiments evaluate state-of-the-art video classification models and identify cross-source and cross-generator detection as a substantial challenge.The analysis also uses semantic labels to identify and filter particularly demanding categories.Source: Introduction · p. 2

Related Work

Prior datasets address image, deepfake-video, and AI-generated-video detection, but the paper identifies gaps in content diversity and benchmark scale or design. Earlier detectors use forensic traces or frame consistency, while video-classification models are also evaluated for this task.

AI-Generated Content Detection Dataset

  • Deepfake datasets such as DFDC, Celeb-DF, FaceForensics++, and DFD primarily contain facial data and lack content diversity.The passage states these datasets use GANs, VAEs, or other swapping techniques (Dolhansky et al. 2020; Li et al. 2020; R¨ossler et al. 2019; google 2019).Source: AI-Generated Content Detection Dataset · p. 2
  • GVD and GenVideo lack prompts or images, video pairs, semantic labels, and cross-source settings; GVF has those features but only 2.8k videos and no cross-source setting.GenVidBench is presented as a 6.78-million-video dataset with semantic labels and a cross-source, cross-generator train/test setting.Source: AI-Generated Content Detection Dataset · p. 2

Generated Video Detections

Dataset Construction

GenVidBench combines a 6.78-million-video collection with paired-source and paired-generator designs, semantic labels, and evaluations of detector performance across sources and generators. Its analyses show that detection is easier when training and testing conditions match, while source and generator shifts pose challenges.

Overview of GenVidBench

  • GenVidBench contains 6,784,490 videos, including two sets of paired videos generated from the same prompts or corresponding images.The first pair includes Pika, VideoCrafterV2, Modelscope, and Text2Video-Zero; the second includes SVD, MuseV, Mora, CogVideo, and HD-VG-130M.Source: Overview of GenVidBench · p. 3

Cross-Source and Cross-Generator Task

  • Same-generator train/test accuracy exceeds 97.40%, but VideoSwin-Tiny reaches 54.66% when trained on Pika and tested on SVD.The authors describe same-generator detection as relatively straightforward and new-generator testing as challenging.Source: Table 3 · p. 4, Cross-Source and Cross-Generator Task · p. 4
  • 68.41% for Pika and 85.51% for SVD on same-source tests fell to 60.95% and 60.15%, respectively, on cross-source tests.Across both video pairs, average same-source accuracy exceeded cross-source accuracy; the results use VideoSwin-Tiny.Source: Table 4 · p. 4, Cross-Source and Cross-Generator Task · p. 5

Content Analysis

  • The study categorizes prompts across object, action, and location dimensions, then uses object categories as the primary dimension for selecting and generating video pairs to ensure balance and semantic richness in the spatial dimension.LLMs extract prompt subjects and aggregate them into abstract categories, limited to no more than ten options. Figure 3 shows prompt-set distributions across multiple dimensions, while Figure 4 shows distributions across semantic categories.Source: Content Analysis · pp. 5–6

Scenario-Specific Task

  • Semantic labels let researchers extract scenario-specific datasets, such as People scenes for human-body generation or Presenting scenes for motion deformation analysis.The dataset can also support analyses of relationships between attributes, such as motion deformation, and detection accuracy.Source: Scenario-Specific Task · p. 6

Experimental Analysis

The experiments benchmark detectors on GenVidBench-6M and a smaller subset, compare performance across datasets and generators, and examine difficult cases, including one semantic scene category.

Implementation Details

  • Training uses eight 224×224 frames per video, batch size 8, MMAction2 default learning rates and settings, and random flipping and cropping.Source: Implementation Details · pp. 6–7

Results of Cross-Source and Cross-generator Task

  • DeMamba leads on GenVidBench-6M with Top-1 accuracy of 85.47%, ahead of MViTv2 at 80.45%; Transformer-based models outperform CNN-based models overall.Among CNNs, TSM has the best Top-1 accuracy at 73.88%, 6.57% lower than MViTv2; real videos are more readily distinguished, while Sora fakes are harder to classify than CogVideo fakes.Source: Results of Cross-Source and… · p. 7
  • On GenVidBench-143k, MViTv2 remains highly accurate, but overall accuracy is lower than on GenVidBench-6M because the smaller subset has less training data.Source: Results of Cross-Source and… · p. 7

Comparison with Advanced Datasets

  • SlowFast scores 82.55% accuracy on NeuralTextures versus 70.06% on GenVidBench; I3D scores 61.88% on GVF versus 60.21% on GenVidBench.The authors conclude that GenVidBench is more challenging than the compared datasets.Source: Comparison with Advanced Datasets · p. 7

Hard Case Analysis

  • Hard cases—generated videos assigned very low probability—vary by generator: SVD poses more difficult cases, while CogVideo produces almost no severely ambiguous cases.The analysis uses VideoSwin-tiny outcomes; Table 9 summarizes hard-case distributions across categories and generator subsets.Source: Hard Case Analysis · p. 7, Table 9 · p. 7

Results on Scenario-Specific Task

  • On Plants, TimeSformer leads with Top-1 accuracy of 75.09%, followed by UniFormerV2 at 64.76% and TPN at 64.24%, while VideoSwin reaches 52.86%.SlowFast reaches 55.30%; SVD has the lowest classification accuracy. The authors report that single-scene model performance differs from all-scene performance and that semantic labels help analyze scenarios.Source: Results on Scenario-Specific Task · p. 7, Table 10 · p. 7

Conclusion

GenVidBench is a six-million-video dataset for AI-generated video detection, with cross-source and cross-generator design and diverse semantic content labels. Experiments find it more challenging than existing datasets, offering opportunities to advance detectors.

  • GenVidBench comprises six million videos for AI-generated video detection, with cross-source and cross-generator design and diverse semantic content labels.Source: Conclusion · p. 7
  • Experiments demonstrate that GenVidBench is more challenging than existing datasets.Source: Conclusion · p. 7
  • The benchmark offers greater opportunities for advancing AI-generated video detectors.Source: Conclusion · p. 7
Loading 2501.11340v3…