PaperlayerView exact revision on arXiv

Source-linked AI summary

Your One-Stop Solution for AI-Generated Video Detection

Long Ma, Zihao Xue, Yan Wang, Zhiyuan Yan, Jin Xu, Xiaorui Jiang, Haiyang Yu, Yong Liao, Zhen Bi

arXiv:2601.11035v1cs.CVcs.AI

TL;DR

Existing AI-generated video datasets often lack model variety, scale, semantic and scene diversity, while benchmarks leave fundamental detection questions insufficiently explored. AIGVDBench comprises over 440,000 videos (more than 14 million images) from 20 open-source and 11 closed-source generative models, with 20,000 and 2,000 videos per model, respectively, simulated under real-world conditions. Detector performance varied substantially with the training generation model and was not directly positively correlated with generation-model quality; the evaluation examines four detector categories.

  • Problem

    Existing AI-generated video datasets often lack model variety, scale, semantic and scene diversity, while benchmarks leave fundamental detection questions insufficiently explored.

    Source: §1 · p. 2

  • Method

    AIGVDBench comprises over 440,000 videos (more than 14 million images) from 20 open-source and 11 closed-source generative models, with 20,000 and 2,000 videos per model, respectively, simulated under real-world conditions.

    Source: §1 · p. 2

  • Results

    Detector performance varied substantially with the training generation model and was not directly positively correlated with generation-model quality; the evaluation examines four detector categories.

    Source: §3.2 · p. 8, §4 · p. 9

  • Takeaways & Limitations

    AIGVDBench provides a large-scale benchmark resource and experimental evaluation intended to support research on AI-generated video detection.

    Source: §4 · p. 9

    The evaluation excludes some approaches, including reconstruction-based methods, because of storage and time limits; dataset construction may also introduce extraneous factors.

    Source: §8 · p. 50

Abstract

from arXiv · show

Recent advances in generative modeling can create remarkably realistic synthetic videos, making it increasingly difficult for humans to distinguish them from real ones and necessitating reliable detection methods. However, two key limitations hinder the development of this field. \textbf{From the dataset perspective}, existing datasets are often limited in scale and constructed using outdated or narrowly scoped generative models, making it difficult to capture the diversity and rapid evolution of modern generative techniques. Moreover, the dataset construction process frequently prioritizes quantity over quality, neglecting essential aspects such as semantic diversity, scenario coverage, and technological representativeness. \textbf{From the benchmark perspective}, current benchmarks largely remain at the stage of dataset creation, leaving many fundamental issues and in-depth analysis yet to be systematically explored. Addressing this gap, we propose AIGVDBench, a benchmark designed to be comprehensive and representative, covering \textbf{31} state-of-the-art generation models and over \textbf{440,000} videos. By executing more than \textbf{1,500} evaluations on \textbf{33} existing detectors belonging to four distinct categories. This work presents \textbf{8 in-depth analyses} from multiple perspectives and identifies \textbf{4 novel findings} that offer valuable insights for future research. We hope this work provides a solid foundation for advancing the field of AI-generated video detection. Our benchmark is open-sourced at https://github.com/LongMa-2025/AIGVDBench.

1. Introduction

AI-generated video detection has not kept pace with increasingly realistic video synthesis. AIGVDBench addresses limitations in dataset representativeness and systematic benchmark analysis.

1. Introduction

  • Realistic, accessible AI-generated videos have fueled “synthetic skepticism,” as viewers increasingly attribute suspicious videos to AI generation.Detection research remains significantly less developed in scale and depth for video than for earlier detection tasks.Source: §1 · p. 2
  • Existing datasets often use outdated or narrow model selections and prioritize video counts over semantic diversity, scene coverage, and balanced generation techniques.These shortcomings undermine the practical utility and generalizability of evaluation results.Source: §1 · p. 2
  • Current benchmarks largely stop at dataset creation, leaving fundamental questions about detection strategies, model detectability, training-sample quality, and detector obsolescence underexplored.The questions include whether whole-sequence or frame-by-frame analysis works better and whether higher-quality generated training samples improve robustness and generalization.Source: §1 · p. 2
  • AIGVDBench introduces a standardized construction pipeline using attribute balancing, comprehensive model selection, public-benchmark alignment, and strict quality control.The benchmark contains over 440,000 videos from 20 open-source and 11 closed-source models, and supports investigations into four research directions.Source: §1 · p. 2

2. AIGVDBench

AIGVDBench combines broad model and task coverage with attribute-balanced real-video prompts and a standardized dataset construction process. Its sections describe the benchmark scope, prompt selection, generation-model selection, and comparisons across detector categories.

2. AIGVDBench

  • The dataset covers 31 generation methods across text-to-video, image-to-video, video-to-video, and real-world scenarios, and exceeds existing alternatives in scale, content diversity, task coverage, and model variety.The 31 models comprise 20 open-source and 11 closed-source methods; Table 1 lists 23 T2V, 6 I2V, and 2 V2V models.Source: §2 · p. 2, Table 1 · p. 3

2.1. Selecting Prompts for Diversity.

  • The prompt-selection process classifies 400,000 OpenVid-HD prompts by content and attribute dimensions, then balances a 20,000-video real set across attributes.The algorithm partitions prompts by attribute combinations, prioritizes sparsely represented categories, and uses multi-attribute prompts when single-attribute prompts are insufficient.Source: §2.1 · pp. 4–5

2.2. Selection of Generation Models.

  • Generation-model selection spans 20 open-source and 11 closed-source systems, considering task support, representativeness, resource use, and realistic test sources.Closed-source models contribute 2,000 generated videos each; model sources include evaluation sets, demonstrations, community examples, and self-generated samples.Source: §2.2 · p. 5

2.3. Compression Format Unification.

  • The section includes AUC comparisons across video-classification, generated-image-detection, and generated-video-detection models, and ACC comparisons for vision-language models.The VLM comparison uses ACC because binary decisions preclude calculating AUC.Source: Table 4 · p. 6, Table 3 · p. 6

3. Evaluation and Analysis

The evaluation uses AIGVDBench to examine detection methods and generator generalization. It finds that generation quality alone does not determine detectability or detector training value, while all four detector paradigms remain open to improvement.

3.1. Benchmark Setup

  • The benchmark trains detectors on one generator and evaluates them across others, using Open-Sora as the training source and grouping methods into four categories.The categories are video classification, generated-image detection, generated-video detection, and multimodal large language models.Source: §3.1 · p. 6

3.2. Findings and Analyses

  • AIGVDBench evaluation identifies four detector paradigms for further investigation, with all four approaches still having room for improvement.The proposed paths include focusing on key real-versus-generated distinctions, controlling training-data volume, leveraging temporal artifacts, and developing interpretable algorithms.Source: §3.2 · pp. 6, 8
  • Classic video classifiers and adapted image detectors are promising, particularly when they discard irrelevant features and reduce data dependency.The paper notes limited video-classifier efficacy on closed-source models and points to Effort and ForgeLens as performing strongly across open- and closed-source models.Source: §3.2 · p. 7
  • Closed-source performance degradation in some image detectors indirectly supports DeCoF’s spatial-temporal artifact hypothesis.DeCoF and Demamba seek to inhibit spatial artifacts while leveraging temporal artifacts; the authors suggest prior data scale and quality may constrain their performance.Source: §3.2 · pp. 7–8
  • Vision-language models lag traditional detectors, while their label-only outputs complicate accuracy assessment; the authors still view them as promising for interpretability and accuracy.The paper’s summary favors interpretable VLM development over further enhancement of traditional video classifiers as a long-term direction.Source: §3.2 · p. 8
  • Generation quality does not consistently predict detectability or detector generalization, and the optimal training generator varies by detector.Wan 2.1 ranked highest overall, yet higher generation quality did not consistently make videos harder to detect; the paper cautions that training-data quality depends on detector characteristics.Source: §3.2 · p. 8

4. Conclusion

The paper presents AIGVDBench as a large-scale resource built with a standardized construction pipeline. Its experiments compare four categories of detection methods and examine previously unexplored research questions.

4. Conclusion

  • AIGVDBench uses attribute balancing and model selection to build a representative dataset of 31 generative models and over 440,000 videos.The conclusion describes the resource as a high-quality benchmark and says the pipeline lays a foundation for research in AI-generated video detection.Source: §4 · p. 9
  • The authors present AIGVDBench as a foundation for advancing AI-generated video detection research.They describe attribute balancing and model selection as supporting dataset diversity and representativeness.Source: §4 · p. 9
  • The benchmark’s comprehensive evaluation examines four categories of detection methods and reveals previously unexplored research questions.The authors hope the work will promote generated-content detection technologies and trustworthy digital media authentication.Source: §4 · p. 9

5. Content Structure in Appendix

The appendix provides further analyses, benchmark and method details, and a section on limitations.

5. Content Structure in Appendix

  • Section 6 covers evaluation metrics and further findings and analyses.It includes additional analyses related to Findings 1 and 2.Source: §5 · p. 14
  • Section 7 documents AIGVDBench, including generation and detection methods, source data, prompt categorization, balance comparisons, encoding, training, and construction timeline.Source: §5 · p. 14
  • Section 8 is devoted to limitations.Source: §5 · p. 14

6. Further Findings and Analysis

The analyses show that detector performance varies with generation task and model source, while examining detector features and training strategies alongside VBench quality dimensions.

6.1. Definition of Evaluation Metrics.

  • VBench evaluates six core dimensions; background and subject consistency assess the temporal coherence of environments and main subjects.Subject consistency concerns identity and visual attributes across frames.Source: §6.1 · p. 14
  • VBench's other dimensions cover frame aesthetics, motion intensity and smoothness, and technical image artifacts.Imaging quality focuses on signal-level distortions, while motion smoothness concerns continuity and naturalness.Source: §6.1 · p. 14

6.2. Further Findings and Analyses of Findings-1

  • Closed-source generation models generally achieve significantly higher overall video quality than open-source models across VBench dimensions.The paper relates this disparity to differences in transparency, scale, and commercial orientation.Source: §6.2 · p. 14, Table 7 · p. 15, Figure 4 · p. 16
  • Open-source videos form a semantically homogeneous, single-domain benchmark, whereas heterogeneous closed-source videos create a more difficult cross-domain detection scenario.The passage attributes the distinction to the open-source data's tight clusters and the varied tasks represented in closed-source data.Source: §6.2 · p. 16
  • TimeSformer, UniformerV2, and Effort perform significantly better on T2V than I2V or V2V, aligning with their stable results on predominantly T2V closed-source videos.The passage identifies strong T2V performance as the main reason for their anomalous stability on closed-source data.Source: Table 8 · p. 16, §6.2 · p. 17
  • CLIP-ViT-L/14 has feature-discrimination capability comparable to trained ResNet but is slightly inferior on videos generated by closed-source models; Effort fine-tuning improves base-model detection by 1.8% on open-source and 28.79% on closed-source models.Features of close-source-generated videos show greater separability from real-video features.Source: §6.2 · p. 18
  • ForgeLens1's small-dataset, more-epoch strategy outperforms ForgeLens3's large-dataset, fewer-epoch strategy, gaining 2.22% on open-source and 8.16% on closed-source models.The reported gains compare the two ForgeLens training strategies on the respective model-source sets.Source: §6.2 · p. 18
  • V2V is the most challenging generation task to detect across models, with real-video references producing generally higher-quality generated content.The lower Dynamic Degree in V2V outputs is attributed primarily to the high frame rate of the reference video.Source: Table 9 · p. 18, §6.2 · p. 18

2. Image-level detectors show a clear task-dependent performance trend.

This section examines how AI-generated video detection performance varies with task, output quality, spatial content, and similarity between test and training generative patterns. It also considers current VLM reliability and the role of balanced content representation in detection datasets.

2. Image-level detectors show a clear task-dependent performance trend.

  • Image-level detectors perform best on I2V, next on T2V, and worst on V2V, inversely matching the tasks’ overall image-quality ranking.The relationship may reflect combined quality metrics rather than any single indicator.Source: §2 · pp. 19–20
  • Video-level detector behavior varies across models, but for the same generator most detectors find V2V hardest, then I2V, then T2V.This ordering aligns with typical generation-quality differences; imaging quality and dynamic extent may help explain why I2V can be harder to detect than T2V.Source: §2 · p. 20
  • Current VLMs remain inadequate for reliable AI-generated video detection, even under alternative frame-labeling protocols, and larger models within an architecture family do not significantly improve performance.Many VLMs return substantial invalid “No Answer” outputs, which the evaluation assigns 50% accuracy.Source: §2 · pp. 20–21
  • DeepSeek-VL2’s strong closed-source results do not establish reliable detection: its judgments shifted across prompt tasks and its open-ended explanations were inconsistent.The authors hypothesize possible recognition bias from training exposure to closed-source videos, while noting that the reason for the strong results remains unclear.Source: §2 · pp. 21–22

6.3. Further Findings and Analyses of Findings-2

  • Across generative models, overall quality does not consistently predict detection difficulty; detector performance instead correlates with specific quality dimensions, with associations not establishing causation.Correlations include imaging and frame-level quality across detector types, dynamic degree for video classifiers, aesthetic quality for image and video detectors, and background and subject consistency for some I2V detectors.Source: §6.3 · pp. 22–24
  • As guidance scale increases, most detectors improve steadily, but AnimateDiff-trained detectors degrade; motion smoothness also worsens throughout the tested range.The authors suggest stronger prompt adherence and greater content uniformity may make synthetic videos easier to distinguish, but the reasons for the divergent detector trends remain unclear.Source: Table 18 · p. 23, §6.3 · p. 25
  • With EasyAnimate, generated-video quality first improves and then deteriorates as sampling steps rise, while detector trends vary with training-data attributes and architecture.Detectors trained on stronger attributes tend to improve up to a point before declining, whereas those trained on lower-quality data often decline and then improve; frame-based detectors are more sensitive to image quality, and I3D rises then falls. Results on the 400-sample dataset also differ from main-body results, underscoring benchmark-scale effects.Source: Table 19 · p. 24, §6.3 · p. 25
  • Detection performance depends more on similarity between test-time generative patterns and detector training data than on generator quality; better generators do not necessarily yield better detectors or harder detection.Cross-evaluations also find that higher evaluation metrics do not guarantee improved detection, and some detectors trained on models such as Wan 2.1 degrade in performance.Source: §6.3 · pp. 24–25

Analysis-2.5: The impact of video type on detection difficulty

  • The evaluation used four spatial content categories from the AIGVDBench test set, with 400 videos per category: 200 real and 200 generated.Categories were restricted to one specified content type, though videos could still contain other spatial elements because of content complexity.Source: §6.3 · p. 37

7. Details of AIGVDBench

This section surveys video-generation models used in AIGVDBench and the constraints on selecting them. It traces a shift from short clips toward longer, more realistic video generation.

7.1. Brief Introduction of Generation Models

  • Video generation has shifted from short, roughly 2–3-second clips focused on basic interframe consistency toward long sequences emphasizing physical laws, commonsense reasoning, and creative composition.The section describes early work as focused on simple prompts and later multimodal work as emphasizing surface-level realism.Source: §7.1 · p. 37
  • Models were selected to fit either 80GB of GPU memory with generation under two minutes per video, or 24GB with generation under eight minutes; dataset construction took nearly four months.The model-selection criteria reflect the available compute resources described by the authors.Source: §7.1 · p. 38
  • The proprietary-model overview includes CausVid, a causal-transformer system designed for fast streaming generation, and Gen3 and Kling, which add features for longer or controllable video generation.CausVid is described as using knowledge distillation and supporting real-time instruction modification; Gen3 and Kling are described as supporting motion or camera controls.Source: §7.1 · p. 38
  • Open-source examples include Open-Sora, which aims to replicate and advance Sora at lower training cost, and EasyAnimate, which uses DiT modifications and a motion module for long-video generation.EasyAnimate’s Slice VAE is described as addressing memory demands for encoding and decoding long videos.Source: §7.1 · p. 38

7.2. Introduction of the Used Detection Methods

AIGVDBench evaluates 33 detectors across four categories, spanning video classifiers, image detectors adapted to video, dedicated video detectors, and vision-language models. The selection excludes face-specific deepfake detectors and methods requiring extensive preprocessing.

7.2. Introduction of the Used Detection Methods

  • Image-detection approaches include artifact-learning methods and frozen pretrained encoders with lightweight heads, while related work also explores augmentation and generalization strategies.Artifact-learning methods target spatial or frequency traces; frozen encoders can support cross-model generalization but may retain forgery-irrelevant information.Source: §7.2 · pp. 40–41
  • The benchmark groups 33 detectors into four categories and excludes face-specific deepfake detectors and methods requiring extensive preprocessing.The categories are video classifiers, image detectors adapted for frame-level video detection, dedicated AI-generated video detectors, and transferred vision-language models.Source: §7.2 · p. 41
  • Vision-language models are evaluated alongside specialized video and image detectors as a distinct detection direction.The supplied models include DeepSeek-VL, InternVL, Qwen-VL, LLaVA, FastVLM, Kimi-VL, and Emu3 variants.Source: §7.2 · pp. 41, 43
  • Video-classification baselines include architectures such as MViT, UniFormer, VideoSwin, VideoMAE, TSM, SlowFast, TimeSformer, I3D, UniFormerV2, and X3D.The supplied descriptions cover transformer, convolutional, self-supervised, and temporal-shift approaches to video representation and classification.Source: §7.2 · p. 41
  • Dedicated video detectors include DeMamba, which targets large-scale detection, and DeCoF, which analyzes consistency across frames.DeCoF uses temporal and sequential features to detect content that may not be apparent in individual frames.Source: §7.2 · pp. 42–43

7.3. Introduction of Original Data

AIGVDBench draws its real videos from OpenVidHD, a high-resolution collection with broad thematic coverage. The benchmark standardizes the real-video compression to H.264.

7.3. Introduction of Original Data

  • Real videos come from OpenVidHD, a curated collection of approximately 433,000 high-resolution 1080p clips totaling about 11,000 hours.OpenVidHD is derived from OpenVid-1M and includes textual descriptions for its clips.Source: §7.3 · pp. 43–44
  • OpenVidHD covers diverse themes, including natural landscapes, human activities, and animal behaviors.The source describes stringent quality control and diverse thematic coverage.Source: §7.3 · p. 44
  • All videos in AIGVDBench are standardized to H.264 compression.The source videos use MPEG-4 Part 2; the benchmark applies H.264 for evaluation consistency.Source: §7.3 · p. 44

7.4. Details of Prompts Categorization

The prompt-categorization process uses a WordNet-based attribute taxonomy, while the balance analysis measures both category coverage and consistency across attributes. The authors report that their selection algorithm yields the highest balance score among compared datasets.

7.4. Details of Prompts Categorization

  • Prompts are classified by scanning for predefined WordNet synsets and keywords, assigning matching category labels to support multi-label categorization.The paper presents the synset and keyword taxonomies in Tables 22 and 23.Source: §7.4 · p. 44, Table 22 · p. 45, Table 23 · p. 46
  • Long prompts caused some misclassification, and using LLMs—including GPT-4—did not fully resolve the issue across more than 400,000 entries.The authors state that prompt length and dataset volume made LLM-based classification infeasible at scale.Source: §7.4 · p. 44

7.5. Balance Comparison with Prior Datasets

  • Relative Uniformity measures distribution uniformity within an attribute, while Complete Uniformity adds a penalty when categories are missing.The missing-category penalty depends on an adjustable intensity parameter.Source: §7.5 · p. 44
  • Mean Complete Uniformity averages balance across attributes, and Uniformity Coordination captures how consistently balanced those attributes are.The two metrics are combined in the Penalized Global Balance Score (PGBS), bounded between 0 and 1.Source: §7.5 · p. 44
  • The dataset achieves the highest PGBS among the compared datasets despite using more extensive prompts.The authors attribute the result to a systematic, scalable selection algorithm that avoids manual selection and its inherent biases.Source: §7.5 · p. 44

7.4. Details of Prompts Categorization

  • The taxonomy covers spatial categories under major content and temporal categories under attribute control.The supplied captions identify these as the spatial major-content and temporal attribute-control views.Source: §7.5 · pp. 45–46, Table 22 · p. 45, Table 23 · p. 46

7.5. Balance Comparison with Prior Datasets

  • The compression comparison distinguishes mixed encoding—real MPEG-4 Part 2 and generated H.264—from unified H.264 encoding.These are labeled Mixed and Unified, respectively.Source: §7.5 · p. 46

7.6. Discussion on Video Encoding Methods

The authors find little accuracy impact from mixed video-compression formats and use unified H.264 to reduce potential confounding. Their training protocols retain model-specific configurations, tailor VLM prompts, and follow a development timeline from February to October 2025.

7.6. Discussion on Video Encoding Methods

  • Compression-format discrepancies had negligible accuracy impact; only X3D trained on mixed-compression data showed a noticeable effect, so the benchmark uses H.264 throughout.Some output logits varied, but these changes had almost no influence on final accuracy.Source: §7.6 · p. 46, Figure 24 · p. 48

7.7. Training details

  • Non-VLM models retain model-specific hyperparameters and training configurations, with a 50-epoch cap and prescribed shorter schedules honored.Most models use 256×256 inputs, while ViT-based models use 224×224; the data input pipeline is the only change to released training configurations.Source: §7.7 · pp. 46–48
  • VLMs are evaluated zero-shot with prompts adjusted for model-specific instruction-following, and frame-based aggregation labels a video REAL if any frame is REAL.A video is labeled FAKE only when all frames are classified FAKE; invalid outputs are categorized as No Answer.Source: §7.7 · pp. 48, 50, Table 27 · p. 48, Figure 25 · p. 49

7.8. Timeline of AIGVDBench Construction

8. Limitation

AIGVDBench’s data coverage and evaluations are constrained by computational resources, generation speed, cost, and access, and its analyses may be affected by dataset-construction factors.

8. Limitation

  • Resource and generation-speed constraints required smaller-parameter versions of some models.Regional access restrictions also excluded closed-source models including Sora 2 and Veo 3.Source: §8 · p. 50
  • Because generating closed-source samples was costly, the authors collected existing videos rather than regenerating the entire test set.Source: §8 · p. 50
  • Storage and time constraints excluded some detector approaches, including reconstruction-based methods, and dataset-construction factors may affect the correctness of analytical findings.The authors cannot guarantee absolute correctness despite efforts to minimize extraneous factors.Source: §8 · p. 50
Loading 2601.11035v1…