PaperlayerView exact revision on arXiv

Source-linked AI summary

CCDF: A Benchmark Dataset for Deepfake Detection in Real-World Surveillance Footage

Baptiste Chopin, Thomas Swearingen, Arun Ross, Antitza Dantcheva, Christian Rathgeb

arXiv:2610.07939v1cs.CV

TL;DR

Existing deepfake video datasets emphasize benign content and older or open-source generators, limiting evaluation against fabricated malicious surveillance footage. CCDF assembles a dataset from frontier commercial systems with raw, cleaned, and altered versions; ten recent detectors do not reliably distinguish its generated videos from real ones.

  • Problem

    Existing deepfake video datasets emphasize benign content and older or open-source generators rather than plausible malicious surveillance footage.

    Source: §1 · p. 2

  • Method

    CCDF grounds generated surveillance videos in human-verified descriptions of real footage, uses three commercial generators, and provides raw, cleaned, and altered dataset versions.

    Source: §1 · pp. 2–3

  • Results

    Ten recent state-of-the-art detectors do not reliably distinguish CCDF-generated videos from real ones, despite strong reported performance on existing benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The benchmark results indicate that existing datasets are not well suited to evaluating some threats posed by generated surveillance footage.

    Source: §1 · p. 3, §6 · p. 11

    The authors did not test whether each generated video fools viewers, and CCDF remains small and limited to three generators.

    Source: §5 · p. 10

Abstract

from arXiv · show

Due to rapid advances in Generative AI, commercial video generation tools can be used to produce fabricated surveillance footage that can fool both human viewers and automated synthetic video detectors. Since these tools are so widely accessible, a malicious user can create a harmful video clip at minimal cost. The production and dissemination of such videos in high-stakes settings, such as crime reporting and elections, can misdirect emergency response efforts or distort political discourse. Existing deepfake video datasets, used by the research community to develop deepfake detection algorithms, exhibit two limitations: (1) they emphasize benign web content rather than footage of possibly malicious activity, and (2) they rely on older or open-source generators that do not represent recent advances in generative systems. We assemble CCtv DeepFakes (CCDF), a video deepfake dataset, to address both gaps. CCDF contains 1840 videos (460 real and 1380 generated) spanning 16 crime and accident categories, with generated content produced using three leading commercial systems: Grok Imagine, Google VEO 3.1, and OpenAI Sora 2. CCDF is a highly realistic, small-scale, manually annotated dataset targeting evaluation of detection models. We release three versions of the dataset: the raw generated data, a cleaned version in which video metadata are standardized between real and synthetic samples to prevent detectors from exploiting trivial cues, and an altered version simulating low-effort post-processing attacks. We evaluate CCDF with ten recent state-of-the-art detectors covering different detection approaches. Our results suggest that these approaches do not reliably distinguish CCDF's generated videos from real ones, despite their strong reported performance on existing datasets. These results further confirm that existing datasets are not well-suited to evaluating certain threats.

1 Introduction

Prior deepfake datasets emphasize benign content and older or open-source generators, while mismatched video formats can enable shortcut detection. CCDF targets plausible malicious surveillance footage from commercial generators, and ten recent detectors do not reliably distinguish its generated videos from real ones.

1 Introduction

  • Existing datasets emphasize benign web content, older or open-source generators, and real–synthetic format mismatches that permit detection from superficial cues.These limitations make it harder to assess detector performance under realistic conditions.Source: §1 · p. 2
  • CCDF contains surveillance footage of crimes and accidents generated by Grok Imagine, VEO 3.1, and Sora 2, with standardized encoding properties between real and synthetic videos.The dataset has 1,840 videos across 16 categories and includes raw, cleaned, and altered versions.Source: §1 · pp. 2–3
  • Ten recent state-of-the-art detectors do not reliably distinguish CCDF-generated videos from real ones, despite strong reported performance on existing benchmarks.The benchmark targets a gap between current evaluation datasets and threat conditions detectors may face in deployment.Source: §1 · p. 3

2 Existing Datasets and Methods

Prior datasets evolved from face-centered detection toward arbitrary video, but gaps in content, generator coverage, and format matching remain. The surveyed detectors use pretrained features, training-data strategies, or dynamic video cues.

2.1 Face-Based Deepfake Datasets

  • Face-focused resources include DFDC’s over 100,000 face videos, DF40’s 40 open-source generators, and web-collected WildDeepfake, but do not address fabricated-event detection.These datasets support face-centric deepfake detection, while the paper distinguishes fabricated events as a separate target.Source: §2.1 · p. 4

2.2 General-Content Video Deepfake Datasets

  • General-content datasets broaden video and generator coverage, yet commonly emphasize benign scenes, older or open-source generators, or mismatched real–synthetic formats.Examples include GenVideo, VidProM, AIGVDBench, VidForensic, and surveillance-style SurFITR; SurFITR uses still images and open-source generators.Source: §2.2 · p. 4

2.3 Deepfake Detection

  • The surveyed detectors span pretrained-feature classification, training-data or augmentation strategies, and dynamic video-feature methods.Examples include CLIP-based approaches, bias-reducing or large-generator training data, and methods using motion, physical consistency, or temporal differences.Source: §2.3 · p. 5

3 CCDF dataset

CCDF is a small-scale evaluation dataset grounded in human-verified descriptions of real surveillance footage and generated with commercial video systems. Its construction uses a two-stage image-then-video process, standardized and altered versions, and a release designed for reproducibility.

3 CCDF dataset

  • CCDF targets surveillance footage of potentially malicious events, using frontier commercial generators and video properties designed to limit common dataset biases.The paper argues that surveillance footage can be more readily accepted as real and has appeared online as real content.Source: §3 · p. 5

3.1 Prompt Creation

  • Prompts derive from 20-second clips sampled from UCF-Crime, XD-Violence, and MSAD, and human reviewers correct them to match source footage.The source clips are manually cut from 666 selected videos; XD-Violence contributes only three categories because many clips are not surveillance footage.Source: §3.1 · p. 6

3.2 Video Generation

  • Generation first creates an image from source, setting, and initial-frame descriptions, then creates video using that image plus action and audio descriptions.The authors report that preliminary tests found this two-stage approach produced higher-quality videos than direct text-to-video generation.Source: §3.2 · pp. 6–7

3.3 Dataset Curation

  • The cleaned set standardizes duration, frame rate, audio, codec, resolution, and generated-video bitrate; the altered set simulates low-effort attacks through frame-rate, color, and resolution changes.Only videos generated successfully by all three systems are retained, yielding 1,840 videos: 460 real and 460 per generator.Source: §3.3 · p. 7

3.4 Dataset Statistics

  • Moderation discarded up to two-thirds of some violent-event categories, Sora 2 blocked more than 150 videos, and raw-set format differences risked shortcut learning.CCDF is smaller than some datasets but focuses on recent generators and surveillance footage.Source: §3.4 · pp. 7–8

3.5 Distribution, Reproducibility, licensing and ethics

  • The release shares raw generated videos and prompts, but not source real videos; code and instructions support reconstruction and reproduction of cleaned and altered sets.File hashes are provided to verify reconstructed source videos against those used in the experiments.Source: §3.5 · p. 8

4 Benchmarks

In zero-shot evaluation, none of the ten detectors reliably distinguished real from generated videos on CCDF’s cleaned set, despite strong performance on other benchmarks. Performance on raw and altered CCDF data was also limited, highlighting challenges from both frontier generators and surveillance footage.

Evaluation setup

  • The benchmark tested ten methods in a zero-shot setting on cleaned and altered CCDF, using author-provided code and weights without retraining.Image-based methods used eight frames sampled at 1 fps, with logits averaged to produce a video-level score.Source: §4 · pp. 8–9
  • GenVideo served as an additional reference benchmark to check method representativeness, image-based approaches on video frames, and the 0.5 accuracy threshold.The paper reports GenVideo test-set results in Table 7 and uses them as a frame of reference for CCDF.Source: §4 · p. 9, Table 7 · p. 10

Results

  • Community-Forensics and Waverep achieved moderate performance on raw CCDF, but both dropped considerably on the cleaned set, indicating reliance at least partly on data-distribution biases.The authors observe a performance drop for all methods and argue this highlights the need to limit biases in dataset design.Source: §4 · pp. 9–10, Table 6 · p. 10
  • On CCDF’s cleaned set, none of the 10 methods accurately distinguished real from generated videos; the paper says this was in stark contrast with results on GenVideo.Source: §4 · p. 9
  • On raw VEO3.1 data, STALL reached 0.599 AUC and 0.587 AP, below the 0.86 AUC and 0.87 AP reported for VEO3 data in [2].The paper concludes that surveillance footage is more challenging beyond the use of frontier models.Source: §4 · p. 10

Altered set

  • Detection on the altered set was also poor, so the authors could not evaluate whether detectors that performed well on cleaned CCDF would withstand simple video alterations.The altered version is intended to support future evaluation of resilience to low-effort post-processing attacks.Source: §4 · p. 10

Implications

  • The findings point to a need for detection datasets that match high-stakes operational scenarios and methods that better detect content from frontier video generators.Source: §4 · p. 10

5 Limitations

The authors identify limits in human validation, dataset scale and generator coverage, and the diversity and length of surveillance content represented.

Human validation

  • The authors did not test whether generated clips reliably fool people; a user study ranking their ability to deceive is future work.Source: §5 · p. 10

Dataset scale

  • CCDF is relatively small compared with web-scraped datasets and currently covers only three generative methods.Source: §5 · p. 10

Coverage

  • Generated clips are limited to 8 seconds, and some of the 16 action categories contain fewer than 10 valid samples.These constraints limit representation of longer video and the diversity of surveillance-camera content.Source: §5 · p. 11

6 Summary

The paper introduces CCDF as a surveillance-footage benchmark for deepfake detection using recent commercial generators. Its benchmark results show that current methods struggle to distinguish generated from real videos, motivating more representative datasets and more robust detectors.

Contribution

  • CCDF focuses on surveillance footage and recent commercial generators to better represent deepfakes that may be used for malicious purposes.Source: §6 · p. 11

Results

  • Current detection methods struggle to distinguish generated videos from real videos on CCDF despite performing well on other datasets.Source: §6 · p. 11

Implications

  • The paper identifies a need for harder generated-video datasets representing malicious attacks and detection methods more resilient to zero-shot detection on recent models.CCDF is presented as one step toward the dataset challenge.Source: §6 · p. 11

A.1 Disclaimer

The disclaimer limits the dataset’s stated purpose to research, development, and education, and prohibits presenting its footage as real.

Dataset description

  • The footage consists of AI-generated surveillance video.Source: §A.1 · p. 14

Permitted purpose

  • The stated uses are research, development, and educational purposes, including computer vision training, anomaly detection research, and model evaluation.Source: §A.1 · p. 14

Use restriction

  • The footage must not be used in a context that could mislead others into believing it is real.Source: §A.1 · p. 14

A.2 Prompt Generation Parameters

Prompt extraction used gemini-3-pro-preview with a safety-focused system instruction and a detailed prompt for factual surveillance-video descriptions.

A.2 Prompt Generation Parameters

  • Prompt extraction used gemini-3-pro-preview with dangerous-content, hate-speech, and harassment settings OFF; all other parameters remained at their defaults.Source: §A.2 · p. 14
  • The system instruction framed the model as a security and safety analyst producing objective, clinical, forensic descriptions of surveillance footage.Source: §A.2 · p. 14
  • Descriptions had to stay grounded in visible or audible evidence, allowing category-level interpretation only for the main action and forbidding speculation about motives, identities, or unseen elements.Source: §A.2 · p. 14

Style Requirements

The style requirements prioritize concise, factual, neutral descriptions and restrict interpretation to the main action when necessary.

Style Requirements

1. Video Source and Technical Details

The first prompt section records technical and visual properties of the surveillance footage, including camera setup and metadata.

1. Video Source and Technical Details

  • The section requires three keypoints covering video quality, camera details, and metadata.Source: §1 · p. 15
  • Video quality should cover resolution, clarity, and sharpness.Source: §1 · p. 15
  • Camera details should state position, angle, zoom, stability, and whether the view is handheld, fixed, or from a specific perspective; metadata overlays should also be noted.Source: §1 · p. 15

2. Background and Setting

The supplied supplement details prompt construction, generator settings, category balance, and detector evaluations across CCDF versions. It reports weak detection on raw and cleaned data and examines format-related effects.

2. Background and Setting; 3. Initial Frame Description; 4. Actions and Events; 5. Audio Description

  • The prompt template asks for footage quality, camera and metadata details, background and setting, the initial frame, event sequence and camera motion, and audio description.Source: §2 · p. 15, §3 · p. 15, §4 · p. 15
  • The curated example describes a static, high-angle street camera, a parked sedan, and a minivan drive-by shooting, with no audio.Source: §5 · p. 16

A.3 Gemini and VEO3.1 Parameters; A.4 Grok Imagine Parameters; A.5 Sora 2 Parameters

A.6 Additional Statistics

  • Category counts are imbalanced, and generator moderation further reduces the size of some categories; the authors say future CCDF versions will address this.Source: Table 8 · p. 17, §A.6 · p. 17

A.7 Full results

A.8 Ablation on the modification of the Cleaned set

  • Resolution and bitrate differences affected several detectors, while setting frame rate to 24 fps or changing video duration produced little or no reported effect.Mirroring real-video resolution caused a consistent performance drop; bitrate changes strongly affected some methods, including C-Forensics and WaveRep.Source: §A.8 · p. 17, Table 19 · p. 23, Table 20 · p. 24

Related papers

Loading 2610.07939v1…