Source-linked AI summary
Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024
Nuria Alina Chandra, Hannah Lee, Ryan Murtfeldt, Lin Qiu, Arnab Karmakar, Emmanuel Tanumihardja, Kevin Farhat, Ben Caffee, Changyeon Lee, Jongwook Choi, Sejin Paik, Aerin Kim, Oren Etzioni
TL;DR
Existing deepfake detectors can perform well on academic datasets that are outdated and unrepresentative of real-world deepfakes, creating a need for more realistic evaluation. The paper introduces Deepfake-Eval-2024, an in-the-wild multimodal benchmark collected from social media and TrueMedia.org users, and finds substantially lower open-source model performance on it, while finetuning improves results. The benchmark remains limited by costly, potentially error-prone labeling and rapid changes in generative AI.
Problem
Academic deepfake benchmarks are outdated and unrepresentative of contemporary real-world deepfakes, despite detectors achieving high performance on them.
Method
The paper constructs Deepfake-Eval-2024 from potentially AI-manipulated media flagged by social media and TrueMedia.org users, and evaluates open-source models before and after finetuning.
Results
AUC drops by 50% for video, 48% for audio, and 45% for image models on Deepfake-Eval-2024 versus prior academic benchmarks, while finetuning improves AUC by 57.6%, 80.6%, and 4.5%, respectively.
Takeaways & Limitations
Contemporary in-the-wild benchmarks expose limitations of off-the-shelf detectors and can improve model performance when used for finetuning.
Takeaways & Limitations
Curating and manually labeling in-the-wild data is costly and susceptible to human error, making the dataset suitable for evaluation but insufficient for wide-scale training.
Abstract
from arXiv · showhide
In the age of increasingly realistic generative AI, robust deepfake detection is essential for mitigating fraud and disinformation. While many deepfake detectors report high accuracy on academic datasets, we show that these academic benchmarks are out of date and not representative of real-world deepfakes. We introduce Deepfake-Eval-2024, a new deepfake detection benchmark consisting of in-the-wild deepfakes collected from social media and deepfake detection platform users in 2024. Deepfake-Eval-2024 consists of 45 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing the latest manipulation technologies. The benchmark contains diverse media content from 88 different websites in 52 different languages. We find that the performance of open-source state-of-the-art deepfake detection models drops precipitously when evaluated on Deepfake-Eval-2024, with AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks. We also evaluate commercial deepfake detection models and models finetuned on Deepfake-Eval-2024, and find that they have superior performance to off-the-shelf open-source models, but do not yet reach the accuracy of deepfake forensic analysts. The dataset is available at https://github.com/nuriachandra/Deepfake-Eval-2024.
1. Introduction
Deepfake-Eval-2024 addresses the gap between high academic-benchmark performance and the increasingly diverse, realistic deepfakes circulating in the wild. It introduces a multimodal benchmark designed to evaluate detectors on contemporary real-world content.
- A fourfold increase in deepfakes detected in fraud occurred between 2023 and 2024, alongside widespread social-media sharing of deepfakes.An estimated 500,000 deepfakes were shared on social media in 2023.
- Academic deepfake detectors achieve AUC values approaching one, but their datasets use outdated manipulations and are not representative of social-media deepfakes.Examples include GenConViT for video, AASIST for audio, and NPR for images.
- Deepfake-Eval-2024 collects potentially AI-manipulated media from social media and TrueMedia.org users in an in-the-wild multimodal benchmark.The collection includes videos, audio, and images and is intended to represent contemporary deepfakes more directly than synthetic benchmarks.
- Figure 1 illustrates the benchmark’s diversity across content styles and generation techniques, including lipsync, faceswap, and diffusion.Rows 1–2 show video and audio examples, while rows 3–4 show images.
2. Related Work
Prior deepfake datasets are often outdated, narrowly sourced, or synthetic, limiting their representation of real-world media. Deepfake-Eval-2024 expands in-the-wild coverage across modalities, sources, languages, and contemporary content.
- Prior in-the-wild datasets were sparse and outdated, including audio data from 2022, image-focused data absent before this benchmark, and video data from 2021 and 2023.The passage identifies Deepfake-Eval-2024 as filling a temporal and modality-specific dataset gap.
- Deepfake-Eval-2024 samples media flagged as potentially AI-manipulated rather than relying on explicit deepfake labels, improving relevance to unlabeled security threats.Users of social media and TrueMedia.org, primarily journalists, flagged the collected media.
- 88 web sources and 52 languages in the combined video and audio datasets substantially expand diversity beyond prior in-the-wild collections.The audio dataset alone spans 42 languages, whereas prior audio datasets used at most two.
- Synthetic datasets support large scale but often use curated videos, limited manipulation techniques, face-focused edits, or repurposed video frames.These design choices fail to capture the distribution of deepfakes circulating on social media.
3. Dataset
Deepfake-Eval-2024 combines user-flagged real-world media with broad multimodal, linguistic, and content diversity. Its labeling process uses source verification and forensic analysis, while audio labels necessarily incorporate commercial detector predictions and remain correlated with existing detectors.
- Dataset composition: 45 hours of video, 56.5 hours of audio, and 1,975 images comprise the benchmark, including real, AI-generated, and AI-manipulated media.Audio includes video-derived and audio-only media, and most videos have corresponding labeled audio.
- Data collection: Users submitted potentially manipulated media through TrueMedia.org, direct uploads, an X bot, and X Community Notes, with X, TikTok, Instagram, and YouTube among the most common sources.The collection draws on media flagged by users, including journalists and fact-checkers, rather than only explicitly labeled deepfakes.
- Dataset attributes: User-flagged collection is estimated to yield more challenging real and fake examples than prior datasets because users often submit media they cannot easily classify.This sampling approach is intended to represent difficult in-the-wild cases rather than only convenient synthetic examples.
- Dataset diversity: The dataset spans 88 web domains and 52 languages, while its circulating content varies across ethnicities, generative models, media styles, and subject matter.It is 78.7% English, with visual data estimated at 54.6% white and 45.4% non-white.
- Data labeling: Labelers verified sources and media characteristics, assigning fake, real, or unknown labels through team-based forensic analysis.Verification included reverse image search, web searches, fact-checking sources, community moderation records, and comparisons with original media.
- Data labeling: Audio examples are labeled fake only when audible indicators coincide with predictions from at least two commercial audio detectors, creating correlation with those detectors.Audio detectors alone are never used, but the authors identify this correlation as a limitation of the audio dataset.
- Labeling agreement: Inter-labeler disagreement was 6.6% for video, 7.9% for audio, and 9% for images, remaining below 10% across modalities.The largest reported discrepancies involved real-versus-unknown and fake-versus-unknown labels.
4. Experiments
The experiments evaluate open-source, commercial, and finetuned deepfake detectors on Deepfake-Eval-2024 and compare them with prior benchmarks. Open-source performance drops substantially on contemporary in-the-wild data, while finetuning and commercial systems improve results without reaching human-analyst accuracy.
- Model evaluation: Open-source models were evaluated across video, audio, and image modalities using modality-specific architectures and pretrained weights.The study included three open-source models per modality and selected models before experimentation without omitting models based on performance.
- Evaluation protocol: Models were compared on Deepfake-Eval-2024, original publication test datasets, and multiple metrics including AUC, F1-score, and accuracy.Original-benchmark metrics were recomputed where needed, with averages reported across multiple test datasets.
- Open-source performance: The maximum open-source AUC on Deepfake-Eval-2024 was 0.58, with many models near 0.5, suggesting reliance on academic-dataset correlations absent from contemporary data.The multimodal models evaluated also performed similarly poorly.
- Open-source performance: Average AUC fell by 50% for video, 48% for audio, and 45% for image models relative to their original academic benchmarks.This contrast indicates that prior academic datasets do not represent contemporary in-the-wild deepfakes well.
- Finetuned performance: Finetuning improved average AUC by 57.6% for video, 80.6% for audio, and 4.5% for images, but peak accuracy remained 0.75, 0.86, and 0.63, respectively.These peaks remained below the 90% lower-bound estimate for human deepfake forensic analysts.
- Commercial performance: Top commercial models outperformed off-the-shelf open-source models and finetuned image models, while performing slightly better than finetuned audio and video models.No evaluated commercial model reached 90% accuracy, and commercial models still needed improvement to match human analysts.
5. Error Analysis
Error analysis identifies media traits associated with detector failures across video, audio, and image data. Several vulnerabilities persist after finetuning, while many errors remain unexplained by human-identifiable traits.
- Video errors: Diffusion-generated videos produced 21.3% lower average accuracy for off-the-shelf GenConViT and FTCN models.After finetuning on Deepfake-Eval-2024, the accuracy gap narrowed to 5.4%, suggesting domain shift contributed to the original underperformance.
- Video errors: Selective face manipulation reduced accuracy by 31%, while non-facial manipulation reduced accuracy by 17.4% relative to other videos.After finetuning, accuracy deficits remained 16.85% and 35.47% for these two manipulation types, respectively.
- Audio errors: Finetuned audio models had 7.21% lower accuracy on non-English audio, 35.39% worse accuracy on silent clips, and 7.66% lower accuracy with background noise.Background music was associated with a 17.94% accuracy drop and a 26.12% increase in false-negative rate.
- Image errors: Finetuned image-model accuracy decreased by 9% on images with text overlays, although category-level differences were not statistically significant.The authors associate this observation with distributional mismatch because existing training datasets lacked text-overlay images.
- Residual errors: Human-identifiable media traits did not account for all errors, including averages of 33% of audio errors with music and 5% of image errors with text overlays.The remaining errors were attributed to other failures in model signal interpretation.
6. Discussion & Limitations
Deepfake-Eval-2024 is valuable for evaluating real-world threats, but its use is constrained by curation costs, labeling errors, rapid dataset obsolescence, and potential misuse.
- In-the-wild data is crucial for evaluating detection models against real-world threats.
- Manual curation and labeling are costly and susceptible to human error, making the dataset suitable for evaluation but insufficient for wide-scale training.
- Rapidly evolving generative AI can quickly make datasets outdated, requiring systems that track emerging deepfakes and regularly update datasets.
- Adversarial actors could use Deepfake-Eval-2024 to develop generation techniques that evade detectors.
- The authors restrict dataset access to verifiably relevant individuals and research institutions under a CC-BY-SA-4.0 license.
Circulated in 2024
The paper reviews popular deepfake detection datasets released before Deepfake-Eval-2024, with coverage focused on the pre-March 4, 2025 landscape.
- The survey covers popular deepfake detection datasets released before Deepfake-Eval-2024.
- The comparison focuses on datasets available before March 4, 2025.
- The section provides a detailed overview and comparison of existing datasets with Deepfake-Eval-2024.
A.1.1. Overlap between Modality Datasets
The survey distinguishes modality overlap and separates audio-visual components where possible, while estimating missing duration values using explicit conventions.
- Overlap between modality datasets: Most video datasets contain manipulated or AI-generated frames without accompanying real or fake audio, while a few provide audio-visual data.
- Overlap between modality datasets: Datasets with separable audio and video components are marked with (A) or (V) to identify the reported modality.
- Overlap between modality datasets: Image datasets composed of individual video frames are omitted from the image survey to avoid duplicate reporting across modalities.
- Duration estimation: When duration is unavailable, estimates may use frame counts at 30fps, average clip lengths, reported estimates, or survey-derived values.
A.2. Dataset Supplementary Figures
The supplementary figures and tables document Deepfake-Eval-2024’s modality summaries, source origins, language distributions, demographics, and evaluation setup.
- Summary statistics: Supplementary tables summarize video, audio, and image statistics for Deepfake-Eval-2024.
- Dataset origins: Figure S1 separates data origins by modality and distinguishes direct uploads to TrueMedia.org from links to social media websites.
- Language distributions: Figure S2 presents language distributions for audio and video content.
- Demographic annotation: A randomly sampled 10% of visual data was manually annotated for perceived race and ethnicity, without definitive claims about subject identity.
- Computational setup: All models were evaluated and finetuned on either a single AWS A10 or a single GCP L4 GPU, with no observed difference between platforms.
A.4. Results Supplementary Figures
The supplementary results provide complete tables for off-the-shelf open-source models, finetuned open-source models, commercial models, and open-source multimodal models across modalities.
- Table S8 reports complete off-the-shelf open-source model results across modalities.
- Table S9 reports complete open-source model finetuning results across modalities.
- Table S10 reports the best commercial model performance on Deepfake-Eval-2024.
- Table S11 reports open-source multimodal model results.
A.5. Dataset Access and Ethics
The dataset uses explicit criteria and source verification to label media, while restricting access because released deepfake data could be misused to develop more realistic or evasive generators.
- Dataset Access and Ethics: Dataset access is gated to verified researchers or deepfake-detection practitioners through institutional evidence and terms of use.
- Dataset Access and Ethics: The authors plan to broaden access in May 2027 because newer generative AI is expected to reduce the risk posed by releasing this dataset.
- Labeling Criteria: Items are labeled real, fake, or unknown using evidence of AI manipulation, absence of manipulation, or inability to confirm either status.
- Labeling Criteria: Media containing three or more specified AI traits is classified using stylistic, anatomical, and sociocultural indicators.
- Ground-Truth Verification: Reverse image search and professional fact-checking sources support ground-truth determination when common AI traits are insufficient.
- Ground-Truth Verification: Source trustworthiness is assessed with AllSides and Ad Fontes Media Bias before using located sources to determine ground truth.