PaperlayerView exact revision on arXiv

Source-linked AI summary

FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset

Hasam Khalid, Shahroz Tariq, Minha Kim, Simon S. Woo

arXiv:2108.05080v4cs.CVcs.MMcs.SDeess.AS

TL;DR

Existing deepfake datasets often lack multimodal labels, diversity, or synchronized audio, limiting realistic audio-video detection research. FakeAVCeleb addresses this gap with a gender- and racially balanced dataset of lip-synced fake videos and cloned audios, which proves generally challenging for baseline detectors.

  • Problem

    Existing deepfake datasets often provide only one modality or lack fine-grained audio-video labels and demographic diversity, hindering multimodal detector development.

  • Method

    The paper constructs FakeAVCeleb from celebrity videos across four ethnic backgrounds, generating deepfake videos and lip-synced cloned audios with popular synthesis methods.

  • Results

    FakeAVCeleb is generally the most complex and challenging dataset for the evaluated baseline detection methods, with overall performance close to Celeb-DF.

  • Takeaways & Limitations

    FakeAVCeleb provides a multimodal, gender- and racially balanced resource for developing and evaluating audio-video deepfake detectors.

  • Takeaways & Limitations

    Existing comparison datasets generally lack demographic diversity, while DFDC does not identify whether audio or video is fake and lacks lip-synced synthesized audio.

Abstract

from arXiv · show

While the significant advancements have made in the generation of deepfakes using deep learning technologies, its misuse is a well-known issue now. Deepfakes can cause severe security and privacy issues as they can be used to impersonate a person's identity in a video by replacing his/her face with another person's face. Recently, a new problem of generating synthesized human voice of a person is emerging, where AI-based deep learning models can synthesize any person's voice requiring just a few seconds of audio. With the emerging threat of impersonation attacks using deepfake audios and videos, a new generation of deepfake detectors is needed to focus on both video and audio collectively. To develop a competent deepfake detector, a large amount of high-quality data is typically required to capture real-world (or practical) scenarios. Existing deepfake datasets either contain deepfake videos or audios, which are racially biased as well. As a result, it is critical to develop a high-quality video and audio deepfake dataset that can be used to detect both audio and video deepfakes simultaneously. To fill this gap, we propose a novel Audio-Video Deepfake dataset, FakeAVCeleb, which contains not only deepfake videos but also respective synthesized lip-synced fake audios. We generate this dataset using the most popular deepfake generation methods. We selected real YouTube videos of celebrities with four ethnic backgrounds to develop a more realistic multimodal dataset that addresses racial bias, and further help develop multimodal deepfake detectors. We performed several experiments using state-of-the-art detection methods to evaluate our deepfake dataset and demonstrate the challenges and usefulness of our multimodal Audio-Video deepfake dataset.

1 Introduction

Recent deepfake generation can produce highly realistic manipulated media and cloned speech, intensifying ethical, security, and privacy concerns and creating an urgent need for detection methods. FakeAVCeleb addresses limitations in existing datasets by providing a fine-grained, multimodal audio-video benchmark designed to support less racially biased detection research.

  • Motivation: Deep learning has enabled realistic fake faces, videos, and real-time cloned human voices, making deepfakes a challenging technical, social, and ethical issue.Voice cloning generates high-quality speech resembling target speakers, while synthesized videos can exhibit accurate lip-sync.
  • Motivation: Deepfake misuse raises ethical, security, and privacy concerns, while effective detection requires high-quality datasets.The introduction identifies detection as inevitable and notes that researchers have proposed varied detection methods.
  • Dataset gap: Existing DFDC data mixes fake video and synthesized audio without audio-video labels, making it difficult to determine which modality is manipulated.The passage identifies DFDC as the only known dataset combining these modalities but emphasizes its labeling limitation.
  • Dataset contribution: FakeAVCeleb provides video and audio deepfakes with accurate lip-sync and fine-grained labels in a previously unavailable multimodal dataset.It is presented as a novel Audio-Video Multimodal Deepfake Detection dataset.
  • Dataset contribution: FakeAVCeleb includes three Audio-Video deepfake types generated from carefully selected real YouTube videos using popular recent generation methods.The dataset construction targets realistic multimodal deepfake examples.
  • Dataset contribution: The dataset selects English-speaking individuals from five major ethnic backgrounds, and baseline evaluation demonstrates the need and usefulness of multimodal deepfake data.The selection is intended to address racial bias issues.

2 BACKGROUND AND MOTIVATION

Existing deepfake datasets largely focus on manipulated videos and face swapping, while suffering from quality, quantity, diversity, and racial-bias limitations. FakeAVCeleb addresses this gap with deepfake videos paired with cloned, lip-synced fake audio for multimodal detection.

  • Existing datasets: Existing public datasets predominantly contain real videos and corresponding face-manipulated fake videos, despite the availability of diverse deepfake-generation methods.Recent datasets have sought improved quality and larger scale.
  • FakeAVCeleb motivation and contribution: FakeAVCeleb generates cloned target-speaker voices, lip-syncs them to facial-reenactment videos, and provides paired fake audio and video for simultaneous multimodal detection.The authors describe it as the first dataset of its kind containing deepfake videos with their respective fake audios.
  • Existing datasets: Early datasets such as UADFV and Deepfake TIMIT provide limited baselines, with UADFV containing only 98 videos and Deepfake TIMIT using real rather than synthesized audio.These datasets are characterized by low quality and quantity.
  • Dataset limitations: Large-scale datasets including FF++ and DFDC improve quantity, but commonly lack demographic diversity, while DFDC also includes difficult recording conditions such as extreme lighting and off-camera walking.FF++ contains 5,000 videos and DFDC contains 128,154 videos, generated using four and eight methods, respectively.
  • Dataset limitations: Several datasets exhibit racial bias, including UADFV, Deepfake TIMIT, and KoDF, whose videos predominantly represent restricted ethnic or linguistic populations.DeeperForensics-1.0 is identified as racially unbiased in the supplied passage, though its description is truncated.

3 Dataset Collection and Generation

FakeAVCeleb was built from 500 carefully selected, ethnically diverse real VoxCeleb2 videos and expanded to 20,000 inspected deepfake videos. It combines face swapping, facial reenactment, voice cloning, and lip synchronization to provide four audio-video configurations for multimodal detection.

  • Real Dataset Collection: The real baseline contains 500 single-person VoxCeleb2 videos averaging 7.8 seconds, selected for clear, centered faces without hats, glasses, or masks.The videos represent diverse ethnicity, gender, and age.
  • Dataset Configurations: The dataset covers four audio-video combinations: ARVR, AFVR, ARVF, and AFVF, representing real or fake audio paired with real or fake video.The reported combination counts are ARVR (500), AFVR (500), ARVF (9,000), and AFVF (10,000).
  • Deepfake Dataset Generation: Researchers generated around 20,000 deepfake videos from the 500 real videos using FaceSwap, FSGAN, Wav2Lip, and SV2TTS.FaceSwap and FSGAN performed face swapping, Wav2Lip performed audio-driven facial reenactment, and SV2TTS generated cloned voices.
  • Quality Control: Generated samples were manually inspected for realistic video, correctly synthesized audio, and lip synchronization, producing a final count of 20,000 videos.Synthesis was applied separately within ethnic and gender groups to make the fakes more realistic and natural.
  • Dataset Configurations: AFVR uses SV2TTS to pair cloned fake speech with real video, whereas ARVF uses FaceSwap, FSGAN, and Wav2Lip to pair real audio with fake video.The ARVF subset contains 9,000 deepfake videos, and Wav2Lip was selected for efficiency, lower time consumption, and better output quality.

4 Benchmark Experiments and Results

The benchmark evaluates FakeAVCeleb v1.2 with multiple state-of-the-art deepfake detectors using frame-level AUC, while combining audio and video for FakeAVCeleb experiments. It also examines performance under H.264 compression and compares FakeAVCeleb with other deepfake datasets.

  • Preprocessing: FakeAVCeleb preprocessing handles videos and audios separately, representing audio as three-channel MFCC images for model input.The videos are face-centered from VoxCeleb2, while MFCC features are computed per audio frame.
  • DeepFake Detection Methods: The experiments compare FakeAVCeleb with other deepfake datasets using eight available state-of-the-art deepfake detection methods.The listed methods include Capsule, HeadPose, VA-MLP/LogReg, Xception, Meso4, and MesoInception.
  • Results and Analysis: The results use FakeAVCeleb v1.2 and evaluate each detector with frame-level Area Under the ROC curve (AUC), rounding classification scores to 10^-3 precision.Table 3 reports the results for each experiment.
  • Results and Analysis: The benchmark additionally reports Xception-comp frame-level AUC on FakeAVCeleb videos compressed with H.264 at medium (15) degrees.The compression analysis addresses quality changes caused by online video upload and distribution.
  • Results and Analysis: For FakeAVCeleb, Figure 4 forms ensemble ROC curves from audio MFCCs and video frames, unlike the video-only evaluation used for the other datasets.The reported AUC scores for the three SOTA models on FakeAVCeleb are 72.5%, 61.7%, and 60.9%.

5 Discussion and Future Work

The discussion emphasizes quality control and responsible availability for FakeAVCeleb, while outlining future updates to expand, modernize, and strengthen the dataset. Planned improvements include adapting to new generation methods, adversarial attacks, polishing methods, and broader public video sources.

  • Data Quality: FakeAVCeleb’s real and generated videos underwent manual screening to select high-quality, center-aligned, uncovered faces and remove corrupted outputs.The authors applied rigorous quality checks before and after face-swapping.
  • Data Availability and Social Impact: The dataset is openly available through its DOI information page, but its accessible generation methods and content could be misused to evade existing deepfake detectors.The authors do not release a separate code repository because the generation methods are already open-sourced and publicly accessible.
  • Future Directions: Future releases will track the latest deepfake video and audio generation methods and account for potential adversarial attacks.The authors plan to construct the dataset accordingly as threats evolve.
  • Future Directions: The authors plan to use recent deepfake polishing methods, enlarge the dataset with more publicly accessible videos such as YouTube-8M, and include these changes in dataset maintenance.These updates target artifact reduction, broader data coverage, and continued maintenance.

6 Conclusion

FakeAVCeleb is presented as a multimodal deepfake dataset for detecting both deepfake videos and audios. It combines synthesized cloned audios with videos designed to reduce gender and racial bias across ages and four major races.

  • Dataset contribution: FakeAVCeleb is a novel Audio-Video multimodal deepfake dataset for detecting deepfake videos and deepfake audios.The dataset contains deepfake videos together with their respective synthesized cloned audios.
  • Dataset design: The dataset contains videos of men and women from four major races across different age groups.Its design targets gender and racial unbiasedness.
  • Dataset generation: FakeAVCeleb was generated using a range of recent, popular deepfake video and audio generation methods.The supplied passage identifies these methods as part of the dataset-generation process, but does not specify them individually.

Broader Impact · A Dataset Publication · A.2 Hosting Platform

FakeAVCeleb addresses the need for realistic multimodal deepfake data by combining deepfake videos with lip-synced synthesized audio generated using four popular methods. The dataset is hosted on a Google Drive account managed by DASH Lab at Sungkyunkwan University.

  • Broader Impact: FakeAVCeleb targets the need for high-quality, realistic datasets to build strong deepfake detectors.The passage identifies dataset quality and realism as requirements for effective detection.
  • Broader Impact: Existing deepfake datasets contain only forged videos or synthesized audio, encouraging unimodal detection methods.The passage contrasts prior unimodal resources with FakeAVCeleb’s multimodal design.
  • A Dataset Publication: FakeAVCeleb includes deepfake videos and synthesized fake audio that is lip-synced to the video.This pairing supports audio-video deepfake detection within the same dataset.
  • A Dataset Publication: The dataset was generated using four popular deepfake-generation and synthetic-voice-generation methods.The passage describes the generation-method coverage used to construct FakeAVCeleb.
  • Broader Impact: FakeAVCeleb therefore provides both visual and auditory manipulated content for multimodal deepfake detection.This point summarizes the dataset’s combined modalities without adding a result beyond the supplied description.
  • A.2 Hosting Platform: The dataset is hosted on a Google Drive account belonging to DASH Lab.The account is managed by Simon S. Woo, a corresponding author at Sungkyunkwan University, South Korea.

A.3 Access to Dataset · B Dataset Generation Methods · B.1 Deepfake Detection Baseline Methods

FakeAVCeleb restricts dataset access through manual screening and constructs multimodal deepfakes using four synthesis methods. It evaluates the dataset with eleven frame-level-AUC detection methods, including adapted multimodal baselines and separate audio-video preprocessing.

  • A.3 Access to Dataset: Users must complete a manually screened dataset request form, while only a small sample is publicly available on GitHub.The access controls aim to limit potential misuse by malicious actors seeking to evade deepfake detectors.
  • B Dataset Generation Methods: The dataset uses four deepfake generation or synthesis methods spanning face swapping, facial reenactment, lip-syncing, and voice cloning.The methods are FaceSwap, FSGAN, Wav2Lip, and SV2TTS.
  • B Dataset Generation Methods: FaceSwap swaps faces between images or videos while retaining body and environmental context, using an encoder with separate decoders for each source identity.The authors use the open-source FaceSwap software to generate high-quality face-swapped videos.
  • B Dataset Generation Methods: FSGAN reenacts target-video pose, angle, and expression from corresponding source frames before transferring and blending missing facial parts.The authors use the official FSGAN GitHub implementation and its best-quality swapping configuration.
  • B Dataset Generation Methods: Wav2Lip performs audio-driven facial reenactment so a video’s mouth movements match a desired speech signal.Lip-sync controls mouth movement using the source speech, while face reenactment manipulates facial features in the target video.
  • B Dataset Generation Methods: SV2TTS is a real-time voice-cloning tool that synthesizes a speaker’s voice from a few seconds of input audio using independently trained encoder, synthesis, and vocoder components.The encoder produces a fixed-dimensional audio embedding, while the synthesis network generates a Mel spectrogram.
  • B.1 Deepfake Detection Baseline Methods: The evaluation compares eleven detection methods using default parameters and frame-level AUC scores from individual-frame classification outputs.The baselines include capsule, head-pose, visual-artifact, Xception, MesoNet, frequency-aware, blending-boundary, lip-movement, multimodal, and CDCN-based approaches.
  • B.1 Deepfake Detection Baseline Methods: Multimodal-1 and Multimodal-2 are adapted by replacing textual inputs with audio processing, while CDCN is modified by removing its third visual modality.Multimodal-1 uses replicated visual networks for video and audio; Multimodal-2 uses two CNN blocks, one per modality.

B.2 Summary of Results

Across eight datasets, Xception-comp achieved the strongest average detector performance, while Headpose performed worst. FF-DF was easiest to detect, whereas CelebDF was hardest, with FakeAVCeleb relatively close to CelebDF.

  • Detector performance: 72.5% was the best average AUC achieved by Xception-comp, while Headpose recorded the worst average at 49.0%.These averages were computed across FF-DF, UADFV, DFD, DF-TIMIT LQ, DF-TIMIT HQ, FakeAVCeleb, DFDC, and Celeb-DF.
  • Dataset difficulty: FF-DF had the highest average AUC across eight detection methods, making it easiest to detect, while CelebDF had the lowest and was hardest.The methods were Headpose, Xception-raw, Xception-comp, VA-MLP, VA-LogReg, MesoInception4, Meso4, and Capsule.
  • Dataset difficulty: FakeAVCeleb’s average detection score was relatively close to CelebDF’s.The comparison used average AUC scores across eight detection methods.

C Additional Experiments · C.1 Unimodal Results · C.2 Results for Vonly Trained Classifier

Additional experiments evaluate FakeAVCeleb v1.2 using unimodal, ensemble, and multimodal settings, with unimodal results showing stronger audio-only than video-only AUC performance. For video-only detection, EfficientNet-B0 achieves the highest reported AUC, while VGG achieves the lowest.

  • C Additional Experiments: The additional experiments evaluate FakeAVCeleb v1.2 in unimodal, ensemble, and multimodal settings.Future FakeAVCeleb versions are planned as dataset quality improves.
  • C Additional Experiments: The reported results are based on the FakeAVCeleb v1.2 database, with newer-version baseline results directed to the project’s GitHub page.The paper notes that new dataset versions will be released as quality improves.
  • C.1 Unimodal Results: The unimodal test set contains real videos and all three categories of fake videos from FakeAVCeleb.Performance is reported for classifiers trained only on audio or only on video.
  • C.1 Unimodal Results: The best AUC scores are approximately 97% for Aonly and 93% for Vonly.These results are presented for the unimodal baselines in Figure 7.
  • C.2 Results for Vonly Trained Classifier: EfficientNet-B0 achieves a 93.3% AUC score for video-only detection, the best result among the reported methods.The comparison is shown in Figure 7.
  • C.2 Results for Vonly Trained Classifier: VGG achieves a 49.6% AUC score for video-only detection, the lowest reported result.The passage contrasts this result with EfficientNet-B0’s performance.
  • C.2 Results for Vonly Trained Classifier: Meso4’s recall indicates that it fails to detect most deepfake videos, while EfficientNet-B0 outperforms Xception on this task.Xception is reported as the best performer on other deepfake datasets such as FaceForensics++.

C.2.1 Results for Aonly Trained Classifier · C.2.2 Summary of Unimodal Results

Unimodal evaluation on FakeAVCeleb shows that audio detection remains difficult: VGG reaches 97.8% AUC, while Meso4 reaches 73.5%, and no baseline performs satisfactorily. EfficientNet-B0 is the most stable across modalities at 95% average performance, supporting the dataset’s realistic deepfake quality.

  • C.2.1 Results for Aonly Trained Classifier: VGG achieves the best audio AUC score, reaching 97.8%.
  • C.2.1 Results for Aonly Trained Classifier: Meso4 records the lowest audio AUC score at 73.5%.The passage attributes this to overfitting the real and fake audio classes.
  • C.2.1 Results for Aonly Trained Classifier: No baselines provide satisfactory audio-detection performance, indicating that SOTA deepfake detectors are unsuitable for deepfake audio detection.
  • C.2.1 Results for Aonly Trained Classifier: The unimodal evaluation trains models using a single modality, either Aonly or Vonly.
  • C.2.2 Summary of Unimodal Results: EfficientNet-B0 exhibits the most stable average performance, at 95% for both Aonly and Vonly.
  • C.2.2 Summary of Unimodal Results: Overall, the poor SOTA detection performance indicates that FakeAVCeleb’s fake audios and videos are realistic and difficult to distinguish from real ones.

C.3 Ensemble Results · C.4 Multimodal Results · D Dataset Request Form

Ensemble classifiers performed best with EfficientNet-B0 but remained limited overall, while baseline multimodal detectors achieved AUC scores ranging from 53.3% to 68.8%. Access to FakeAVCeleb was provided through a Google request form requiring accurate information and agreement to its terms.

  • C.3 Ensemble Results: 82.8% was the best ensemble performance, achieved by EfficientNet-B0 compared with Xception at 51.4% and F3Net at 47.6%.Meso4 ranked second at 58.2%.
  • C.3 Ensemble Results: Figure 9 reports multimodal detection performance across the video and audio modalities using three open-source multimodal methods.The figure caption identifies the evaluated modalities as V and A.
  • C.3 Ensemble Results: 55.9% was the third-highest AUC score for MesoInception4, while Face X-ray achieved 53.5%.These results indicate that ensemble methods did not achieve high detection scores overall.
  • C.3 Ensemble Results: Soft- and hard-voting produced similar ensemble prediction scores, with no significant performance difference when combining two classifiers.The passage attributes this similarity to the ensemble containing only two classifiers.
  • C.4 Multimodal Results: 68.8% was Multimodal-2's score after 50 training epochs, exceeding CDCN's 66.2% and Multimodal-1's 53.3% AUC scores.Each baseline was trained for 50 epochs, with Multimodal-1 evaluated at its best-performing epoch.
  • D Dataset Request Form: Researchers could request access to FakeAVCeleb through a Google form and use its provided citation when using the dataset.The form required correct information and agreement to follow the terms and conditions.
Loading 2108.05080v4…