Source-linked AI summary
Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward
Momina Masood, Marriam Nawaz, Khalid Mahmood Malik, Ali Javed, Aun Irtaza
TL;DR
The paper addresses the growing ability to create realistic audio-visual deepfakes and the limited breadth of prior surveys. It reviews generation and detection methods, datasets, evaluation standards, challenges, and future directions across audio and visual modalities, finding strong detector performance alongside emerging challenges from realism-preserving manipulations.
Problem
Accessible tools and advanced deep-learning methods enable realistic deepfakes for disinformation and other harms, while existing surveys have mainly focused on detecting images and videos.
Method
The paper comprehensively reviews tools and machine-learning approaches for audio and visual deepfake generation and detection, including datasets, evaluation standards, challenges, and future directions.
Results
The review finds that most detectors use deep learning and show robust performance close to 100%, but removing forensic fingerprints while preserving realism creates a new challenge.
Takeaways & Limitations
Understanding deepfake creation, detection mechanisms, limitations, and future directions requires considering the ongoing interaction between generation and detection methods across audio and video.
Takeaways & Limitations
High-quality deepfake generation requires substantial target-specific data and computationally complex retraining, while detector development is constrained by incomplete, sparse, and noisy training data.
Abstract
from arXiv · showhide
Easy access to audio-visual content on social media, combined with the availability of modern tools such as Tensorflow or Keras, open-source trained models, and economical computing infrastructure, and the rapid evolution of deep-learning (DL) methods, especially Generative Adversarial Networks (GAN), have made it possible to generate deepfakes to disseminate disinformation, revenge porn, financial frauds, hoaxes, and to disrupt government functioning. The existing surveys have mainly focused on the detection of deepfake images and videos. This paper provides a comprehensive review and detailed analysis of existing tools and machine learning (ML) based approaches for deepfake generation and the methodologies used to detect such manipulations for both audio and visual deepfakes. For each category of deepfake, we discuss information related to manipulation approaches, current public datasets, and key standards for the performance evaluation of deepfake detection techniques along with their results. Additionally, we also discuss open challenges and enumerate future directions to guide future researchers on issues that need to be considered to improve the domains of both deepfake generation and detection. This work is expected to assist the readers in understanding the creation and detection mechanisms of deepfakes, along with their current limitations and future direction.
1 Introduction
Deepfakes have become easier to create and disseminate as digital media, social platforms, machine learning, and accessible tools have advanced. This survey reviews audio-visual deepfake categories, generation and detection methods, challenges, datasets, and future directions.
- 1 Introduction: Accessible devices and social media have accelerated multimedia creation and distribution, making false information harder to distinguish from truth.The passage links this difficulty to increasingly sophisticated machine-learning methods that manipulate multimedia content.
- 1 Introduction: Deepfake videos include face-swap, lip-synching, puppet-master, face synthesis and attribute manipulation, alongside audio deepfakes.Face-swap manipulation replaces a source person's face with a target person's face to portray the target performing the source's actions.
- 1 Introduction: The paper aims to explain audio-visual deepfake generation and detection methods, their trends, limitations, challenges, implications, and future directions.Its stated organization covers methods, open challenges, datasets, future trends, and conclusions.
- 1 Introduction: The review examines published approaches for generating and detecting audio and visual manipulations using a documented literature-review protocol.The protocol is described in Table 1.
2 Disinformation and Misinformation using Deepfakes
Deepfakes can support misinformation and disinformation through manipulated audio-visual content shared on social media. Their uses include inflammatory social influence and efforts to affect public opinion or obscure reality.
- 2 Disinformation and Misinformation using Deepfakes: Misinformation is false or inaccurate information regardless of intent, whereas disinformation involves strategies to fabricate information for planned political or financial objectives.The passage describes social media as facilitating the spread of false news.
- 2 Disinformation and Misinformation using Deepfakes: Independent trolls may use manipulated racist or sexist audio-visual content to provoke disorder, emotional reactions, and hatred.The passage presents trolls as hobbyists who spread inflammatory information to cause social reactions.
3 DeepFakes Evolution
Deepfake evolution spans traditional image manipulation, shallow fakes, increasingly sophisticated generative methods, and specialized software for visual and audio synthesis. The paper situates these developments historically and illustrates their social and financial uses.
- 3 DeepFakes Evolution: Traditional image manipulation combines splicing, inpainting, or copy-move operations with post-processing such as scaling, rotation, and color adjustment.The passage identifies a manipulated portrait from 1860 as an early example.
- 3 DeepFakes Evolution: Shallow fakes use accessible editing operations such as slowing, speeding, cutting, and selective splicing to alter the context of existing footage.A selectively edited video of Nancy Pelosi received more than 2.2 million views within 48 hours.
- 3 DeepFakes Evolution: Deepfake development progressed from the 1997 Video Rewrite Program to photorealistic face generation and modern audio-visual manipulation tools.Video Rewrite automatically reanimated facial movements in existing video to match a different audio track.
- 3 DeepFakes Evolution: Open-source and commercial tools support face synthesis, face swapping, face reenactment, voice synthesis, and real-time voice cloning.Examples include DeepFaceLab, FaceSwapGAN, Face2Face, StyleGAN, Overdub, Respeecher, and SV2TTS.
- 3 DeepFakes Evolution: Audio deepfakes use speech-synthesis systems such as WaveNet, Tacotron, and Deep Voice to imitate voices and enable fraudulent applications.The paper reports a fraudulent transfer of $243,000 involving voice-mimicking AI trained on recordings obtained from the internet.
4 Audio Visual Deepfakes Types and Categorization of Literature
The paper organizes audio-visual deepfakes by the manipulated modality and reviews creation and detection methods within each category. It also examines their technologies, capabilities, limitations, challenges, and future trends.
- 4 Audio Visual Deepfakes Types and Categorization of Literature: Audio-visual deepfakes are broadly categorized into visual and audio manipulations according to the targeted forged modality.Figure 1 presents this categorization.
- 4 Audio Visual Deepfakes Types and Categorization of Literature: For each deepfake category, the review analyzes both creation and detection approaches.The analysis is intended to provide a deeper understanding of the various approaches.
4.1 Face-swap
Face-swap deepfakes replace a source face with a target identity while preserving or transferring facial content, using traditional, autoencoder, GAN, and latent-space methods. Detection combines handcrafted and deep features, but performance remains sensitive to video manipulation and compression conditions.
- Generation: Face-swap replaces the source person’s face with a target face while portraying the target performing actions actually done by the source.Traditional approaches detect faces, select a similar candidate, and replace facial regions.
- Generation: Autoencoder-decoder generation learns latent facial features with a shared encoder-decoder pair, then swaps decoders so face A receives face B’s features.
- Generation: GAN-based face-swap methods can outperform autoencoder-decoder approaches because they do not require explicit subject-specific training images.
- Detection: Face-swap detection uses handcrafted descriptors, multimedia stream features, biological signals, facial landmarks, CNNs, and CNN-RNN models.Reported limitations include failure on manipulated videos, re-encoding attacks, multi-time compression, and videos longer than two seconds.
- Detection: Detection performance remains constrained by computational expense and low performance on some datasets or settings.
4.2 Lip-syncing
Lip-syncing deepfakes modify a target video’s mouth movements to match arbitrary audio or text. Modern deep-learning methods improve visual realism and temporal consistency, while detectors assess audiovisual synchronization but remain vulnerable to limited mouth motion and sparse manipulation.
- Generation: Lip-syncing synthesizes a target identity’s mouth region so manipulated video is consistent with arbitrary audio input.Natural synthesis requires appropriate lip movements and surrounding facial expressions.
- Generation: Earlier lip-syncing methods required frame reselection or transcription with target emotions and generalized poorly to unseen faces.
- Generation: Recent approaches use consecutive frames and visual-quality or contrastive losses to model temporal correlation and generate convincing audio-to-video or text-to-video results.
- Detection: Detection methods compare audio and visual channels using MFCCs, mouth landmarks, audiovisual dissimilarity, or emotion-based correlations.A modality dissimilarity score aggregates audiovisual dissonance across video segments for real-fake classification.
- Detection: Performance decreases when videos contain limited mouth movement, and siamese detection can fail when only a few frames are manipulated.
4.3 Puppet-master
Puppet-master, or face reenactment, transfers a source actor’s facial gestures and movements onto a target identity. Methods progressed from 3D facial modeling to neural rendering, while detection uses visual artifacts and motion-related cues with restricted applicability.
- Generation: Puppet-master manipulates a target person’s facial expressions, eye movements, head movements, and mouth motion using a source actor’s gestures.
- Generation: Face2Face enabled real-time alteration of facial movements in generic RGB videos using a standard webcam and image rendering.
- Generation: Later approaches use SPADE residual blocks and multi-scale feature-map concatenation to preserve spatial information and handle large pose changes or exaggerated facial actions.
- Detection: Handcrafted detection features include eye and teeth artifacts, missing reflections, and optical-flow-related cues, but some methods require visible eyes or teeth.
4.4 Face Synthesis
Face synthesis generates photorealistic human faces that may not correspond to real people, driven by advances in deep generative models. Progressive training and StyleGAN-based designs improved resolution and visual quality, while detection remains difficult for AI-manipulated images.
- Generation: Face synthesis creates photorealistic human-face images that may or may not represent people existing in real life.
- Generation: Progressive GAN training addressed earlier high-resolution limitations by increasing output resolution and adding network layers during training.
- Generation: StyleGAN mapped latent vectors into an intermediate space that controlled visual features and reduced feature correlation through disentanglement.
- Generation: StyleGAN achieved 1024 × 1024 generated images with fine details, while StyleGAN2 further improved perceived quality by removing unwanted artifacts.
- Detection: Traditional image-forgery techniques generally perform poorly on AI-synthesized facial images, although CNNs trained on pixel co-occurrence matrices can detect manipulated content.
4.5 Facial Attribute Manipulation
Facial attribute manipulation alters selected facial regions while preserving irrelevant areas, using multi-domain generation methods and detection systems that increasingly rely on deep-learning fingerprints.
- Facial attribute editing changes specific appearance attributes, including eyeglasses, viewpoint, skin texture, age, and gender, while leaving irrelevant regions unchanged.
- StarGAN addresses the limitations of independently trained pairwise image translators by using one generator for translations among multiple domains.
- Detection methods include handcrafted pixel co-occurrence features, CNN classification, computationally costly approaches, and PRNU-based spatial-spectral score fusion.
- Existing facial attribute manipulation detectors often achieve performance close to 100%, largely because manipulated samples retain GAN fingerprint information.
- Removing GAN fingerprints while preserving image realism creates a challenge for high-performing facial attribute manipulation detection frameworks.
4.6 Audio Deepfakes Generation
Audio deepfakes use text-to-speech and voice-conversion systems to create realistic synthetic or identity-transferred speech. The review covers their generation pipelines and detection methods, while noting remaining realism and generalization challenges.
- AI-synthesized audio can clone a person’s voice and make it say content the person never said, producing realistic fake voices from text or utterances.
- Combining synthetic audio with visual manipulation can make deepfake videos more convincing, although synthesized speech still lacks some target-specific qualities such as expressiveness and emotion.
- TTS Voice Synthesis: TTS synthesizes a speaker’s natural-sounding voice from input text, while earlier concatenative systems combined recorded speech fragments into new utterances.
- TTS Voice Synthesis: WaveNet, Tacotron, and DeepVoice3 represent significant speech-synthesis developments for generating realistic speech from text.
- TTS Voice Synthesis: Tacotron uses an encoder, attention-based decoder, and post-processing network, but its multiple recurrent components make training economically inefficient.
- Voice Conversion: Voice conversion changes the perceived speaker identity while preserving the linguistic content of the source speech.
- Voice Conversion: Neural networks, vocoders, GANs, VAEs, and autoencoders support non-parallel voice conversion and can disentangle speaker information from linguistic content.
- Audio Deepfake Detection: Audio-deepfake detection research addresses synthetic speech and includes continual-learning methods designed to improve generalization to unseen spoofing manipulations.
4.7 Discussion
The review finds major advances in deepfake generation and strong detection performance, but removed manipulation fingerprints and subtle identity-preserving edits make reliable detection increasingly difficult.
- Generation: Deepfake generation has improved across face-swap, face-reenactment, lip-sync, face synthesis, and attribute manipulation, making fake content harder for people to distinguish from genuine content.
- Generation: Generation advances include unpaired training, one/few-shot learning, temporal discriminators, optical flow, blending networks, multiple losses, and perceptual loss.
- Generation: Synthetic voice quality has improved through voice adaptation, one/few-shot learning, self-attention, and cross-lingual voice transfer, but natural utterances under varied settings remain challenging.
- Detection: Most existing audiovisual detectors use deep learning and report robust performance close to 100%, often leveraging fingerprints and visible artifacts in manipulated samples.
- Detection: Face swaps are generally easier to detect because replacing the entire face often leaves artifacts, whereas expression swaps and lip-sync are more challenging.
- Detection: Detection approaches that remove forgery information while maintaining realism create new challenges even for high-performing attribute-manipulation detectors.
- Detection: Figure 10 categorizes detection techniques by manipulation type using colors for face-swap, face-reenactment, lip-sync, facial synthesis, and attribute manipulation, with asterisks marking deep-learning approaches.
4.8 Open Challenges in Deepfakes Generation
Deepfake generation remains constrained by data, computation, identity preservation, occlusion, illumination changes, temporal coherence, and the naturalness of synthetic audio.
- Generalization: Generative models require substantial training data and time, while obtaining enough samples for a specific victim and retraining for each identity remain difficult.
- Generalization: A generalized model is needed to support multiple unseen target identities or targets with few available training samples.
- Identity Leakage: Face reenactment can leak identity when target and driving identities differ substantially.
- Occlusions: Occlusions from hands, hair, glasses, or hidden facial regions can create inconsistent facial features in manipulated content.
- Temporal Coherence: Frame-wise generation can cause flickering and jitter, motivating temporal context, coherence losses, recurrent networks, or combined approaches.
- Audio Realism: Synthetic audio still needs more natural emotions, pauses, breathiness, and speaking pace.
4.9 Challenges in Deepfakes detection methods
Deepfake detectors face substantial robustness challenges because public datasets contain artifacts unlike manipulated internet content, while post-processing can erase forensic clues.
- Dataset quality: Public deepfake datasets often contain artifacts such as flickering, facial blurriness, over-smoothing, limited head motion, and identity leakage.These ambiguities make benchmark content differ from manipulated material encountered online.
- Dataset quality: High-quality combined audio-visual deepfake datasets remain limited, constraining evaluation across both modalities.
- Social media laundering: Social media laundering strips metadata, downsamples, and compresses content, removing forgery clues and increasing false-positive detection rates.Signal-level methods are especially affected; training and evaluation data should simulate these effects.
5 Deepfake Datasets
The paper reviews standard datasets for evaluating manipulated visual and audio content, emphasizing representative data and cross-dataset validation for measuring generalization.
- Dataset evaluation: Representative datasets and cross-dataset validation are important for assessing detection accuracy and generalization power.
- Visual datasets: UADFV contains 98 videos, but its low visual quality and obvious alterations make the fakes easy to detect.The dataset includes 49 real YouTube videos and 49 FakeApp-generated fake videos.
- Dataset comparisons: Tables 17 and 18 compare datasets used for deepfake and audio-fake detection.
- Audio datasets: LJSpeech provides 13,100 clips totaling 24 hours from one female speaker, whereas M-AILABS contains 999 hours and 32 minutes across nine languages and multiple speakers.
- Audio datasets: ASVspoof-2019 supports logical-access and physical-access audio spoofing analysis using known attacks for training/development and unknown attacks for evaluation.Its source corpus contains clips from 107 speakers, while the evaluation set contains 48 speakers.
6 Future Directions
Future work must address the escalating realism and scope of deepfake generation while making detection more robust to evolving attacks, sparse data, and multimodal complexity.
- 6.1 Creation: Deepfake generation is advancing toward high-quality, identity-agnostic, single-image, temporally consistent, and real-time manipulation.
- 6.1 Creation: Future synthetic media may combine target-specific audio-visual synthesis, natural speech, full-body pose changes, and convincing expressions.
- 6.2 Detection: Blockchain and smart contracts have been proposed to track manipulated information’s origin and history through stored hashes.
- 6.2 Detection: Current detectors are vulnerable to compression, noise, lighting variation, anti-forensic processing, and adversarial inputs.
- 6.2 Detection: Detection research should explore attack-aware defenses, knowledge-infused learning, and fusion of landmark, PPG, audio, anomaly, and signature-based features.
7 Conclusion
The survey reviews audio and video deepfake generation and detection, identifies their strengths, weaknesses, challenges, and future directions, and situates them within an ongoing competition.
- The survey comprehensively reviews audio and video deepfake generation and detection methods, including their strengths and weaknesses.
- The paper discusses current challenges and future directions for both deepfake creation and identification.
- The authors describe an ongoing cold war between deepfake generation and detection methods, where improvements in one challenge the other.