Source-linked AI summary
Ariadne's Thread of LipSync: Unraveling Forgeries via Inconsistency between Lip Motions and Head Poses
Tianyi She, Jiawei Liu, Weifeng Liu, Hanqing Zhao, Weiming Zhang, Kejiang Chen
TL;DR
LipSync forgeries can remain difficult to detect as generation methods improve synchronization and suppress visual artifacts, while source attribution is also needed for forensic accountability. LipDA models lip–head inconsistency for detection and temporal fingerprints for attribution; the paper reports superior performance on both tasks, including generalization to unseen techniques.
Problem
Existing defenses struggle with LipSync forgeries as generation methods improve lip synchronization and suppress visual artifacts, while binary detection alone does not identify the source generator.
Source: §1 · p. 1Method
LipDA jointly detects and attributes forgeries by modeling lip–head inconsistencies and temporal dynamics associated with generator fingerprints.
Source: §1 · p. 2Results
Takeaways & Limitations
Global physiological inconsistency between lip movements and head poses provides a forgery signal that supports a unified approach to detection and source attribution.
Source: §7 · p. 9Applicability to broader AI-generated image forgery domains remains unexplored.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Recent advances in LipSync generation technology have led to the creation of highly realistic videos, posing severe societal risks. However, existing defense strategies struggle against LipSync forgeries, as advanced LipSync generation methods not only achieve better lip synchronization but also eliminate visual artifacts. An important reason is that they overlook an inherent biological coupling between lip movements and head poses in natural speech videos. In this paper, we propose LipDA, a novel framework for joint LipSync Detection and Attribution, which takes advantage of the inconsistency between head and lip. For detection, the framework learns to quantify this discrepancy by contrasting lip and pose features from authentic versus forged videos. For attribution, our method is designed to capture the unique temporal dynamics and audio-visual synchronization patterns that act as the fingerprint of models, enabling source tracing. We conduct extensive experiments on two challenging LipSync datasets as well as our own proposed large-scale and multi-generator dataset. LipDA achieves over 97\% AUC in detection and 97.5\% accuracy in model attribution, significantly outperforming existing methods. Code and the proposed LipSync-A dataset are available at https://github.com/AnsonShe/LipDA.
1. Introduction
LipSync forgeries preserve a target’s identity while fabricating speech, and current defenses struggle as generators suppress the artifacts and synchronization errors they rely on. LipDA uses lip–head inconsistencies for detection and temporal fingerprints for attribution, with reported gains in generalization and robustness.
1. Introduction
- LipSync preserves the victim’s identity and visual context while fabricating speech never uttered, making detection and source attribution important for forensic accountability.The paper characterizes LipSync as a fact attack rather than an identity attack.Source: §1 · p. 1, Figure 1 · p. 1
- Existing general and LipSync-specific defenses struggle because localized manipulations dilute spatial cues, while newer generators mitigate visual artifacts and audio-visual mismatches.The paper also notes that binary real/fake classification does not identify the source generator.Source: §1 · p. 2
- Natural speech couples lip articulation with head dynamics, whereas LipSync generation can decouple them, creating measurable inconsistencies and generator-specific temporal fingerprints.These signals motivate the framework’s detection and source-attribution tasks.Source: §1 · p. 2
- LipDA detects forgeries by contrasting lip and pose features and attributes them using temporal dynamics and audio-visual synchronization patterns.The framework uses contrastive learning with cross-modal temporal modeling in a two-stage design.Source: §1 · p. 2
- LipDA significantly outperforms existing detectors and shows cross-domain generalization to unseen models and general deepfakes, with stability under visual and audio perturbations.The introduction also reports superior cross-dataset generalization and robustness against state-of-the-art methods.Source: §1 · p. 2
2. Related Work
LipSync generation has moved from earlier geometric and keypoint approaches toward audio-driven and diffusion-based methods that improve realism and motion quality. Detection research has shifted toward temporal and audio-visual cues, but these cues are increasingly mitigated, and source attribution remains under-addressed.
2.1. LipSync Generation
- Diffusion-based generators produce expressive motion and high-fidelity textures, reducing the jittering artifacts that earlier artifact-based detectors used.Earlier work included geometric transformations, keypoint-driven video methods, and audio-driven approaches such as Wav2Lip.Source: §2.1 · p. 2
2.2. LipSync Forgery Detection
- LipSync detection research moved from spatial and frequency artifacts toward temporal coherence, semantic consistency, and audio-visual synchronization cues.Examples include visual-only semantic or temporal methods and multimodal methods that compare audio and visual representations.Source: §2.2 · p. 3
- High-fidelity textures and synchronized lip motions increasingly mitigate the artifacts and synchronization errors that prior detection strategies rely on.The related-work discussion identifies diminishing returns for these strategies as generators improve.Source: §2.2 · p. 3
- Existing detection literature remains largely focused on binary classification, leaving source-model attribution for forensic accountability under-addressed.The paper identifies source attribution as a critical task beyond real-versus-fake classification.Source: §2.2 · p. 3
3. Analysis of LipSync Forgery Signals
The analysis identifies disrupted lip–head coordination as a signal separating natural speech from LipSync forgeries and proposes temporal motion patterns as model-family fingerprints. It also introduces LipSync-A, a labeled multi-generator dataset for attribution research.
3.1. Inter-Class Signal: Real vs. Fake
- Video-driven forgeries transplant a driver’s motion and head pose, while audio-driven approaches can separate lip synthesis from head-pose generation, disrupting natural lip–head coupling.Video-driven patterns can create subtle temporal artifacts; audio-driven methods may retain static pose or generate pose independently.Source: Figure 2 · p. 3, §3.1 · p. 3
- Video-driven forgeries show reduced facial action-unit intensity, whereas audio-driven forgeries exhibit erratic AU17 peaks despite mimicking authentic lip dynamics.The analysis uses AU12 and AU17 as representative action units.Source: §3.1 · p. 3, Figure 3 · p. 4
3.2. Intra-Class Signal: Model Fingerprints
- Pose-feature embeddings from different model families form well-separated t-SNE clusters, supporting inter-frame head-motion dynamics as a basis for attribution.The analysis uses disjoint source identities and diverse driving signals when sampling forged videos.Source: §3.2 · p. 4, Figure 4 · p. 4
3.3. Dataset construction
- LipSync-A provides 16,000 synthetic videos from 15 generators spanning 7 architectures, with source-generator labels for attribution evaluation.The paper presents it as a large-scale dataset explicitly designed for LipSync forensics.Source: §3.3 · p. 4
- Table 1 compares LipSync datasets by generator architectures and forged-video sample counts.Its caption defines #Gens. as generator architectures and #Fakes as forged video samples.Source: Table 1 · p. 4
4. Methodology
LipDA is a unified two-stage framework for LipSync detection and source attribution. It uses lip-pose inconsistency for detection and audio-visual synchronization and temporal motion patterns for attribution.
4. Methodology
- LipDA combines contrastive lip-pose inconsistency detection with audio-visual synchronization and temporal-motion modeling for source attribution.Source: §4 · p. 4
4.1. Problem Formulation
- Detection is framed as binary classification, while attribution identifies the generating model from a candidate set.Source: §4.1 · p. 4
4.2. Stage I: Pose-Lip Contrastive Optimization
- Stage I aligns lip and pose embeddings for authentic videos and separates forged pairs with a margin-based contrastive loss, alongside a classifier on concatenated features.The final Stage I objective jointly optimizes the contrastive and binary-classification losses.Source: §4.2 · p. 5
4.3. Stage II: Cross-Modal Temporal Modeling
- Stage II uses keypoint sequences to model temporal motion patterns and lip ROIs with audio features to estimate synchronization.TDM and MSM are trained independently from Stage I and used concurrently during inference.Source: §4.3 · pp. 5–6
- The attribution classifier concatenates temporal and synchronization features into a fingerprint and trains on forged samples with their source-model labels.Source: §4.3 · p. 6
4.4. Inference
- At inference, the detection and attribution classifiers jointly predict whether a video is real or fake and, for a forgery, its source model.Source: §4.4 · p. 6
5. Experiment
The experiments evaluate LipDA’s detection and attribution performance, generalization to unseen forgeries, robustness, and design choices. LipDA outperforms comparison methods across the reported tasks and datasets.
5.1. Experimental Setup
- Evaluation uses LipSync-A for training and in-domain detection, with zero-shot tests on AVLips and TalkHeadBench and additional cross-manipulation and perturbation tests.The reported metrics include video-level ACC, AUC, F1-score, AP, FPR, and FNR.Source: §5.1 · p. 6
5.2. Comparison to Existing Methods
- +7.46% ACC and +7.97% AUC over SpeechForensics in the audio-visual setting on three LipSync datasets.The comparison reports that LipDA significantly outperforms existing baselines in both video-only and audio-visual settings.Source: §5.2 · p. 6, Table 2 · p. 7
- Over 12 percentage points average F1-score improvement over TALL, with robust performance on the most challenging Transformer-based forgeries.SpeechForensics and AVAD show extremely low F1-scores on GAN and Transformer categories.Source: §5.2 · p. 7, Table 3 · p. 7
5.3. Generalizability to Unseen Forgery
- 92.76% ACC and 1.54% FNR on Celeb-DF despite training exclusively on LipSync forgeries; LipDA also achieves the highest ACC and F1 scores on unseen LipSync samples.The results cover cross-manipulation generalization to identity-swap manipulations and unseen LipSync techniques.Source: §5.3 · p. 7, Table 4 · p. 7, Table 5 · p. 8
5.4. Robustness Evaluation
- LipDA remains stable under texture-level corruptions and common audio resampling or time shifts, while direct audio corruption affects it.Baselines deteriorate under Gaussian blur, compression, and contrast adjustments; Fig. 6 presents robustness against unseen corruptions.Source: §5.4 · p. 8, Figure 6 · p. 8
5.5. Ablation Study and Analysis
- Detection accuracy improves with video length, saturating at approximately 8 seconds under Non-random sampling; window length trades temporal information against redundant motion.Fig. 7 plots detection accuracy by sliding-window length and video length, comparing Random and Non-random sampling.Source: §5.5 · p. 8, Figure 7 · p. 8
- Removing alignment loss drops ACC by 17.3%, while replacing ResNet-18 with CLIP-ViT reduces performance by 12.0%.Removing the lip component causes a detection performance collapse; LSTM is superior for pose features.Source: §5.5 · p. 8, Table 6 · p. 8
- Unseen real-sample lip and pose embeddings intermingle, whereas fake-sample embeddings form distinct, separable clusters.Fig. 8 visualizes the Stage I feature space for unseen real and fake test samples.Source: §5.5 · pp. 8–9, Figure 8 · p. 9
6. Discussion
LipDA shifts detection toward global physiological coordination between lip movement and head pose. The discussion identifies scope, audio-sensitivity, and visibility-related limits.
6. Discussion
- Global lip–pose coordination offers a forgery signal beyond local artifacts or explicit audio-visual mismatches.The authors argue that modeling high-dimensional coordination of head motion with speech is substantially harder for an adversary than optimizing lip-audio synchronization loss.Source: §6 · p. 9
- Texture-degradation robustness supports the claim that LipDA uses physiological constraints rather than brittle visual cues.Source: §6 · p. 9
- Applicability beyond AI-generated image forgeries remains unexplored; attribution degrades under some audio perturbations, and detection degrades with stationary speakers, downward gaze, or prolonged occlusion.Source: §6 · p. 9
7. Conclusion
The conclusion presents global inconsistency between lip movements and head poses as a forgery signal. LipDA models that coordination for detection and attribution, with reported generalization and robustness.
7. Conclusion
- Global physiological inconsistency between lip movements and head poses is presented as a novel, robust forgery signal.Source: §7 · p. 9
- LipDA is a two-stage framework that models this coordination to unify forgery detection and source attribution.Source: §7 · p. 9
- The authors report superior performance on both tasks, generalization to unseen techniques, and robustness to significant compression and blurring.Source: §7 · p. 9
A. Additional Dataset Details
The supplementary material details LipSync-A’s data and generator coverage, explains the proposed physiological signal, and describes evaluation datasets and attribution baselines.
A. Additional Dataset Details
- LipSync-A covers audio- and video-driven generation, with about 16,000 samples from 15 techniques and identity- and driving-signal-disjoint train, validation, and test splits.Its source data include LaPa and HDTF identity inputs and VFHQ and VCTK driving signals; the evaluation split contains 3,926 training, 841 validation, and 842 testing samples.Source: §A.2 · p. 13, A. Additional Dataset Details · p. 13, §A.1 · p. 13
B.1. Detection Motivation
- The authors argue that lip articulation is largely deterministic while head pose is high-dimensional and one-to-many, making their joint distribution difficult for generators to model.They describe common generators as decomposing the modalities and thereby losing correlation information between phoneme articulation and subtle head movements.Source: §B.1 · p. 14
B.2. Attribution Motivation
- The paper introduces a five-family taxonomy for audio-driven LipSync forgeries to support attribution aligned with generative mechanisms.The listed families are GAN-based, diffusion-based, Transformer-based, VAE-based, and statistical models.Source: §B.2 · pp. 14–15
B.3. Inter-Class Signal Motivation
- For video-driven forgeries, transferring a driver’s motion to another identity can shift frames outside the source’s natural manifold and produce localized temporal artifacts.Source: §B.3 · p. 15
Audio-Driven Paradigm
- Audio-driven approaches can leave lip–pose desynchronization through static inherited poses, pose anchoring, or independently learned lip and pose streams.The described mechanisms include reconstruction-based, diffusion-based, and independent-learning methods.Source: Audio-Driven Paradigm · p. 15
C.1. Evaluation Dataset Selection
- The evaluation dataset selection includes AVLips, TalkHeadBench, and Celeb-DF, with differing real/forged-video and audio availability.TalkHeadBench’s test set pairs 476 real videos with 448 forged videos that include audio; Celeb-DF evaluation uses its standard test set of 980 videos.Source: §C.1 · pp. 15–16
C.2. Rationale for Baseline Selection
- The comparison covers visual-only approaches to lip cues, artifacts, temporal inconsistency, and frequency cues, alongside multimodal asynchrony and alignment methods.The listed multimodal baselines also include SpeechForensics and DFD-FCG.Source: §C.2 · p. 16
C.3. Baseline Adjustments for Attribution Task
- For attribution, selected baselines’ output layers were adapted from four-way to five-way classification and trained with cross-entropy loss.The baselines include TALL, AVAD, SpeechForensics, and AVH-align; most visual-only detectors were considered ill-suited to multi-class attribution.Source: §C.3 · p. 16
C.4. Evaluation Protocol of Main Results
The evaluation protocol uses LipSync-A for training and in-domain assessment, with additional tests on unseen datasets and generators. The accompanying baseline analysis links frequency-detector failures to the localized, temporally smoothed nature of LipSync forgeries.
C.4. Evaluation Protocol of Main Results
- LipDA trains and undergoes in-domain detection evaluation on LipSync-A, then faces zero-shot tests on AVLips, TalkHeadBench, unseen generators, and cross-manipulation Celeb-DF.The training split contains 1,614 authentic and 2,312 forged videos from 10 generators spanning seven architectural paradigms.Source: §C.4 · p. 16
- The 15 baselines use released weights or are fine-tuned on the same LipSync-A split, with source-benchmark re-evaluation checking that fine-tuning preserves their capabilities.The reproduced results closely track originally reported accuracies on FF++ and AVLips.Source: §C.4 · p. 17
- On 160,532 test frames, NPR and FreqNet show 99.8% false-negative rates and rank real frames as more suspicious (mean score: real 0.028 vs. fake 0.002).The paper attributes the inverted ROC ordering to a mismatch between detectors trained on high-frequency full-face artifacts and LipSync's localized, temporally smoothed edits.Source: §C.4 · p. 17
D. Additional Robustness Study
- The additional robustness study evaluates LipDA and baselines under visual and audio perturbations intended to simulate post-processing and re-compression.The supplied protocol describes this study's scope but not its outcome.Source: D. Additional Robustness Study · p. 17
D.1. Visual Robustness
LipDA is tested against visual and audio corruptions and under degraded landmark extraction, alongside an open-set attribution discussion. The reported results indicate resilience to many perturbations, while identifying conditions that weaken the available cues.
D.1. Visual Robustness
- At Level-3 visual corruption, LipDA reaches 99.11% average AUC, 13.58 percentage points above LipForensics, and remains above 99.5% AUC under compression, blur, and pixelation.The visual study tests seven perturbation types at five intensity levels.Source: §D.1 · pp. 17, 19, Table 8 · p. 17, Table 9 · p. 18
D.2. Audio Robustness
- Under audio perturbations, LipDA retains AUC above 97.3% even for heavy additive noise and substantial pitch distortion, while resampling and reverberation also leave performance exceptionally high.The study applies 12 audio corruptions and reports results on AVLips.Source: §D.2 · p. 19, Table 10 · p. 19
D.3. Landmark Extraction Robustness
- Under Level-5 visual perturbations, JPEG compression, block-wise occlusion, and contrast keep landmark failure rates below 9% and AUC above 96%; Gaussian noise raises landmark failure to 34.7%, yet AUC remains 90.98%.The sliding-window method retains 86.57% valid windows under Gaussian noise at Level 5; saturation also harms lip-encoder appearance features, so AUC decline does not track landmark failure alone.Source: Table 11 · p. 19, §D.3 · p. 19
E.1. Open-set Attribution
- Family-level attribution maps unseen generators to known architectural families, but entirely novel paradigms still require explicit open-set recognition or out-of-distribution detection.Held-out OmniSync and KDTalker are assigned to Diffusion at 90.9% and 97.3% accuracy, while EAT is assigned to Transformer at 99.0%.Source: §E.1 · p. 20
E.2. Connection to Generative Fingerprint Methods
The paper contrasts LipDA's temporal and cross-modal fingerprints with prior image-level, spatial signatures. Evaluations report detection across generator families, fine-grained model attribution, and zero-shot transfer to full-video and unseen generators.
E.2. Connection to Generative Fingerprint Methods
- Unlike image-level fingerprint methods based on spatial signatures, LipDA uses temporal head-motion dynamics and lip-audio synchronization to characterize generator families.The paper argues that static pixel signatures are not directly transferable because LipSync preserves authentic identity and visual context while modifying localized lip dynamics.Source: §E.2 · p. 20
F.1. Per-Generator Detection on LipSync-A
- On LipSync-A, visual-only LipDA averages 94.6% accuracy and 97.4% AUC across ten generators, with slightly lower AUC on SadTalker linked to its near-static source-frame head poses.The result is reported against a shared pool of 346 authentic samples.Source: §F.1 · p. 20, Table 12 · p. 21
F.2. Fine-Grained 15-Way Model-Level Attribution
- In the complementary 15-way test, LipDA reaches 82.67% accuracy and 82.04% macro-F1; confusion is most pronounced among keypoint-driven models, while Figure 11's failure cases are shared with evaluated baselines.The 15-way target differs from the 5-way family-level protocol and is presented as a stress test, not a directly comparable refinement.Source: §F.2 · p. 21, Table 13 · p. 21, Figure 11 · p. 22
F.3. Generalization to Full-Video Generation
- Across five unseen generators, LipDA attains 87.0% average detection accuracy, including 88.2% on full-video generator Wan-S2V, and correctly assigns LatentSync to Diffusion at 97.5% accuracy.These zero-shot findings support transfer of the detection cue and attribution fingerprints beyond the LipSync editing pipelines used in training.Source: Table 14 · p. 22, §F.3 · p. 22
F.4. Bad Case Analysis
The analysis identifies shared failure conditions for detectors using visible lip dynamics and head motion, then examines attribution components and qualitative patterns in LipSync-A. The reported attention and sample comparisons illustrate the local and global motion cues and generator diversity considered by LipDA.
F.4. Bad Case Analysis
- LipDA's shared failure conditions include nearly stationary heads under complex camera motion, persistent downward gaze that obscures the lips, and prolonged occlusion that destabilizes landmarks.These cases mark an operational limit for detection based on visible lip dynamics and well-defined head motion; the paper points to confidence-aware inference and complementary upper-face or full-body cues as future directions.Source: §F.4 · p. 22
F.5. Attribution Ablation
- The full attribution model reaches 97.50% accuracy by combining MSM synchronization patterns with TDM temporal cues; MSM alone scores 92.92% accuracy and TDM alone 74.78%.The ablation reports averages across five generator families.Source: Table 15 · p. 23, §F.5 · p. 23
F.6. Dataset Case Analysis
- Attention maps show broad facial context for the pose encoder and localized lip motion for the lip encoder, while dataset comparisons highlight generator-specific differences in articulation, motion amplitude, and head-pose dynamics.Figures 13 and 14 compare forgeries from the same source identity and driving video and samples from five audio-driven generator families; Figure 15 visualizes LipSync-A samples.Source: §F.6 · p. 23, Figure 13 · p. 23, Figure 14 · p. 24