Source-linked AI summary
Audio-Driven Adversarial Defense for 3D Talking Face Generation with totally Visual Fidelity Preservation
Rui-Qing Sun, Chen-Hao Cui, Hui-Yang Zhao, Tian Lan, Zhijing Wu, Xian-Ling Mao
TL;DR
Audio-driven 3D talking face generation can turn monocular reference videos into reusable, speech-driven 3D portraits, while visual defenses may damage facial quality and weaken under common transformations. The paper introduces psychoacoustically masked audio perturbations to interfere with facial animation without directly altering portrait appearance. Experiments show degraded talking-face generation with favorable perceptual quality.
Problem
Audio-driven 3D talking face generation enables reusable 3D portrait reconstruction and arbitrary-speech animation, while visual defenses are limited by perceptual distortion and transformation fragility.
Method
The method shifts protection to speech audio by placing adversarial perturbations in psychoacoustically masked frequency regions.
Results
The proposed method effectively degrades audio-driven 3D talking face generation while preserving favorable perceptual quality.
Takeaways & Limitations
Psychoacoustically guided audio perturbations provide a practical direction for privacy-preserving portrait protection when visual fidelity is critical.
Takeaways & Limitations
Visual-domain protection remains constrained by facial distortion and weakened effectiveness under resizing, resampling, compression, and related transformations.
Abstract
from arXiv · showhide
The rapid development of generative portrait models has raised growing concerns about privacy leakage and identity misuse. In particular, audio-driven 3D talking face generation can reconstruct a reusable 3D portrait of a target person from a monocular video and animate it with arbitrary speech, making realistic identity impersonation alarmingly practical. Existing proactive defenses mainly operate in the visual domain by injecting subtle perturbations into acial regions to disrupt identity acquisition. However, such perturbations often compromise visual quality due to the strong structural priors and social sensitivity of human faces, and are easily weakened by common real-world transformations such as resizing. To overcome these limitations, we propose an imperceptible audio defense for audio-driven 3D talking face generation by shifting protection from the visual modality to the audio modality. Specifically,we exploit psychoacoustic masking to hide protective perturbations within perceptually masked frequency regions of the speech signal, thereby reducing perceptual distortion while suppressing reliable facial animation. Extensive experiments demonstrate that the proposed method effectively degrades 3D talking face generation while preserving favorable perceptual quality. These findings highlight psychoacoustically guided audio perturbations as a practical and promising direction for privacy-preserving portrait protection.
1 Introduction
Audio-driven 3D talking face generation creates serious identity-misuse risks, while visual defenses can harm facial fidelity and weaken under real-world transformations. The paper therefore shifts protection to psychoacoustically masked audio perturbations, which suppress facial animation while preserving perceptual quality.
- Audio-driven 3D talking face generation reconstructs personalized 3D portraits from monocular video and animates them with arbitrary speech.
- Visual-domain defenses perturb facial regions to disrupt identity extraction, landmark estimation, or portrait reconstruction.
- Visual perturbations can reduce facial quality, introduce artifacts, and lose effectiveness after resizing, resampling, or compression.
- The proposed defense shifts intervention to driving audio, using psychoacoustic masking to hide perturbations in perceptually masked frequency regions.
- Extensive experiments show effective degradation of 3D talking face generation while preserving favorable perceptual quality.
2 Related Works
3D-field talking face generation improves identity consistency and controllability but increases privacy risks because reference videos can yield reusable portraits driven by arbitrary speech. Existing visual defenses remain vulnerable to visual distortion, transformations, and preprocessing, motivating an audio-based cross-modal defense.
- NeRF and 3D Gaussian Splatting reconstruct subject-specific 3D portraits from monocular videos with stronger identity consistency and view-consistent rendering.
- Current 3D-field talking face models can create controllable digital portraits and realistic videos from collected reference videos and arbitrary speech.
- Visual defenses may visibly damage faces and are fragile under resizing, resampling, compression, and preprocessing that can partially restore or purify perturbations.
- This work shifts protection from facial regions to audio, treating driving speech as an intervention point for preserving portrait appearance while disrupting facial animation.
- Unlike pure speech or speaker defenses, this setting must disrupt the audio-to-geometry mapping while remaining imperceptible to listeners.
3 Preliminaries
3D-field talking-face generation reconstructs dynamic 3D portraits by combining reference-derived identity, pose, and expression priors with audio-controlled facial motion. The defense setting targets public audio reuse, applying psychoacoustic constraints while requiring resilience to common transmission distortions.
- 3.1 Audio-Driven Talking Face Generation in 3D Fields: 3D-field TFG represents the human head as a dynamic 3D structure and renders audio-driven facial motion into 2D images.The pipeline maps acoustic features to a 3D spatial representation before rendering.
- 3.1 Audio-Driven Talking Face Generation in 3D Fields: The reference video is disentangled into identity, pose, and expression priors, while driving audio modulates 3D spatial samples through a latent lip space.Training-only extraction operations are distinguished from audio-visual encoding and neural rendering used in both training and inference.
- 3.1 Audio-Driven Talking Face Generation in 3D Fields: Efficient 3D-field TFG frameworks include NeRF-based volumetric fields, 3DGS-based point fields, and hybrid fields combining 3D morphable models with neural feature volumes.Point-based models can predict Gaussian-center offsets or modulate a canonical deformation field from audio features.
- 3.1 Audio-Driven Talking Face Generation in 3D Fields: Audio-to-geometry mapping transforms a static speaker-specific field into an audio-synchronized dynamic field.The formulation identifies F_static as the static spatial prior and F_sync as the audio-synced dynamic field.
- 3.2 Principles of Psychoacoustic Masking: Psychoacoustic protection keeps adversarial audio noise below hearing thresholds by exploiting critical-band masking, the absolute threshold of hearing, and a global masking threshold.The global threshold aggregates tonal and non-tonal maskers, frequency spreading, and the absolute threshold in the intensity domain.
- 3.3 Threat Model and Defense Objective: The defense embeds perturbations in an inaudible zone subject to PSD(δ) ≤ θ(j), while the threat model requires effectiveness against black-box reuse after compression and resampling.The adversary harvests publicly shared audio to drive a 3D-field TFG model; standard lossy compression and resampling are treated as channel distortions.
4 Methodology
The method shifts proactive defense from visually sensitive facial pixels to psychoacoustically constrained audio, then optimizes that audio to induce incorrect mouth geometry in a frozen talking-face pipeline. Its coupled construction and optimization objectives target semantic disruption while preserving perceptual and visual quality.
- Audio Perturbation Construction: The framework constructs a frequency-domain perturbation within a psychoacoustic budget and reconstructs it as an adversarial waveform.The perturbation amplitude is bounded by the masking-derived capacity map, and inverse STFT produces the time-domain waveform.
- Core Semantic Conflict Establishment: The perturbed waveform is passed through frozen audio features and a pretrained 3D-field renderer conditioned on the target identity.The resulting frames are analyzed with a differentiable landmark detector to support optimization against rendered mouth geometry.
- Core Semantic Conflict Establishment: The mouth deconstruction objective encourages rendered mouths to collapse toward a closed state despite speech that should produce open-mouth articulation.This creates an inconsistent audio-to-geometry mapping and weakens stable lip-motion learning from protected data.
- Optimization Objectives: Four complementary objectives jointly enforce attack effectiveness, spectral energy preservation, psychoacoustic masking, and temporal-frequency smoothness.Spectral regularization discourages signal-energy suppression, while masking and smoothness terms reduce perceptual violations and unnatural artifacts.
- Asynchronous Dual-Loop Optimization: The primary optimization loop drives the perturbation toward semantic attack directions while retaining reasonable spectral behavior.It evaluates rendered frames with the landmark detector and combines mouth, spectral-energy, and smoothness terms.
- Discussion: The overall design attacks the cross-modal driving signal rather than facial pixels, aiming for visually lossless protection against personalized 3D talking-face generation.The upper branch maintains psychoacoustic safety, while the lower branch enforces incorrect mouth dynamics through landmark supervision.
5 Experiments
Experiments evaluate the defense on an 11-identity HDTF subset against visual perturbation and audio-removal baselines using mouth-region generation and audio-perceptual metrics. The method preserves audio perceptual quality while maintaining effective disruption and visual fidelity, whereas stronger disruption from silencer comes with degraded appearance.
- Experimental Settings and Baselines: The evaluation uses an 11-identity HDTF subset, comparing the method with antifake and silencer across generation-quality and perceptual-audio metrics.M-LMD, M-SSIM, and M-PSNR assess mouth-region generation, while CDPAM measures defended-audio perceptual distance.
- Quantitative Results: Lower CDPAM than antifake indicates substantially less perceptible audio perturbation, while M-LMD, M-SSIM, and M-PSNR show comparable defense effectiveness.The reported comparison supports interference with downstream talking-face generation without noticeably compromising perceptual quality.
- Quantitative Results: Silencer achieves the strongest M-LMD, M-SSIM, and M-PSNR disruption, but directly degrading visual input substantially worsens visual appearance and user experience.Its stronger disruption metrics therefore arise alongside severely compromised protected content.
- Quantitative Results: The proposed method provides effective defense while preserving the natural appearance of shared visual content, yielding a more practical defense-quality trade-off.This trade-off is presented as preferable for deployment where benign users should retain an acceptable viewing experience.
- Qualitative Results: Visual-domain defenses introduce grainy noise, color inconsistency, and corrupted textures, whereas the proposed method preserves clean facial structure, texture, and color across identities.The qualitative comparison describes the proposed protection as visually lossless and perceptually indistinguishable from clean data for human observers.
- Qualitative Results: Despite high visual fidelity, the method still disrupts speech-driven synthesis, supporting protection that does not require sacrificing visual quality.The stated deployment target is user-uploaded content that remains visually acceptable while resisting unauthorized talking-face generation.
17 end
The optimization procedure combines main and psychoacoustic objectives, with a secondary refinement stage after the primary optimization reaches its target. The listed losses jointly encode attack, energy, smoothness, masking, and regularization terms.
- 17 end: The primary objective minimizes L_main = L_atk + λ_engL_eng + λ_smoothL_smooth + λ_maskL_mask while updating B.The expression explicitly combines attack, energy, temporal-smoothness, and masking losses.
- 17 end: After the primary stage, secondary refinement minimizes L_psy = λ_maskL_mask + λ_smoothL_smooth + λ_reg∥B∥1 while updating B.This stage retains masking and smoothness terms and adds an L1 regularizer.
A.1 Ablation Experiments
The ablation study shows that temporal smoothing, band constraints, and psychoacoustic masking progressively improve adversarial-audio perceptual quality, with masking providing the best reported result.
- Ablation Experiments: CDPAM decreases from 0.1379 without constraints to 0.0941 with temporal smoothing, 0.0817 with an added band constraint, and 0.0634 with psychoacoustic masking.The progression indicates complementary contributions from smoothing, frequency-region restriction, and masking.
- Ablation Experiments: Psychoacoustic masking achieves the best CDPAM score of 0.0634, demonstrating its critical role in reducing perceptually noticeable noise.The passage attributes the final improvement to constraining perturbations within perceptually masked regions.