Source-linked AI summary
Proactive Detection of Voice Cloning with Localized Watermarking
Robin San Roman, Pierre Fernandez, Alexandre Défossez, Teddy Furon, Tuan Tran, Hady Elsahar
TL;DR
Voice cloning and increasingly realistic speech synthesis create a need for reliable audio authenticity detection. AudioSeal uses jointly trained generator and detector networks with localized detection and perceptual objectives. It achieves robust, imperceptible, sample-level watermark detection with detection up to two orders of magnitude faster than existing models.
Problem
As synthesized voices become difficult to distinguish from real speech, existing passive detection and watermarking methods have limitations in reliability, localization, or speed.
Method
AudioSeal jointly trains a watermark generator and localized detector, using perceptual and localization objectives with augmentations and optional multi-bit attribution.
Results
AudioSeal achieves state-of-the-art robustness and imperceptibility, sample-level localization, and detection up to two orders of magnitude faster than existing models.
Takeaways & Limitations
AudioSeal supports proactive detection, localization, and attribution of AI-generated speech for real-time and large-scale traceability.
Takeaways & Limitations
For adversarial watermark attacks, the training code and awareness that audio is watermarked can be public, but detector weights should remain confidential.
Abstract
from arXiv · showhide
In the rapidly evolving field of speech generative models, there is a pressing need to ensure audio authenticity against the risks of voice cloning. We present AudioSeal, the first audio watermarking technique designed specifically for localized detection of AI-generated speech. AudioSeal employs a generator/detector architecture trained jointly with a localization loss to enable localized watermark detection up to the sample level, and a novel perceptual loss inspired by auditory masking, that enables AudioSeal to achieve better imperceptibility. AudioSeal achieves state-of-the-art performance in terms of robustness to real life audio manipulations and imperceptibility based on automatic and human evaluation metrics. Additionally, AudioSeal is designed with a fast, single-pass detector, that significantly surpasses existing models in speed - achieving detection up to two orders of magnitude faster, making it ideal for large-scale and real-time applications.
1. Introduction
AudioSeal addresses weaknesses in passive detection and conventional watermarking by enabling localized, imperceptible, and efficient detection of AI-generated speech. It combines joint generator–detector training with strong robustness, sample-level localization, model attribution, and substantially faster detection.
- Motivation: Passive classifiers can become unreliable as synthetic and authentic speech converge, while existing watermarking methods may lack localization, tolerate short spans poorly, and incur slow detection.WavMark’s synchronization bits also reduce message capacity by 31%.
- AudioSeal: AudioSeal jointly trains a generator and detector to embed watermarks, localize their presence at each sample, and attribute audio to a specific model or version.The detector is trained with masked sections of audio, while multi-bit messages support attribution without affecting the detection signal.
- Results: AudioSeal achieves near-perfect detection across diverse audio edits, sample-level localization, faster detection than WavMark, and near-perfect attribution among 1,000 models.Its single-pass detector outputs logits at every time-step, avoiding synchronization procedures used by earlier methods.
- AudioSeal: AudioSeal introduces an auditory-masking-inspired perceptual loss to improve watermark imperceptibility while retaining detection performance.The contribution targets better perceptual quality of the embedded signal.
2. Related Work
Prior work detects synthetic speech either passively through classifiers or actively through watermarks, but existing watermarking methods largely lack localized detection and can be inefficient or vulnerable to edits.
- Synthetic speech detection: Passive detection classifies audio as natural or synthesized without altering it, but may fail as generative models make the two harder to distinguish.
- Imperceptible watermarking: Watermarking embeds imperceptible, detectable signals; zero-bit schemes detect presence, while multi-bit schemes encode attribution messages.
- Imperceptible watermarking: Deep-learning audio watermarking predominantly uses generator-decoder frameworks for multi-bit payloads, whereas relatively few works explore zero-bit detection.
- Imperceptible watermarking: WavMark hides 32 bits in one-second segments and detects them by sliding windows, using synchronization bits before decoding the payload.
3. Method
AudioSeal jointly trains a watermark generator and sample-wise detector, combining perceptual, localization, and robustness objectives to support detection, localization, and attribution.
- 3.1. Training pipeline: The method jointly trains a generator that adds a watermark and a detector that outputs local detection logits, optimizing perceptual distortion and detection objectives.
- 3.1. Training pipeline: Random masking with original audio, silence, or other audio trains the detector to localize watermark presence at individual samples.
- 3.1. Training pipeline: Signal augmentations including filtering, noise, echo, resampling, and speed changes improve robustness to audio editing.
- 3.1. Training pipeline: The detector produces a soft decision at every time step, enabling whole-audio detection by averaging scores and localization by thresholding sample-wise outputs.
- 3.2. Losses: TF-Loudness uses frequency-band and time-window loudness differences with softmax weighting, exploiting auditory masking to improve watermark imperceptibility.
- 3.3. Multi-bit watermarking: Multi-bit extension embeds learnable message representations and adds per-bit detector outputs, with decoding loss applied where the watermark is present.
4. Audio/Speech Quality
AudioSeal prioritizes perceptual speech quality over signal-to-noise metrics and achieves equal or better perceptual quality than WavMark, with human ratings close to ground truth.
- AudioSeal achieves the same or better perceptual quality than WavMark according to the reported audio-quality metrics.
- AudioSeal consistently outperforms its SI-SNR-oriented comparison on PESQ, STOI, and ViSQOL, reflecting its perceptual-quality objective.
- Human MUSHRA ratings for AudioSeal samples are very close to the ground-truth average score of 80.49.
5. Experiments and Evaluation
AudioSeal is evaluated for detection, robustness, localization, attribution, and efficiency against passive classifiers and WavMark. It achieves strong robustness and precise localization while substantially reducing detection time.
- AudioSeal achieves perfect detection across Voicebox samples, whereas passive classification degrades as generated speech quality improves.
- AudioSeal achieves an average AUC of 0.97 versus 0.84 for WavMark across audio edits.
- AudioSeal reaches 0.99 IoU when one second of speech is manipulated, compared with 0.35 for WavMark.
- AudioSeal localizes manipulated speech at approximately 1/16k-second sample resolution, while WavMark operates at one-second resolution.
- Attribution improves overall accuracy through decoupled detection and attribution, but incurs occasional false attributions.
- AudioSeal is 14x faster for generation and up to 485x faster for detection than WavMark.
6. Adversarial Watermark Removal
The adversarial evaluation examines watermark removal under white-box, semi-black-box, and black-box knowledge assumptions. Attacks become more effective as watermarking information is disclosed, while detector confidentiality limits their effectiveness.
- The threat models differ by adversarial knowledge, from detector access in white-box attacks to API-based surrogate training in black-box attacks.
- White-box attacks increase detection error by around 80% while maintaining high audio quality with PESQ > 4.
- Semi-black-box and black-box attacks require substantial audio-quality degradation to increase detection error by 50%.
- The analysis concludes that detector weights should remain confidential to preserve watermarking effectiveness against deliberate attacks.
7. Conclusion
AudioSeal redesigns audio watermarking for sample-level detection, localization, and attribution of AI-generated speech. Its jointly trained architecture combines perceptual and detection objectives to provide robust, precise, and efficient watermarking.
- AudioSeal combines a watermark generator and detector with loudness, differentiable augmentation, and masked sample-level detection losses.
- AudioSeal achieves robust detection under audio edits, precise localization, and orders-of-magnitude faster runtime than synchronization-based methods.
- Adversarial analysis indicates that detector weights must remain private for watermarking to remain an effective mitigation.
Impact Statement
The impact discussion situates AudioSeal within broader synthetic-speech and audio-generation research while acknowledging risks associated with watermarking and AI-generated media. It emphasizes the need for responsible governance alongside detectability.
- Watermarking can support transparency and traceability but may also enable surveillance, whistle-blower identification, and copyright enforcement.
- Zero-shot speech models can synthesize personalized speech from short recordings and perform speech infilling or translation while preserving vocal style.
- Prior audio-generation research has explored autoregressive, signal-to-noise, spectrogram, and adversarial objectives for audio quality.
- Synchronization is central to watermark extraction but can be disrupted by time and pitch scaling, motivating alternative synchronization techniques.
B. False Positive Rates - Theory and Practice
The theoretical false-positive analysis assumes independently random extracted bits, but empirical WavMark outputs violate that assumption, making theoretical FPR estimates unreliable in practice.
- Theory: Under genuine audio, independently random extracted bits make the matching count binomial, so thresholding it yields a theoretical false-positive rate.The test flags audio when the number of matching bits reaches threshold τ, with FPR defined as the probability that this count exceeds τ.
- Practice: For WavMark’s 32-bit detector, genuine-sample soft scores center near 0.38 rather than the expected 0.5, biasing decisions toward bit 0.The empirical study extracts soft bits from 10k genuine one-second samples.
- Practice: Theoretical and empirical FPRs diverge when the hidden message is all zeros, invalidating theoretical detection-rate guarantees in practice.Figure 8 compares the two FPRs across threshold values τ.
C.2. Another architecture
AudioSeal’s performance remains consistent when its EnCodec-based architecture is replaced with the slower, heavier DPRNN architecture, while mixed speech–music signals are evaluated separately.
- Another architecture: AudioSeal maintains similar performance with DPRNN instead of EnCodec, indicating that model capacity is not limiting its results.The comparison is reported for generator and detector architectures, with IoU computed for one second of watermark in ten-second audio.
- Another architecture: The DPRNN comparison uses a much slower and heavier architecture while preserving AudioSeal’s reported performance.This supports consistency across architectures rather than dependence on the specific EnCodec-based model.
- Mixed signals: The mixed-signal evaluation tests whether AudioSeal detects watermarks when watermarked speech and music are normalized and summed.The results distinguish watermarked from non-watermarked mixtures using presence or absence indicators.
C.4. Out of domain (OOD) evaluations
AudioSeal generalizes across languages, modalities, and real-speech datasets, while remaining stronger than WavMark for most tested augmentations but weaker under highpass filtering beyond training conditions.
- Out of domain evaluations: No sample was misclassified among 10k examples in each tested out-of-distribution dataset, including voice-cloning outputs and other audio modalities.The evaluated samples included outputs from various voice-cloning models and additional audio domains without training on AI-generated speech.
- Out of domain evaluations: AudioSet, ASVspoof, and FakeAVCeleb produced performance similar to the primary evaluations on real human speech and deepfake-related data.AudioSet contains environmental sounds, while ASVspoof and FakeAVCeleb are deepfake detection datasets.
- Robustness: AudioSeal outperforms WavMark for most augmentation strengths, but WavMark performs much better against highpass filters above AudioSeal’s training range.The comparison is plotted in Figure 10 as detection accuracy against augmentation strength.
- Robustness: AudioSeal’s TF-loudness loss places watermarks in lower-frequency regions where speech carries more energy, unlike WavMark’s higher-frequency placement.The paper relates this placement to auditory masking and contrasts lowpass and highpass filtering at the same cutoff.
D.2. Robustness Augmentations
Robustness training uses diverse signal augmentations and EnCodec-based generator and detector networks, with evaluations spanning audio quality, adversarial attacks, and malicious detectors.
- Robustness augmentations: Training and evaluation apply noise, smoothing, compression, filtering, speed, resampling, gain, attenuation, and echo augmentations with specified parameter settings.The listed transformations include pink and white noise, AAC and MP3 compression, EnCodec processing, bandpass, highpass, lowpass, speed, resampling, boost, ducking, and echo.
- Network architectures: The generator uses an EnCodec-derived encoder–decoder, while the detector upsamples encoded features and outputs two-class sample-wise probability scores.The generator contains convolutional blocks and LSTMs; the detector adds a transposed convolution and softmax classifier.
- Evaluation: Audio quality is assessed with MUSHRA ratings from 0 to 100 using 100 ten-second speech samples and at least 20 participants per sample.Ground-truth audio averaged 80.49, while the low-compression anchor averaged 53.21.
- Security evaluations: The study also evaluates a gradient-based detector attack and trains a malicious classifier using more than 80k watermarked eight-second Voicebox samples plus genuine speech.The attack seeks a nearby signal whose detector prediction is reversed.