Source-linked AI summary
AudioNoisePrints: Model-free audio watermarking using spatial correlation in flow matching TTS
Timothy Tin-Long, Jian Zhu, Aidan Pine, Mengzhe Geng
TL;DR
Audio watermarking for diffusion and flow-matching TTS lacks evidence that noise-output correlations transfer effectively from image models to speech synthesis. AudioNoisePrints uses those correlations for training-free watermarking and is more robust than AudioSeal under strong augmentations while avoiding generation overhead and quality compromise.
Problem
It was unclear whether the noise-output correlation behind prior image watermarking could support robust watermarking in diffusion and flow-matching speech-synthesis models.
Method
AudioNoisePrints compares predetermined initial noise with generated audio or Mel-spectrograms using cosine similarity and an empirical p-value, with an optional lightweight detector for stronger augmentations.
Results
AudioNoisePrints is more robust than AudioSeal under most strong augmentations, including cropping and speed changes, while exhibiting similar spatial correlations across tested TTS and vocoder models.
Takeaways & Limitations
The findings support training-free, plug-and-play watermarking for diffusion and flow-matching TTS without retraining, generation overhead, or compromised generation quality.
Abstract
from arXiv · showhide
We present AudioNoisePrints, a training-free watermarking pipeline for flow matching and diffusion TTS models, which requires minimal extra computation during inference and does not require retraining the TTS model or reducing the generation quality. We exploited the fact that there are strong correlations between the initial Gaussian noises and the generated outputs in diffusion and flow matching models, such that a simple cosine correlation between the initial noise and the generated output can be used to perform watermaking. Moreover, we train a lightweight detector on top for more aggressive augmentations. Our method outperforms AudioSeal, a strong baseline for audio watermarking under strong augmentations. We experimented on F5TTS and other TTS and vocoder models, and concluded that they all exhibit similar spatial correlation properties, suggesting our watermarking scheme can be used for more flow-matching TTS models and even vocoders in the future.
1 National Research Council Canada, Canada 2 University of British Columbia, Canada · 1. Introduction
Flow matching and diffusion models enable high-fidelity speech synthesis, but their properties beyond generation quality remain understudied. AudioNoisePrints investigates originating-noise correlations for training-free watermarking in diffusion and flow-matching speech-synthesis models.
- 1. Introduction: Flow matching and diffusion models have advanced high-fidelity speech synthesis across multi-speaker TTS systems and vocoders.Examples include MatchaTTS, E2TTS, F5-TTS, DiffWave, and WaveFM.
- 1. Introduction: Their properties beyond generation quality remain largely unknown.
- 1. Introduction: Generated outputs correlate more strongly with their originating noise than with other randomly generated noise.Prior work observed this relationship between latent representations and generated data, particularly in diffusion models.
- 1. Introduction: AudioNoisePrints is the first reported investigation of NoisePrints-style watermarking in diffusion or flow-matching speech-synthesis models.The original NoisePrints work studied a latent diffusion image model using a VAE.
- 1. Introduction: Experiments with F5-TTS and MatchaTTS showed stronger robustness under strong augmentation than state-of-the-art post-hoc deep watermarking models.The scheme requires no TTS retraining or extra inference computation and avoids the quality-versus-robustness trade-off.
- 1. Introduction: An external detector further improved results against strong augmentations.
2. Background
Diffusion and flow matching models iteratively transform Gaussian noise into target data, with model outputs guiding this trajectory as vector fields. Prior work shows spatial similarity between originating noise and generated outputs, motivating AudioNoisePrints as a training-free in-model audio watermarking pipeline that avoids retraining and preserves generation quality.
- Generative model background: Diffusion and flow matching models reverse Gaussian noise into target data through iterative steps, with their outputs guiding the trajectory toward the data distribution.The paper considers Euler ODE sampling for flow matching experiments.
- Existing audio watermarking: Post-hoc watermarking modifies already generated audio, adding computation and potentially altering quality, while open-source schemes are vulnerable to overwriting attacks.An attacker can detect the watermarking model and overwrite its watermark using the same open-source model.
- Noise-output correlation: Generated outputs have higher cosine similarity with their originating noise than with independently sampled noise, and corresponding elements show higher Pearson correlation.The observed relationship suggests that generated patterns are also present in the original noise.
- Existing audio watermarking: In-model watermarking avoids some post-hoc limitations but typically requires retraining the generative model, substantial computational resources, and may reduce generation quality.These constraints make it difficult to add watermarking to an existing audio-generation pipeline.
- AudioNoisePrints motivation: AudioNoisePrints leverages noise-output spatial similarity to watermark audio during generation without retraining the audio model while preserving most or all generation quality.The approach extends the NoisePrints principle from latent image generation to diffusion and flow matching audio models.
3. Method
AudioNoisePrints adapts NoisePrints to audio by measuring similarity between initial noise and generated audio or Mel-spectrograms in the output domain. It supplements model-free cosine similarity with a trained detector to improve robustness and simplify detection under audio-specific transformations.
- NoisePrints adaptation: The method computes distance between original noise x0 and generated audio or Mel-spectrograms xL in the audio or Mel-spectrogram domain, using functions such as cosine similarity.Unlike the original approach, the distance is not calculated in latent space; detection uses an empirical p-value instead of a fixed threshold τ.
- Audio-specific challenges: Audio-specific attacks, including frequency perturbations, cropping, and speed augmentation, complicate watermark recovery and can reduce detection accuracy.These challenges arise because audio lacks a fixed size like a predetermined latent representation.
- Detector approach: The Detector approach trains an external convolutional neural network to classify whether audio generated by a specific TTS diffusion or flow-matching model came from designated noise.Training uses augmented examples to increase robustness against the detection complexities introduced by audio transformations.
- Detector approach: The Detector approach preserves the model-free benefit of avoiding TTS retraining, which could degrade quality, while improving inference speed and reducing detection complexity.Additional training is applied only to the detector, not to the TTS model.
- Detector implementation: The Audio NoisePrints detector is a 4-layer Conv2D ResNet trained with binary cross-entropy to distinguish Mel-spectrograms generated from designated noise from those generated from random noise.Training feeds the detector Mel-spectrograms from both randomly initialized and specified noise.
4. Results and Discussion
AudioNoisePrints achieves roughly 97–98% top-line accuracy and improves robustness over AudioSeal under strong augmentations while requiring no generation overhead or TTS retraining. Cosine similarity is selected as the distance function, and experiments show the original distance consistently exceeds randomized distances across TTS and vocoder models.
- AudioNoisePrints reaches around 97–98% accuracy and significantly outperforms AudioSeal under strong augmentations, except MP3.The comparison spans F5TTS and MatchaTTS, which use different architectures and training objectives.
- The plug-and-play method adds no computation overhead during generation and does not require retraining the TTS models.The passage contrasts this simplicity with in-model or in-generation watermarking approaches.
- Cosine similarity and dot product achieve acceptable performance, with cosine similarity performing slightly better and cutoff= 1.0 yielding the highest accuracy.All subsequent experiments use cosine similarity as the distance function.
- DiffWave provides additional validation using a diffusion vocoder with a bidirectional dilated convolution architecture distinct from MatchaTTS and F5-TTS.Experiments use official implementations, with VCTK checkpoints for Matcha-TTS and the LJSpeech checkpoint for DiffWave.
- p-value 0.0 occurs for every one of 500 MatchaTTS and DiffWave audio samples, indicating dorg >> drand generally holds for both models.Figure 3 shows a large separation between randomized distances drand and original distances dorg.
5. Conclusions
AudioNoisePrints is a model-free scheme for fast audio watermarking that adds zero computational overhead during generation and avoids compromising generation quality. It is more robust than AudioSeal under strong augmentations, trading off robustness rather than generation quality.
- Conclusions: AudioNoisePrints provides model-free, fast audio watermarking without requiring model training.The scheme is presented as a model-free alternative to commonly used post-hoc watermarking methods.
- Conclusions: Zero computational overhead during generation distinguishes AudioNoisePrints from the compared post-hoc watermarking approach.The passage identifies zero generation-time overhead as a property of the methodology.
- Conclusions: AudioNoisePrints avoids compromising generation quality, with robustness serving as the trade-off.The conclusion explicitly frames the quality-preserving design in terms of a robustness trade-off.
- Conclusions: AudioNoisePrints is more robust than AudioSeal under strong augmentations.AudioSeal is described as a commonly used baseline trained with thousands of examples.