Source-linked AI summary
NeuMark-Native: Robust Text-to-Speech-Native Watermarking Through Full Utilization of Neural Audio Codec Latent Space
Annan Wu, Wen-Chin Huang, Tomoki Toda
TL;DR
Most neural speech watermarks are added after TTS, leaving them confined to a waveform-level, bypassable step. NeuMark-Native embeds payloads into generated codec layers during synthesis while keeping the TTS model and codec frozen. Across two corpora, it shows strong detection robustness, including under neural-codec resynthesis, while preserving speech quality close to synthetic speech.
Problem
Most neural speech watermarking is applied after TTS as a waveform perturbation, leaving a separate, bypassable step and a waveform-limited carrier.
Source: §1 · p. 1Method
NeuMark-Native adapts watermark modules on pre-synthesized TTS codec tokens and embeds payloads into generated codec layers before decoding, with the TTS model and codec frozen.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2Results
Across two corpora, NeuMark-Native shows strong detection robustness, particularly under neural-codec resynthesis, while preserving naturalness, intelligibility, and perceptual quality close to synthetic speech.
Source: §1 · p. 1, §3.2 · p. 3, §4 · p. 4Takeaways & Limitations
The frozen-backbone design permits transfer to compatible codec-based TTS models through watermark-only adaptation.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Speech watermarking offers proactive traceability for synthetic speech, yet most existing models operate only after text-to-speech (TTS) synthesis by adding a watermark perturbation to the generated waveform. This post-hoc design leaves watermarking as an external step that can be omitted or bypassed and restricts the watermark to a shallow waveform representation. We propose NeuMark-Native, a TTS-native watermarking framework for neural codec-based synthesis. It embeds payload information into every generated codec-latent layer before waveform decoding, improving watermark persistence under downstream digital signal processing (DSP) and neural codec resynthesis. NeuMark-Native keeps the pretrained TTS model and the neural codec frozen, while optimizing only the watermark modules on generated codec tokens. Experiments on two corpora under 11 DSP attacks and 9 neural-codec attacks demonstrate robust watermark detection while preserving naturalness, intelligibility, and speech quality close to synthetic speech.
1. INTRODUCTION
NeuMark-Native moves watermarking into codec-based TTS synthesis, addressing the bypassable post-hoc interface and extending watermarking across generated codec layers. It adapts watermark modules on TTS-generated tokens while keeping the TTS model and codec frozen, and the paper reports strong codec-resynthesis robustness with speech quality comparable to waveform-domain methods.
1. INTRODUCTION
- Increasingly realistic codec-based TTS, including VALL-E 2 [1], MaskGCT, and CosyVoice 3 [3], heightens the need for reliable synthetic-speech provenance.Speech watermarking embeds recoverable information to provide proactive traceability.Source: §1 · p. 1
- Post-hoc methods add watermark perturbations to the final waveform, confining the carrier and leaving watermarking as a separate step that can be omitted or bypassed.Fig. 1 depicts watermarking after TTS synthesis.Source: §1 · p. 1, Figure 1 · p. 1
- TraceableSpeech embeds a watermark in one continuous codec representation, but requires joint codec training and later TTS retraining for watermark changes, incurs a speech-quality cost, and leaves codec-resynthesis robustness unaddressed.Its TTS-native synthesis path is illustrated in Fig. 2.Source: §1 · p. 1, Figure 2 · p. 1
- NeuMark [13] keeps the codec frozen and distributes watermark evidence across RVQ layers to retain detectability after neural codec resynthesis, but its carriers come from real speech rather than TTS generation..Source: §1 · p. 1
- NeuMark-Native adapts watermark modules on pre-synthesized TTS codec tokens and embeds payloads natively during synthesis while keeping the TTS model and codec frozen.The offline preparation pairs TTS-generated codec tokens with their decoded synthetic speech.Source: §1 · p. 1, Figure 3 · p. 2
- NeuMark-Native substantially outperforms competing methods in codec-resynthesis robustness while achieving speech quality comparable to waveform-domain models.The evaluation covers two corpora under 11 DSP and 9 neural-codec attacks.Source: §1 · pp. 1–2
2. PROPOSED METHOD
NeuMark-Native prepares paired TTS-generated codec-token and speech samples, then trains watermark modules on those tokens while keeping the TTS model and codec frozen.
2.1. Offline generated-token preparation
- Each training sample pairs tokens predicted from a transcript and same-speaker prompt with their decoded synthetic speech, making TTS-generated tokens the watermark carrier.The cached tokens expose the watermark modules to token characteristics encountered during synthesis.Source: §2.1 · p. 2
2.2. TTS-native training objective
- For each 16-bit message, cross-attention modifies every RVQ stream; frozen decoding and encoding support paired reference and distorted branches for message recovery.Only the watermark modules learn from the difference between the paired branches.Source: §2.2 · p. 2
- The objective retains six NeuMark losses and adds mel, MOS, speaker-similarity, and ASR terms, with the paired reference anchoring content and acoustics.The added terms are weighted in the joint objective.Source: §2.2 · p. 2
- The speech-aware terms encourage higher predicted naturalness, preserve the prompt speaker’s identity, and preserve target linguistic content without frame-aligned labels.Naturalness uses a frozen UTMOS predictor; speaker similarity uses a frozen verification model, and ASR uses CTC.Source: §2.2 · p. 2
- Watermark modules are optimized with discriminators frozen, while the discriminators are updated separately using detached watermarked speech.The TTS model, codec, and speech-assessment networks remain frozen.Source: §2.2 · p. 2
- The direct variant applies a NeuMark checkpoint trained on natural-speech codec tokens to TTS tokens, while the adapted variant fine-tunes it on paired training data.Both variants keep the TTS model and codec frozen and use the same embedding procedure during synthesis.Source: §2.2 · p. 2
3. EXPERIMENTAL EVALUATION
Across two corpora, the evaluation tests detection robustness, speech quality, and the effects of removing TTS-native losses or generated-token training. NeuMark-Native performs best overall on detection while retaining speech quality close to the unwatermarked reference.
3.1. Setup
- Adaptation uses approximately 157,000 LibriTTS cuts, ten epochs, and training augmentations spanning 11 DSP attacks and EnCodec resynthesis at 3, 6, and 12 kbps.The approximately 69-hour run used one NVIDIA H100 GPU.Source: §3.1 · p. 3
- Evaluation uses 2,078 LibriTTS test-clean utterances and 1,088 SeedTTS-en prompt–text pairs to test zero-shot cloning and generalization to unseen texts and speakers.The attack evaluation includes identity, 11 DSP attacks, and nine codec attacks; DAC and SNAC are unseen during training.Source: §3.1 · p. 3
3.2. Performance evaluation
- NeuMark-Native (adapted) achieves the best overall detection on both corpora, while the direct variant has the highest overall payload-bit accuracy.Table 1 reports means across identity, DSP, and neural-codec conditions.Source: §3.2 · p. 3, Table 1 · p. 4
- Speech quality remains close to the unwatermarked reference and competitive with WavMark and AudioSeal across naturalness, speaker similarity, intelligibility, and perceptual fidelity.Table 2 evaluates quality against the same synthetic speech.Source: §3.2 · p. 3, Table 2 · p. 4
3.3. Ablation study
- Removing TTS-native losses keeps UTMOS and SSIM close to AudioSeal and yields the best LibriTTS PESQ and SeedTTS-en STOI.The ablation removes UTMOS, WavLM-similarity, and ASR objectives while retaining the same initialization and NeuMark objectives.Source: §3.3 · pp. 3–4, Table 3 · p. 4
- Replacing generated tokens with real-speech tokens improves low-FPR detection and payload recovery over the full adapted model, but modestly lowers quality.Generated-token training favors the overall robustness–quality balance; real-speech tokens favor strict low-FPR detection and bit recovery.Source: §3.3 · pp. 3–4
4. CONCLUSION
NeuMark-Native embeds payloads in every generated RVQ stream before decoding, using pre-synthesized TTS tokens while keeping the TTS model and codec frozen. Across two corpora it achieves strong detection, particularly under neural-codec resynthesis, while maintaining speech quality close to synthetic speech.
4. CONCLUSION
- NeuMark-Native adapts NeuMark on pre-synthesized TTS tokens and embeds payloads into every generated RVQ stream before decoding.The TTS model and neural codec stay frozen; only watermark modules are adapted.Source: §4 · p. 4
- Across two corpora, NeuMark-Native provides the strongest overall detection and low-FPR performance, with particularly large gains under neural-codec resynthesis.Source: §4 · p. 4
- It retains naturalness, intelligibility, speaker similarity, and perceptual quality close to synthetic speech and supports transfer to compatible codec-based TTS models through watermark-only adaptation.Source: §4 · p. 4