Source-linked AI summary
NeuMark: Neural Codec Resynthesis-Robust Audio Watermarking in the Codec Latent Space
Annan Wu, Wen-Chin Huang, Tomoki Toda
TL;DR
Existing watermarking methods can withstand many DSP attacks, but neural-codec resynthesis can discard evidence not aligned with codec-preserved structure. NeuMark embeds a 16-bit message across SpeechTokenizer RVQ layers and trains for detection and recovery. NeuMark variants show stronger resynthesis robustness, with a trade-off between reference choices for quality evaluation.
Problem
Neural-codec resynthesis can weaken audio watermarks because codec reconstruction preserves quantized acoustic structure while discarding details that are not stable under reconstruction.
Source: §I · p. 1Method
NeuMark embeds a recoverable 16-bit message across SpeechTokenizer RVQ latents and trains for watermark detection and message recovery under DSP and neural-codec distortions.
Source: §I · p. 1, §VI · p. 5Results
NeuMark R-O achieves the highest average robustness, followed closely by NeuMark R-R, while NeuMark variants retain the highest extraction accuracy under speed perturbation.
Source: A. Watermark Extraction Robustness · p. 4Takeaways & Limitations
Using reconstructed and original references distinguishes watermark distortion from codec reconstruction effects when evaluating transparency and robustness.
Source: §I · p. 2
Abstract
from arXiv · showhide
Audio watermarking is increasingly important for tracing generated speech. Several audio watermarking methods have been proposed to embed the watermark in various domains, such as waveform, timbre feature, or latent representations, for making the embedded watermark robust against traditional digital signal processing (DSP) attacks. On the other hand, modern neural codecs introduce a different threat from DSP attacks: they resynthesize speech through quantized acoustic representations and can remove the embedded watermark evi- dence that is not aligned with codec-preserved structure. In this paper, we propose NeuMark, a codec-latent audio watermarking framework that embeds watermark evidence into SpeechTok- enizer acoustic tokens to address this resynthesis threat. NeuMark uses cross-attention to inject a 16-bit message across residual vector quantization (RVQ) layers, distributing the watermark over codec-aligned latent structure. Experimental results show that NeuMark substantially improves robustness under neural- codec resynthesis while supporting both watermark detection and message recovery. We also analyze the trade-off between reconstruction-referenced transparency and original-referenced robustness.
I. INTRODUCTION
Audio watermarking can support provenance tracing, but neural-codec resynthesis can discard watermark evidence that is not aligned with codec-preserved structure. NeuMark embeds a recoverable message in codec-aligned SpeechTokenizer latents and evaluates robustness alongside transparency against both original and reconstructed references.
I. INTRODUCTION
- Neural-codec resynthesis can weaken watermark evidence even when methods withstand DSP attacks, because codecs preserve quantized acoustic structure while discarding unstable details.Prior watermarking performance degrades under neural-codec resynthesis.Source: §I · p. 1
- NeuMark embeds a recoverable message in SpeechTokenizer’s codec-aligned RVQ latents and trains for frame-level detection and message recovery under DSP and codec-resynthesis distortions.Its frozen tokenizer provides multiple codec-aligned acoustic token streams.Source: §I · p. 1
- The paper compares NeuMark with waveform-, timbre-feature-, and latent/generation-aware methods under DSP edits and codec-resynthesis attacks.The comparison is described in Table I.Source: §I · pp. 1–2, Table I · p. 2
- NeuMark examines a latent codec watermarking robustness–transparency trade-off using reconstruction-referenced and original-referenced variants.The evaluation uses both original and reconstructed references to distinguish watermark distortion from codec reconstruction effects.Source: §I · p. 2
II. COMPARISON WITH LATENT-SIDE METHODS
NeuMark differs from prior latent-side methods in where it injects watermark evidence: it distributes the message across RVQ layers rather than injecting it after aggregation or only into speaker-specific layers. Its pipeline encodes an utterance and 16-bit message into modified codec latents, then decodes and tests them under distortions.
II. COMPARISON WITH LATENT-SIDE METHODS
- NeuMark injects the message into each RVQ layer, distributing watermark evidence across codec layers; TraceableSpeech injects after RVQ aggregation, while VoiceMark uses speaker-specific VQ layers.The methods also differ in simulated training attacks and quality-reference targets.Source: §II · p. 2
A. Overview
- In the NeuMark training pipeline, an input utterance x ∈ R^L and a 16-bit message m ∈ {0,1}^16 are provided; frozen SpeechTokenizer encoder E produces e ∈ R^(1024×T), where 1024 is the latent channel dimension and T the number of latent frames.Source: A. Overview · p. 2
- A trainable Transformer watermark encoder modifies the RVQ streams, and the frozen SpeechTokenizer decoder produces watermarked speech for distortion and extractor training.The extractor predicts frame-level watermark probabilities and message bits after stochastic distortion.Source: A. Overview · p. 2
B. Cross-Attention-Based Watermark Encoder
NeuMark conditions each RVQ stream on a 16-bit message using cross-attention, then adds the resulting watermark features to the stream. Training applies sampled distortions and uses either codec-reconstructed or original audio as the quality reference.
B. Cross-Attention-Based Watermark Encoder
- Learnable message-memory vectors condition each RVQ stream through Transformer cross-attention, whose projected features are added residually before the modified streams are decoded.The pretrained SpeechTokenizer remains fixed during training.Source: B. Cross-Attention-Based Watermark… · p. 2
C. Training Distortion Operator
- For each mini-batch, the distortion operator samples one augmentation from the training pool; masking is a training augmentation, not an evaluation attack.The distorted watermarked speech is passed to the extractor.Source: C. Training Distortion Operator · p. 2
D. Training Reference Variants
- NeuMark R-R uses codec-reconstructed audio as its reference, isolating watermark-induced distortion from distortion already introduced by the codec.The reconstruction is computed with the frozen SpeechTokenizer encoder and decoder.Source: D. Training Reference Variants · p. 2
- NeuMark R-O uses the original source audio as its reference, training the embedder to both embed the watermark and compensate for codec reconstruction error.The two variants differ in the reference target used across quality losses and discriminator training.Source: D. Training Reference Variants · p. 2
E. Training Objective
NeuMark jointly optimizes speech-quality preservation and watermark extraction, combining quality-oriented losses with message-decoding and watermark-detection losses. The objective also guides adversarial, feature, spectral, and latent representations.
E. Training Objective
- The encoder and extractor optimize a weighted objective combining speech-quality preservation with watermark extraction and detection.The discriminator is optimized separately.Source: E. Training Objective · pp. 2–4
- Adversarial training encourages watermarked speech to match natural and codec-reconstructed speech using a quality reference selected from the original or codec reconstruction.The discriminator classifies reference audio as real and watermarked audio as fake.Source: E. Training Objective · pp. 2–3
- Feature matching penalizes differences between discriminator features of the quality reference and watermarked speech.The matching uses feature maps from discriminator layers.Source: E. Training Objective · p. 3
- Multi-scale mel loss addresses spectral reconstruction, while latent cosine loss constrains the direction of content latents before and after embedding.The quality reference is either original speech or codec reconstruction, depending on the training variant.Source: E. Training Objective · p. 3
- The extractor predicts a 16-bit message and frame-level watermark probabilities, with detection loss masked to speech frames using clean-utterance VAD labels.Negative samples are clean speech or masked regions.Source: E. Training Objective · p. 3
A. Data
The evaluation uses LibriSpeech test-clean, while NeuMark training uses three LibriTTS training subsets.
A. Data
- LibriSpeech test-clean, containing 2,620 utterances, served as the shared test set for all watermark models.Source: A. Data · p. 4
- NeuMark was trained on LibriTTS train-clean-100, train-clean-360, and train-other-500 at 24 kHz.Source: A. Data · p. 4
B. Baselines
NeuMark is compared with five watermarking baselines using 16-bit payloads. TraceableSpeech is retrained for the comparison, and some baselines lack reconstruction references for quality evaluation.
B. Baselines
- The 16-bit-payload comparison includes WavMark, AudioSeal, Timbre Watermarking, TraceableSpeech, and VoiceMark.Source: B. Baselines · p. 4
- TraceableSpeech was retrained on LibriTTS with EnCodec resynthesis augmentation because no pretrained checkpoint was available.WavMark, AudioSeal, and Timbre Watermarking were treated as non-reconstruction baselines in Table III quality evaluation because they lack paired codec-reconstruction references.Source: B. Baselines · p. 4
C. Training Details
Training uses a varied distortion pool, including DSP edits and codec resynthesis; evaluation tests robustness against DSP and neural-codec attacks. Detection, bit recovery, and perceptual quality use distinct metrics and reference protocols.
C. Training Details
- Each NeuMark variant trained for 150,000 steps with batch size 32 on four RTX 3090 GPUs, taking approximately 97 hours.Source: C. Training Details · p. 4
- Training augmentations included identity, DSP edits, EnCodec resynthesis at 3, 6, and 12 kbps, and masking, with masking applied to 20% of waveform samples.For masking, selected samples were zeroed or replaced with temporally aligned original-audio samples; the distortion types had different sampling weights.Source: C. Training Details · p. 4
D. Evaluation Attacks
- Evaluation covered named DSP edits and neural resynthesis with DAC, EnCodec, and WavTokenizer; EnCodec was treated as seen, while DAC and WavTokenizer were unseen attacks.Unseen codecs tested generalization beyond the codec used during robustness training.Source: D. Evaluation Attacks · p. 4
E. Metrics
- Robustness was measured by watermark detection and bit recovery, while perceptual quality used PESQ, STOI, and SI-SNR under original-reference and reconstruction-reference protocols.Reconstruction-reference quality evaluation did not apply to non-reconstruction baselines.Source: E. Metrics · p. 4
V. RESULTS AND ANALYSIS
NeuMark R-O achieves the highest average robustness, followed closely by R-R, with its advantage most apparent under neural-codec resynthesis. Speed perturbation remains challenging, and extreme WavTokenizer compression degrades watermarking capacity.
A. Watermark Extraction Robustness
- NeuMark R-O achieves the highest average robustness, followed closely by NeuMark R-R, and NeuMark variants yield the highest extraction accuracy under speed perturbation.Speed perturbation remains challenging for all methods because it changes temporal alignment.Source: A. Watermark Extraction Robustness · p. 4
- Waveform methods remain highly competitive under DSP attacks but degrade severely under neural-codec resynthesis, where NeuMark preserves higher detection and bit recovery.The neural-codec average highlights this difference.Source: A. Watermark Extraction Robustness · p. 4
- Unseen DAC and WavTokenizer results test generalization beyond the codec used during robustness training, while WavTokenizer’s extreme compression strongly degrades watermarking capacity.The results suggest future designs need stronger invariance to single-quantizer resynthesis rather than relying only on multi-RVQ residual capacity.Source: A. Watermark Extraction Robustness · p. 4
B. Perceptual Quality
NeuMark R-R provides the strongest reported reconstruction-referenced perceptual quality, while R-O improves robustness at a quality cost. The conclusion describes a framework for 16-bit message recovery and an evaluation protocol that measures quality against both original and reconstructed audio.
B. Perceptual Quality
- Non-reconstruction baselines yield higher fidelity against the original signal because they bypass codec reconstruction, while R-R evaluation isolates watermark-only distortion for latent methods.R-R is more informative for assessing latent-space watermark distortion.Source: B. Perceptual Quality · p. 4
- DSP Avg. and Neural Codec Avg. are uniformly computed over 12 and 6 attacks, respectively, while Average is uniform over all 18 attacks.The note marks these as attacks unseen during training.Source: B. Perceptual Quality · p. 5
- Table III distinguishes evaluation references—original audio x and reconstructed control xrec—from NeuMark training targets xref = x or xref = xrec.R-O and R-R label these reference choices in the metric columns and NeuMark model names, respectively.Source: Table III · p. 5
- Under R-R evaluation, NeuMark R-R achieves the highest PESQ and SI-SNR and matches TraceableSpeech in STOI at the reported precision.NeuMark R-O improves robustness but has lower R-R quality because original-audio training requires both embedding and codec restoration, introducing larger acoustic perturbations.Source: B. Perceptual Quality · p. 5
VI. CONCLUSION
- NeuMark combines a Transformer cross-attention watermark encoder over SpeechTokenizer latents with a multi-objective loss, and the evaluation compares watermark domains across DSP and neural-codec attacks.The protocol measures quality against both original audio and codec reconstruction.Source: §VI · p. 5