Source-linked AI summary
Combining Self-Embedding Audio Watermarking with Ultra-Low-Bitrate Neural Codecs
Yigitcan Özer, Xin Wang, Zhe Zhang, Junichi Yamagishi
TL;DR
Localized speech manipulation makes integrity verification difficult, motivating proactive watermarking that can detect and localize edits while recovering overwritten content. This work embeds compact neural codec representations into speech, evaluates extensions across localization, LSB depth, codecs, and manipulations, and finds complete payload recovery without bit errors under ideal conditions, with codec choice driving detection and localization performance.
Problem
Localized speech edits are difficult to detect and localize, while classical hash-based watermarking cannot recover overwritten content.
Method
The framework embeds a compact neural codec representation into the carrier and uses its decoded self-reconstruction for training-free detection, localization, and recovery.
Results
Across codec, bit-depth, and manipulation conditions, the embedded payload is recovered without bit errors; detection and localization remain imperfect, with codec choice dominant.
Takeaways & Limitations
The framework provides codec-fidelity recovery of manipulated speech regions alongside integrity verification, unlike hash-based verification.
Takeaways & Limitations
Evaluation is conducted under ideal channel conditions, and detection and localization remain imperfect, especially for deletion.
Abstract
from arXiv · showhide
Partial manipulation of speech recordings, where only localized segments of an utterance are altered, poses a significant challenge for content integrity verification, as reliable detection and localization of such edits becomes harder as the manipulated proportion decreases. Watermarking offers a proactive defense alternative by embedding auxiliary information prior to distribution; classical hash-based schemes achieve near-perfect detection and localization under ideal conditions, but the original content cannot be recovered once a segment is manipulated. Building on a prior self-embedding audio steganography framework, this work presents an initial exploration of proactive defense performance under ideal conditions, extending the investigation along three axes: frame-level localization, multi-bit least significant bit variants, and evaluation across multiple ultra-low-bitrate neural codec representations. By embedding a compact neural codec representation rather than a cryptographic hash, the framework additionally enables recovery of the manipulated regions, while supporting training-free detection and localization without spoofed examples. Experiments across four controlled manipulation types under ideal channel conditions show that the embedded payload, and hence an approximate reconstruction of the authentic content, is always fully recovered without bit errors. The results also indicate that the choice of neural codec is the dominant factor for detection and localization performance.
I. INTRODUCTION
Partial speech manipulation complicates integrity verification because only localized segments are altered. This work extends neural codec self-embedding toward frame-level localization, multi-bit embedding, and evaluation across ultra-low-bitrate codecs, while retaining recovery of manipulated content.
- Partial manipulation alters localized speech segments while preserving most of the original recording, complicating integrity verification.
- Classical hash-based watermarking can localize tampering exactly under ideal conditions but cannot recover overwritten spoken content.
- Self-embedding watermarking embeds an authentication and recovery description derived from the host signal into the carrier itself.
- The proposed framework embeds a compact neural codec representation into the carrier, then compares the received signal with its self-reconstruction for detection and localization.
- The present study extends prior work to frame-level localization, k ∈{1, 2, 4} multi-bit LSB variants, and multiple codecs and manipulation types.
- Codec reconstruction fidelity primarily governs detection and localization performance in the evaluated framework.
II. SELF-EMBEDDING AUDIO WATERMARKING USING NEURAL AUDIO CODECS
The framework embeds repeated neural codec representations within a capacity-limited watermark, recovers them by majority voting, and compares reconstructed and received signals for detection and localization.
- Imperceptibility strictly limits the embedding budget, making direct waveform or lossless-content embedding infeasible.
- Ultra-low-bitrate neural codecs compress the carrier into representations small enough to fit repeatedly within the available embedding budget.
- The payload consists of R consecutive repetitions of the codec representation, with R = ⌊M/N⌋ as the maximum number fitting within capacity.
- Repeated representations enable majority-vote decoding, improving robustness of the recovered codec representation when localized manipulations corrupt some copies.
- The framework covers substitution, insertion, and deletion through a spoofing operator that modifies localized time intervals.
- Verification reconstructs the authentic waveform from the decoded payload and derives a mismatch score against the received signal.
- The mismatch score supports both utterance-level binary detection and frame-level localization through local alignment costs projected onto the time axis.
B. Detection and Localization Score via Dynamic Time Warping (DTW)
The framework compares a received signal with its self-reconstruction using DTW, producing mismatch scores for utterance-level detection and frame-level localization. DTW accommodates local timing deviations while highlighting structural mismatches caused by partial manipulation.
- DTW aligns the received signal y with its self-reconstruction R(y) to separate structural mismatches from local timing offsets.The alignment uses normalized log-mel spectrogram features.
- π*: optimal warping path; c_t: local cosine distance between aligned feature steps.
- The utterance-level detection score averages the smoothed per-step DTW costs, with higher values indicating greater manipulation likelihood.
- For localization, local DTW costs are aggregated by target frame, smoothed, and compared with binary ground-truth labels from word-level boundaries.
C. Capacity and Repetition Analysis
Payload repetition is determined by embedding capacity and codec representation size. More available capacity relative to payload size permits more repetitions, strengthening majority-vote recovery against partial manipulation.
- R = ⌊C/(ρ + 64)⌋ repetitions per second, where C is embedding capacity, ρ is codec representation size, and 64 bits encode synchronization.The total embedding budget satisfies M = C · T/fs.
D. Least Significant Bit (LSB) Embedding
The framework uses repetitive LSB embedding to improve payload recovery under partial manipulation. Bit depth controls both embedding capacity and waveform distortion, while majority voting reconstructs the codec payload from intact repetitions.
- Temporally repetitive LSB embedding enables payload recovery when portions of the watermarked signal are manipulated.
- k ∈{1, 2, 4} replaces the k least significant bits of each sample, with capacity–distortion tradeoffs controlled by bit depth.Maximum perturbation is 2^k −1 quantization steps per sample.
- The k = 1 configuration is the standard single-bit LSB scheme evaluated in prior work.
- Table I reports the number of repetitions R for each codec–bit-depth combination using a 1 s carrier.
- Each payload repetition begins at a distinct sample index, and majority voting estimates codec bits from intact copies.
E. Ultra-Low-Bitrate Neural Audio Codecs
The study evaluates three openly available ultra-low-bitrate neural codecs with differing architectures and speech bitrates. Their representations provide the payload used for self-embedding and self-reconstruction.
- SNAC uses multi-resolution quantizers within an RVQ framework and reaches 0.98 kbps for speech at 24 kHz.
- SemantiCodec separates semantic and acoustic information across two VQ layers and reaches bitrates as low as 0.31 kbps.
III. MANIPULATION CONDITIONS AND DATASET
The evaluation uses four waveform-level partial manipulations on 1,480 AV-Deepfake1M validation utterances under ideal channel conditions. Replacement creates local deviations, whereas deletion and insertion induce temporal shifts that disrupt subsequent payload alignment.
- Experimental Conditions: Evaluation covers four controlled manipulation types without channel degradation, compression, or additive noise.The conditions characterize achievable accuracy in the absence of channel factors.
- Manipulation Types: Direct replacement substitutes word-level segments with acoustically matched authentic material from the same speaker.No resynthesis is applied, so the manipulation is local and detectable through payload mismatch.
- Manipulation Types: TTS replacement substitutes corresponding word-level regions with segments synthesized by zero-shot voice-cloning systems.Ground-truth word boundaries come from AV-Deepfake1M annotations.
- Manipulation Types: Deletion removes a word segment, while insertion adds a donor segment from the same or a different speaker.Both operations induce temporal shifts that disrupt subsequent payload repetitions.
- Dataset and Processing: The dataset retains 1,480 validation utterances with authentic recordings and Whisper word-level transcriptions; donor recordings come from VoxCeleb2.Waveform manipulations use overlap-add concatenation, while no codec or vocoder processing is applied.
A. Experimental Conditions
The framework is evaluated with three ultra-low-bitrate neural codecs, three LSB bit depths, and DTW-based mismatch scoring. These choices test codec representations, payload capacity, and frame-level alignment behavior.
- Neural Codecs: SNAC, SemantiCodec, and TAAE operate at 0.98 kbps, 0.65 kbps, and 0.4 kbps, respectively.SNAC codec processing uses 24 kHz, while watermarking and evaluation remain at 16 kHz.
- LSB Embedding: LSB embedding evaluates k ∈{1, 2, 4} bits per sample with a 64-bit synchronization preamble before each payload repetition.The variants examine different capacity–distortion settings.
- DTW-Based Scoring: DTW aligns received signals with self-reconstructions using 40-band log-mel spectrograms, 16 ms frame shifts, and cosine local dissimilarity.Smoothed per-frame cost profiles support detection and localization scoring.
B. Evaluation Methodology
Detection uses utterance-level EER, while localization uses frame-level ROC AUC from smoothed DTW mismatch profiles. Payload recovery is error-free across configurations, but codec choice produces imperfect and differing performance.
- Evaluation Metrics: EER measures detection using the mean of the top 1% of smoothed per-frame DTW cosine costs.The threshold is selected where false-positive and false-negative rates are equal; lower EER is better.
- Evaluation Metrics: Frame-level localization uses ROC AUC between smoothed DTW costs and binary ground-truth frame labels.AUC 1.0 is perfect localization, whereas 0.5 corresponds to random guessing.
- Imperceptibility: PESQ reaches 4.64 for all k ∈{1, 2, 4}, indicating perceptually transparent embedding at every evaluated bit depth.4.64 is the maximum attainable wideband PESQ score.
- Hash-Based Upper Bound: The hash baseline attains detection EER near 0% and localization AUC above 0.999 across manipulation types under ideal conditions.Its performance is an upper bound because localized modifications change decoded hashes, but hash payloads cannot recover speech.
- Detection: SemantiCodec has the lowest mean EER at 14.47%–14.78%, SNAC reaches 15.00%–15.15%, and TAAE reaches 32.05%–32.11%.Direct replacement is most detectable, deletion is hardest, and longer manipulations improve detection.
- Localization: SNAC and SemantiCodec achieve mean AUC of 0.839–0.840, while TAAE reaches 0.785; direct and TTS replacement yield AUC from 0.841 to 0.912.Deletion is hardest to localize because it compresses the DTW path rather than producing a high-cost region.
- Recovery Quality: The framework reconstructs manipulated speech at codec-level fidelity with zero payload bit errors, yielding PESQ 1.75, 1.63, and 1.65 for SNAC, SemantiCodec, and TAAE.Recovered speech remains intelligible, natural, and speaker-consistent, while reconstruction quality and detection performance are decoupled.
V. CONCLUSION
The paper presents training-free neural-codec self-embedding that recovers manipulated content, while detection and localization remain imperfect under ideal conditions. Codec reconstruction characteristics dominate performance, and deletion is the greatest challenge.
- V. CONCLUSION: A compact ultra-low-bitrate neural codec representation is embedded into the carrier signal for training-free self-recovery watermarking.The recovered payload reconstructs authentic manipulated regions at codec fidelity with intelligible, natural-sounding speech.
- V. CONCLUSION: Across all codec–bit-depth configurations, payload recovery has zero bit errors, but detection and localization remain imperfect even under ideal conditions.Deletion poses the greatest challenge.
- V. CONCLUSION: Performance is driven primarily by codec reconstruction characteristics, while LSB bit depth has only a limited role.Future work targets robustness to channel distortions and greater payload compression.