PaperlayerView exact revision on arXiv

Source-linked AI summary

Combining Self-Embedding Audio Watermarking with Ultra-Low-Bitrate Neural Codecs

Yigitcan Özer, Xin Wang, Zhe Zhang, Junichi Yamagishi

arXiv:2608.25289v1cs.SDcs.AI

TL;DR

Localized speech manipulation makes integrity verification difficult, motivating proactive watermarking that can detect and localize edits while recovering overwritten content. This work embeds compact neural codec representations into speech, evaluates extensions across localization, LSB depth, codecs, and manipulations, and finds complete payload recovery without bit errors under ideal conditions, with codec choice driving detection and localization performance.

  • Problem

    Localized speech edits are difficult to detect and localize, while classical hash-based watermarking cannot recover overwritten content.

    Source: §I · p. 1

  • Method

    The framework embeds a compact neural codec representation into the carrier and uses its decoded self-reconstruction for training-free detection, localization, and recovery.

    Source: §I · p. 2, B. Detection and Localization Score… · p. 3

  • Results

    Across codec, bit-depth, and manipulation conditions, the embedded payload is recovered without bit errors; detection and localization remain imperfect, with codec choice dominant.

    Source: C. Results · pp. 5–6, §V · p. 6

  • Takeaways & Limitations

    The framework provides codec-fidelity recovery of manipulated speech regions alongside integrity verification, unlike hash-based verification.

    Source: C. Results · pp. 5–6, §V · p. 6

    Evaluation is conducted under ideal channel conditions, and detection and localization remain imperfect, especially for deletion.

    Source: C. Results · p. 5, §V · p. 6

Abstract

from arXiv · show

Partial manipulation of speech recordings, where only localized segments of an utterance are altered, poses a significant challenge for content integrity verification, as reliable detection and localization of such edits becomes harder as the manipulated proportion decreases. Watermarking offers a proactive defense alternative by embedding auxiliary information prior to distribution; classical hash-based schemes achieve near-perfect detection and localization under ideal conditions, but the original content cannot be recovered once a segment is manipulated. Building on a prior self-embedding audio steganography framework, this work presents an initial exploration of proactive defense performance under ideal conditions, extending the investigation along three axes: frame-level localization, multi-bit least significant bit variants, and evaluation across multiple ultra-low-bitrate neural codec representations. By embedding a compact neural codec representation rather than a cryptographic hash, the framework additionally enables recovery of the manipulated regions, while supporting training-free detection and localization without spoofed examples. Experiments across four controlled manipulation types under ideal channel conditions show that the embedded payload, and hence an approximate reconstruction of the authentic content, is always fully recovered without bit errors. The results also indicate that the choice of neural codec is the dominant factor for detection and localization performance.

I. INTRODUCTION

Partial speech manipulation complicates integrity verification because only localized segments are altered. This work extends neural codec self-embedding toward frame-level localization, multi-bit embedding, and evaluation across ultra-low-bitrate codecs, while retaining recovery of manipulated content.

  • Partial manipulation alters localized speech segments while preserving most of the original recording, complicating integrity verification.Source: §I · p. 1
  • Classical hash-based watermarking can localize tampering exactly under ideal conditions but cannot recover overwritten spoken content.Source: §I · p. 1
  • Self-embedding watermarking embeds an authentication and recovery description derived from the host signal into the carrier itself.Source: §I · p. 1
  • The proposed framework embeds a compact neural codec representation into the carrier, then compares the received signal with its self-reconstruction for detection and localization.Source: §I · p. 1, Figure 1 · p. 2
  • The present study extends prior work to frame-level localization, k ∈{1, 2, 4} multi-bit LSB variants, and multiple codecs and manipulation types.Source: §I · p. 2
  • Codec reconstruction fidelity primarily governs detection and localization performance in the evaluated framework.Source: §I · p. 2

II. SELF-EMBEDDING AUDIO WATERMARKING USING NEURAL AUDIO CODECS

The framework embeds repeated neural codec representations within a capacity-limited watermark, recovers them by majority voting, and compares reconstructed and received signals for detection and localization.

  • Imperceptibility strictly limits the embedding budget, making direct waveform or lossless-content embedding infeasible.Source: A. Problem Formulation · p. 2
  • Ultra-low-bitrate neural codecs compress the carrier into representations small enough to fit repeatedly within the available embedding budget.Source: A. Problem Formulation · p. 2
  • The payload consists of R consecutive repetitions of the codec representation, with R = ⌊M/N⌋ as the maximum number fitting within capacity.Source: A. Problem Formulation · p. 2
  • Repeated representations enable majority-vote decoding, improving robustness of the recovered codec representation when localized manipulations corrupt some copies.Source: A. Problem Formulation · pp. 2–3
  • The framework covers substitution, insertion, and deletion through a spoofing operator that modifies localized time intervals.Source: A. Problem Formulation · p. 2
  • Verification reconstructs the authentic waveform from the decoded payload and derives a mismatch score against the received signal.Source: A. Problem Formulation · p. 3
  • The mismatch score supports both utterance-level binary detection and frame-level localization through local alignment costs projected onto the time axis.Source: A. Problem Formulation · p. 3

B. Detection and Localization Score via Dynamic Time Warping (DTW)

The framework compares a received signal with its self-reconstruction using DTW, producing mismatch scores for utterance-level detection and frame-level localization. DTW accommodates local timing deviations while highlighting structural mismatches caused by partial manipulation.

C. Capacity and Repetition Analysis

Payload repetition is determined by embedding capacity and codec representation size. More available capacity relative to payload size permits more repetitions, strengthening majority-vote recovery against partial manipulation.

  • R = ⌊C/(ρ + 64)⌋ repetitions per second, where C is embedding capacity, ρ is codec representation size, and 64 bits encode synchronization.The total embedding budget satisfies M = C · T/fs.Source: C. Capacity and Repetition Analysis · p. 3

D. Least Significant Bit (LSB) Embedding

The framework uses repetitive LSB embedding to improve payload recovery under partial manipulation. Bit depth controls both embedding capacity and waveform distortion, while majority voting reconstructs the codec payload from intact repetitions.

E. Ultra-Low-Bitrate Neural Audio Codecs

The study evaluates three openly available ultra-low-bitrate neural codecs with differing architectures and speech bitrates. Their representations provide the payload used for self-embedding and self-reconstruction.

III. MANIPULATION CONDITIONS AND DATASET

The evaluation uses four waveform-level partial manipulations on 1,480 AV-Deepfake1M validation utterances under ideal channel conditions. Replacement creates local deviations, whereas deletion and insertion induce temporal shifts that disrupt subsequent payload alignment.

Experimental Conditions

  • Evaluation covers four controlled manipulation types without channel degradation, compression, or additive noise.The conditions characterize achievable accuracy in the absence of channel factors.Source: §III · p. 4

Manipulation Types

  • Direct replacement substitutes word-level segments with acoustically matched authentic material from the same speaker.No resynthesis is applied, so the manipulation is local and detectable through payload mismatch.Source: §III · p. 4
  • TTS replacement substitutes corresponding word-level regions with segments synthesized by zero-shot voice-cloning systems.Ground-truth word boundaries come from AV-Deepfake1M annotations.Source: §III · p. 4
  • Deletion removes a word segment, while insertion adds a donor segment from the same or a different speaker.Both operations induce temporal shifts that disrupt subsequent payload repetitions.Source: §III · p. 4

Dataset and Processing

  • The dataset retains 1,480 validation utterances with authentic recordings and Whisper word-level transcriptions; donor recordings come from VoxCeleb2.Waveform manipulations use overlap-add concatenation, while no codec or vocoder processing is applied.Source: §III · p. 4

A. Experimental Conditions

The framework is evaluated with three ultra-low-bitrate neural codecs, three LSB bit depths, and DTW-based mismatch scoring. These choices test codec representations, payload capacity, and frame-level alignment behavior.

Neural Codecs

  • SNAC, SemantiCodec, and TAAE operate at 0.98 kbps, 0.65 kbps, and 0.4 kbps, respectively.SNAC codec processing uses 24 kHz, while watermarking and evaluation remain at 16 kHz.Source: A. Experimental Conditions · p. 5

LSB Embedding

  • LSB embedding evaluates k ∈{1, 2, 4} bits per sample with a 64-bit synchronization preamble before each payload repetition.The variants examine different capacity–distortion settings.Source: A. Experimental Conditions · p. 5

DTW-Based Scoring

  • DTW aligns received signals with self-reconstructions using 40-band log-mel spectrograms, 16 ms frame shifts, and cosine local dissimilarity.Smoothed per-frame cost profiles support detection and localization scoring.Source: A. Experimental Conditions · p. 5

B. Evaluation Methodology

Detection uses utterance-level EER, while localization uses frame-level ROC AUC from smoothed DTW mismatch profiles. Payload recovery is error-free across configurations, but codec choice produces imperfect and differing performance.

Evaluation Metrics

  • EER measures detection using the mean of the top 1% of smoothed per-frame DTW cosine costs.The threshold is selected where false-positive and false-negative rates are equal; lower EER is better.Source: B. Evaluation Methodology · p. 5
  • Frame-level localization uses ROC AUC between smoothed DTW costs and binary ground-truth frame labels.AUC 1.0 is perfect localization, whereas 0.5 corresponds to random guessing.Source: B. Evaluation Methodology · p. 5

Imperceptibility

  • PESQ reaches 4.64 for all k ∈{1, 2, 4}, indicating perceptually transparent embedding at every evaluated bit depth.4.64 is the maximum attainable wideband PESQ score.Source: C. Results · p. 5

Hash-Based Upper Bound

  • The hash baseline attains detection EER near 0% and localization AUC above 0.999 across manipulation types under ideal conditions.Its performance is an upper bound because localized modifications change decoded hashes, but hash payloads cannot recover speech.Source: C. Results · p. 5

Detection

  • SemantiCodec has the lowest mean EER at 14.47%–14.78%, SNAC reaches 15.00%–15.15%, and TAAE reaches 32.05%–32.11%.Direct replacement is most detectable, deletion is hardest, and longer manipulations improve detection.Source: C. Results · p. 5

Localization

  • SNAC and SemantiCodec achieve mean AUC of 0.839–0.840, while TAAE reaches 0.785; direct and TTS replacement yield AUC from 0.841 to 0.912.Deletion is hardest to localize because it compresses the DTW path rather than producing a high-cost region.Source: C. Results · p. 5

Recovery Quality

  • The framework reconstructs manipulated speech at codec-level fidelity with zero payload bit errors, yielding PESQ 1.75, 1.63, and 1.65 for SNAC, SemantiCodec, and TAAE.Recovered speech remains intelligible, natural, and speaker-consistent, while reconstruction quality and detection performance are decoupled.Source: C. Results · pp. 5–6

V. CONCLUSION

The paper presents training-free neural-codec self-embedding that recovers manipulated content, while detection and localization remain imperfect under ideal conditions. Codec reconstruction characteristics dominate performance, and deletion is the greatest challenge.

V. CONCLUSION

  • A compact ultra-low-bitrate neural codec representation is embedded into the carrier signal for training-free self-recovery watermarking.The recovered payload reconstructs authentic manipulated regions at codec fidelity with intelligible, natural-sounding speech.Source: §V · p. 6
  • Across all codec–bit-depth configurations, payload recovery has zero bit errors, but detection and localization remain imperfect even under ideal conditions.Deletion poses the greatest challenge.Source: §V · p. 6
  • Performance is driven primarily by codec reconstruction characteristics, while LSB bit depth has only a limited role.Future work targets robustness to channel distortions and greater payload compression.Source: §V · p. 6

Related papers

Loading 2608.25289v1…