PaperlayerView exact revision on arXiv

Source-linked AI summary

No Time to Collapse: Unlocking Robustness and Multiplexed Capacity in Frozen Audio Watermarkers

Xuanye Wang, Linxi Li, Yechen Wang, Liwei Jin, Qianwei Guo, Carsten Maple

arXiv:2609.29737v1cs.SD

TL;DR

Fixed temporal reductions limit how frozen watermark detectors use surviving evidence and separate multiple messages. This paper trains a Conformer decoder on adapter-pooled detector outputs while leaving the encoder and detector frozen. Across three watermarkers it improves attacked-message recovery, raises AUROC point estimates, and supports recovery of two alternating payloads.

  • Problem

    Fixed reductions cannot learn context-dependent weighting of surviving detector windows and typically collapse the evidence into one payload, limiting recovery of multiple messages.

    Source: Introduction · p. 1

  • Method

    A system-specific adapter pools frozen detector soft outputs into windows, and a learned Conformer decoder predicts messages without retraining the encoder or detector.

    Source: Introduction · p. 1

  • Results

    Across three watermarkers, the learned decoder improves attacked-message recovery and raises detection AUROC point estimates, while multiplexing improves two-payload recovery over fixed decoding.

    Source: Introduction · p. 1, Conclusion · p. 7

  • Takeaways & Limitations

    A decoder replacement can improve recovery when temporal evidence survives, but cannot compensate when a frozen detector fails to preserve both payloads.

    Source: Conclusion · p. 7

    Evaluation uses one training seed per system and predominantly Emilia speech; generalization to music, other datasets, and other encoders remains untested.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Modern neural audio watermarking systems typically embed a message repeatedly across time and then collapse the resulting temporal evidence into a single payload using averaging, voting, or another fixed aggregation rule. We argue that this temporal collapse limits both robustness and the recovery of multiple payloads, and that the limitation can be addressed without retraining the underlying watermarker. We freeze a pretrained watermarker's encoder and detector and train only a low-latency Conformer-based decoder. The decoder consumes the detector's temporal soft outputs, which a system-specific adapter pools into a sequence of window-level representations, and predicts the embedded message. On three frozen watermarkers (AURA, AudioSeal, and WavMark), the learned decoder improves recovery of attacked messages and yields higher detection AUROC point estimates on all three. Under controlled full- and partial-coverage multiplexing, it improves joint-exact recovery of two alternating payload words by 9.7-48.0, 6.9-17.3, and 8.2-14.2 percentage points, respectively, under one to three chained attacks on feasible clips.

Introduction

Neural watermarkers reduce temporal detector evidence to one message using fixed rules, which cannot adapt to surviving evidence or reliably separate multiple payloads. The paper replaces that reduction with a learned decoder while keeping the watermarker frozen.

Introduction

  • Fixed averaging, voting, or error-correcting reductions cannot learn context-dependent window weights, and usually return only one payload per utterance.When messages are multiplexed, their evidence can be conflated into ambiguous or unstable predictions.Source: Introduction · p. 1
  • A learned Conformer decoder replaces fixed temporal reduction while leaving each watermarker’s encoder and detector frozen.A system-specific adapter pools detector soft outputs into windows for the decoder.Source: Introduction · p. 1
  • The decoder improves attacked-message recovery and yields higher detection AUROC point estimates across three watermarkers.The systems are AURA, AudioSeal, and WavMark.Source: Introduction · p. 1
  • The Conformer recovers two alternating payloads, improving joint-exact recovery over grid-aware fixed baselines under full and partial coverage.The evaluation separates exact recovery on known watermarked clips from presence detection and uses held-out data with paired significance analysis.Source: Introduction · pp. 1–2

Related Work

Prior audio watermarking work spans classical signal-processing methods and jointly trained neural systems. The paper focuses on how these systems organize and decode temporal payload evidence, positioning learned sequence aggregation as a way to recover messages from frozen detectors.

Audio watermarking

  • Classical audio watermarking uses waveform or spectral transforms, while newer systems jointly train neural embedders and detectors, including AudioSeal, WavMark, and AURA.These approaches balance imperceptibility, capacity, and robustness; benchmarks increasingly evaluate robustness (Liu et al. 2024b; Özer et al. 2025).Source: Related Work · p. 2

Temporal organization and native decoding

  • WavMark and AURA use disjoint blockwise carriers, whereas AudioSeal’s overlapping support can mix evidence near adjacent-payload boundaries.WavMark encodes 32-bit codewords in 1-s chunks; AURA assigns 32-bit codewords to non-overlapping 2-s regions, while AudioSeal embeds a 16-bit payload.Source: Related Work · p. 2
  • All three evaluated systems reduce temporal evidence to one clip-level payload using hand-specified aggregation rules.AudioSeal averages per-sample logits, WavMark averages synchronization-approved chunks, and AURA error-corrects sliding-window predictions before majority voting.Source: Related Work · p. 2

Learned aggregation and sequence models

  • The paper uses a Conformer to map per-window detector outputs to messages, weighting evidence by reliability and locating payloads in time rather than averaging blindly.Unlike prior decoder-centric work on self-vocoding distortions, this work targets frozen-detector temporal-output aggregation and multiple-payload separation.Source: Related Work · p. 2

Method

The method replaces a frozen watermarker’s fixed temporal reduction with a trainable Conformer decoder that reads adapter-pooled window outputs. In multiplex mode, grid information and separate pooling queries support decoding two alternating payloads.

Method

  • A watermarker-specific adapter turns frozen detector outputs into an ordered window sequence, replacing native fixed reductions without changing the encoder or detector.Fixed reductions cannot learn context-dependent window reliability and typically discard which temporal slots produced the evidence.Source: Method · p. 2
  • A shared Conformer architecture processes the sequence, using one attention-pooling query per target message and a payload head for each message.Each watermarker has its own adapter and separately trained decoder; a presence head predicts one clip-level presence logit.Source: Method · p. 2
  • An optional auxiliary head trains on individual windows, excluding boundary-crossing and unwatermarked windows in multiplex mode to preserve localized payload evidence.In single-message mode, majority-watermarked windows predict the clip payload; in multiplex mode, slot-pure majority-watermarked windows predict their assigned message.Source: Method · p. 2
  • Multiplexing assigns A and B to alternating slots on a known fixed grid, with two pooling queries recovering two messages and 2k payload bits per clip.Windows receive flags for an A slot, a B slot, or no unambiguous assignment; grid-aware baselines receive the same slot information.Source: Method · p. 3

Experimental Setup

The setup freezes each watermarker and trains a system-specific Conformer decoder on its detector’s temporal outputs. Training and evaluation cover varied attacks and coverage settings, with payload recovery scored separately from detection.

Decoder Architecture

  • Each frozen detector’s system-specific window representations feed a Conformer that uses one or two attention-pooling queries to predict one or two payloads and a clip-level presence score.Adapters provide AURA logits from overlapping 2-s windows, AudioSeal logits pooled over non-overlapping 0.1-s windows, or WavMark pattern and payload scores from non-overlapping 1-s chunks.Source: Decoder Architecture · p. 3

Training

  • The 1,061,905-clip source pool supplies AudioSeal and WavMark’s deterministic source-disjoint 90/5/5 splits; AURA uses a 50,000-example source-disjoint 70/15/15 single-message split and a separate 303,973-example multiplex dataset.AudioSeal and WavMark clips are prepared at 5 s for single-message and 8 s for multiplex training. Checkpoints are selected by task: validation exact-message accuracy for single-message and joint-exact recovery for multiplex.Source: Training · pp. 3–4
  • For AudioSeal and WavMark, bit loss applies to positive clips, presence loss to all clips, and auxiliary loss to eligible windows; training samples one to three attacks per clip.In multiplex training, auxiliary supervision is limited to majority-watermarked, slot-pure windows; boundary and unwatermarked windows remain inputs but are excluded from that loss. Coverage is system-specific, including partial AudioSeal multiplex examples retaining one A and one B slot.Source: Training · p. 4

Evaluation

  • Evaluation uses the first 11 attacks, excluding timing warps that invalidate the fixed temporal grid; AURA’s per-attack evaluation additionally includes two timing warps.Each clip is tested under single attacks and random chains of two or three attacks, with clean audio as a control.Source: Evaluation · p. 4
  • Single-message tests include about 5,000 clips per watermarker, while multiplex tests use 1,504 clips for AURA and AudioSeal and 742–763 per WavMark cell.Multiplex partial-coverage rates include only feasible clips, where both payloads retain over 0.5 s of watermark in a 1-s slot interval; the mask is not decoder input.Source: Evaluation · p. 4
  • Payload recovery is exact bit-for-bit accuracy on known-positive clips, and multiplex joint-exact recovery requires both payloads to match; detection is assessed separately by AUROC.Recovery differences use paired McNemar tests, paired bootstrap confidence intervals, and Holm–Bonferroni correction; AUROC values are point estimates without paired significance tests.Source: Evaluation · pp. 4–5
  • Baselines receive the same attacked waveform and frozen detector, and multiplex baselines receive the known A/B grid; the WavMark BiGRU comparison uses paired held-out clips.The BiGRU has 171k parameters versus 1.96M for the Conformer; comparisons are interpreted only within each watermarker.Source: Evaluation · p. 5

Results

The Conformer improves joint-exact recovery across attacked multiplex conditions, single-message recovery under most single attacks, and both exact-message accuracy and AUROC under chained attacks. Results differ by watermarker, and WavMark ablations identify partial-coverage augmentation as the only tested component removal with a detectable recovery change.

Capacity: Recovering Two Multiplexed Messages

  • WavMark’s full-coverage gain rises from +8.2 pp at K=1 to +14.2 pp at K=3, and remains significant when partial-coverage clips lacking usable evidence count as failures.Its grid-aware decoder can discard payload estimates when synchronization bits are wrong, while the Conformer combines the soft pattern-and-payload sequence.Source: Capacity: Recovering Two Multiplexed… · p. 5
  • AudioSeal remains detector-limited: on unattacked clips, a diagnostic pass recovers A on 98.7% but B on 0.3%, while the Conformer improves every attacked cell.The low absolute recovery and decreasing gains with attack depth indicate an upstream capacity limit that decoder-side aggregation alone cannot remove.Source: Capacity: Recovering Two Multiplexed… · pp. 5–6

Results

  • 29 of 35 system–attack cases match or favor the Conformer; all six decreases are on AudioSeal, at most 0.7 pp and nonsignificant after correction.Gains include +23.2 pp for AudioSeal under low-pass filtering, +28.9 pp for WavMark under median filtering, and +8.8 pp for AURA under quantization; white-noise gains are at most 0.5 pp.Source: Results · p. 5, Table 2 · p. 6
  • Under K=2, 3 attack chains, the Conformer outperforms the baseline in every exact-message and AUROC cell; gains are +1.4 to +4.2 pp for AudioSeal, +8.0 to +11.4 pp for WavMark, and +3.4 to +6.2 pp for AURA.As a secondary check, validation-calibrated detect-then-decode accuracy improves in all six AudioSeal and WavMark regimes.Source: Results · p. 5, Table 3 · p. 6

Ablation Study

  • Only removing partial-coverage augmentation produces a detectable change: −1.11 pp all-attacked and −1.44 pp partial-attacked; removing auxiliary loss or slot flag shows no difference.A 171k BiGRU retains much of learned aggregation’s benefit, but the Conformer advantage persists and grows under K=2,3; these ablations are WavMark-specific.Source: Ablation Study · pp. 6–7, Table 5 · p. 7

Capacity: Recovering Two Multiplexed Messages

Limitations

The evaluation is bounded by its speech-heavy, single-seed training, attack distributions, aligned multiplex setup, and within-system comparisons.

Limitations

  • Training used one seed per system and predominantly Emilia speech, leaving generalization to music, other datasets, and other encoders untested.The source pool also included a small FSD50K component.Source: Limitations · p. 7
  • For AudioSeal and WavMark, training and evaluation attacks came from the same timing-warp-free family, so robustness to desynchronization is not established.AURA’s per-attack timing-warp rows are out-of-distribution and may misalign audio with detection windows.Source: Limitations · p. 7
  • Multiplex experiments assume a known, sample-aligned ABAB grid and do not test blind phase recovery after cropping or temporal offsets.Source: Limitations · p. 7
  • Multiplex results support only within-system comparisons, and AudioSeal’s absolute two-payload recovery remains too low for a reliable multiplex channel.Systems differ in payload width, sample rate, detector geometry, source data, and training coverage; fixed and learned decoders also receive different temporal sampling paths.Source: Limitations · p. 7

Conclusion

The paper replaces fixed temporal aggregation with a learned decoder while keeping the encoder and detector frozen. It reports improved attacked-message recovery and two-payload multiplex recovery, with limits when the frozen detector does not preserve both payloads.

Conclusion

  • The learned decoder improves attacked message recovery and raises detection AUROC on three watermarkers, while their encoders and detectors remain frozen.The conclusion also reports significant within-system gains over grid-aware fixed aggregation for multiplex recovery under one to three chained attacks.Source: Conclusion · p. 7
  • The learned decoder recovers two time-multiplexed payloads that shipped clip-wide decoders cannot separate, with significant within-system gains under one to three chained attacks.Source: Conclusion · p. 7
  • AudioSeal’s low absolute multiplex recovery indicates that aggregation cannot compensate when its frozen detector fails to preserve both payloads.The conclusion identifies desynchronization attacks, non-speech audio, and more than two multiplexed payloads as next steps.Source: Conclusion · p. 7

Generative AI Use Disclosure

The disclosure limits LLM use to manuscript polishing and assigns scientific work and verification to the authors.

Generative AI Use Disclosure

Loading 2609.29737v1…