PaperlayerView exact revision on arXiv

Source-linked AI summary

WARP: A Unified Benchmark for Invisible Image Watermarking -- Robustness and Protection Against Attacks

Khaled Abud, Aleksey Yakushev, Aleksandr Akimenkov, Irina Serzhenko, Kirill Aistov, Egor Kovalev, Dmitry Obydenkov, Sergey Lavrushkin, Anastasia Antsiferova, Dmitriy Vatolin, Yury Markin, Kirill Lukianov

arXiv:2609.40031v1cs.CVcs.AIcs.MM

TL;DR

Existing benchmarks do not cover modern watermarking methods and complex attacks in a unified evaluation of quality, robustness, and overwriting. WARP evaluates a broad set of methods and attack scenarios, finding that no method is uniformly robust and that re-embedding often overwrites post-hoc watermarks. These results point to method-specific strengths and vulnerabilities, while the benchmark remains limited to images.

  • Problem

    Existing benchmarks lack coverage of recent techniques, complex attacks, and a unified assessment of perceptual quality, robustness, and overwriting.

    Source: §1 · p. 1

  • Method

    WARP provides a reproducible benchmark of modern watermarking methods and attack strategies, including competitive re-embedding evaluation.

    Source: §1 · p. 2

  • Results

    No method is uniformly robust; re-embedding frequently overwrites post-hoc watermarks, while generative and key-based methods resist it.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Watermark persistence depends on the embedding mechanism, and method-specific strengths and vulnerabilities inform strategy selection.

    Source: §1 · p. 2, §6 · p. 8

    The benchmark covers image watermarking, not video, audio, or text, so its conclusions do not generalize across those modalities.

    Source: D Limitations and Future Work · p. 20

Abstract

from arXiv · show

Digital image watermarking is increasingly critical in media contexts, as emerging regulations and industry practices require marking AI-generated content and ensuring traceable sources to prevent manipulation or misuse. Recent advances in invisible watermarking methods highlight the need to update existing benchmarking practices to reflect current techniques and evaluation criteria. We address this by introducing WARP -- a unified framework and benchmark for evaluating the robustness of invisible watermarks. WARP incorporates 32 recent classical, deep, and generative watermarking methods, as well as 34 different erasing techniques, ranging from traditional distortions to more sophisticated adversarial, purification, and re-embedding attacks. It provides standardized, reproducible, and easily scalable protocols for evaluating perceptual quality, watermark readability, and attack resilience. Using WARP, we extensively evaluate current invisible watermarking techniques, collecting the largest robustness benchmark in the field. Results identify the most robust approaches under both distortion and adversarial conditions, and reveal consistent relationships between watermarking methods and the attack strategies most effective against them. Our experiments also highlight that some of the watermarking methods considered are highly vulnerable to reembedding, even if they are robust to standard distortions. The code is made available at https://github.com/ispras/wibe.

1 Introduction

Existing benchmarks do not adequately cover modern watermarking methods, complex attacks, or the joint assessment of quality, robustness, and overwriting. The paper introduces WARP to standardize evaluation and examine these gaps, including competitive re-embedding.

1 Introduction

  • Existing benchmarks omit recent deep and generative methods, complex removal and re-embedding attacks, and unified evaluation of quality, robustness, and overwriting.These gaps hinder systematic comparison across threat models and identification of vulnerabilities.Source: §1 · pp. 1–2
  • WARP aims to provide a comprehensive, reproducible benchmark for perceptual quality and robustness across representative watermarking and attack strategies.Its three research questions address survival under diverse attacks, method-specific failure modes, and whether re-embedding can overwrite a watermark without compromising the carrier image.Source: §1 · p. 2
  • The benchmark evaluates 32 classical, deep, and generative watermarking methods across 34 adversarial scenarios using an open-source, modular framework.The authors describe it as the largest collection of methods and attacks in the field at submission time.Source: §1 · p. 2
  • WARP introduces a methodology for systematically testing competitive re-embedding, an underexamined threat to watermark persistence.The results are intended to inform strategy selection based on method-specific strengths and vulnerabilities.Source: §1 · p. 2

2 Related Work

Invisible watermarking spans classical post-hoc techniques, learned architectures, and methods embedded during generation. Prior robustness work and benchmarks cover varied attacks and protocols, while WARP expands their scope and performs a large comparative evaluation.

2.1 Invisible Watermarking Methods

  • Generative methods embed watermarks during image creation, while post-hoc methods add them to existing images and may suit legacy or arbitrary content.The surveyed methods range from classical transform-domain approaches to neural architectures and in-generation diffusion methods.Source: §2.1 · pp. 2–3
  • Watermarking methods span zero-bit to multi-bit settings and in-generation to post-generation paradigms, with distinct approaches targeting capacity, imperceptibility, and geometric resilience.Examples include ChunkySeal [52] for higher capacity, TreeRing [69] for geometric distortions, and Stable Signature [14] through generation-model fine-tuning.Source: §2.1 · pp. 2–3

2.2 Watermark Robustness Evaluation

  • Robustness research examines natural distortions as well as targeted regeneration and adversarial attacks that can remove watermarks.Reported vulnerabilities include pattern extraction by averaging marked images and removal of TreeRing through pixel shifts.Source: §2.2 · p. 3
  • Existing benchmarks vary in scope: WAVES includes 20 attacks but evaluates three watermarks, while W-Bench tests 11 methods against 13 attacks.MarkDiffusion provides a toolkit rather than a large-scale comparative benchmark, and WIBE focuses on modularity without detailed benchmark results.Source: §2.2 · p. 3
  • WARP extends WIBE into a large comparative benchmark, while its evaluation pipeline proceeds from embedding and quality assessment through attacks, extraction, and reporting.The authors characterize WARP as integrating the largest collection of watermarking and removal techniques to date.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

3 Robustness Evaluation Framework

WARP is a modular framework designed to make watermark robustness experiments consistent and reproducible. Its configurable workflow and dependency isolation support comparisons across diverse methods and scalable extensions.

3 Robustness Evaluation Framework

  • WARP uses a plugin-based architecture and YAML experiment configuration to set up evaluation protocols flexibly and transparently.It supports multiple data formats and storage backends, real-time visual feedback, and multi-GPU experiment orchestration.Source: §3 · p. 4
  • Its workflow separates watermark embedding, quality assessment, attack application, extraction, and result logging into configurable processing stages.This modular design is intended to facilitate rapid experimentation and fair comparisons across methods.Source: §3 · p. 4
  • WARP isolates incompatible method dependencies with an automatic resolver and allows new methods, attacks, metrics, and datasets to be added through its lightweight API.The framework also adds re-embedding and multi-step attack setups to WIBE's design.Source: §3 · p. 4

4 Benchmark

The benchmark combines diverse watermarking methods, attacks, datasets, and evaluation measures to assess robustness alongside image quality. It includes conventional and advanced threats, plus a separate experiment on competitive re-embedding.

3 Benchmark

  • The benchmark includes 26 post-hoc and six generative watermarks, spanning classical algorithms, learned methods, and capacities from zero-bit detection to 1024-bit payloads.Its 34 primary attacks are grouped into traditional distortions, regeneration, adversarial purification, and erasure methods.Source: §4 · pp. 4–5
  • A separate re-embedding experiment uses the 26 post-hoc watermarks as attacks against all 32 benchmark methods.The study evaluates victim-attacker pairs in this scenario.Source: §4 · p. 5
  • Robustness evaluations sample 1,000 images each from COCO and DiffusionDB, with smaller cross-dataset samples from NIPS, KonIQ, and DIV2K.The experiments process more than 2.8 million images and use over 8,000 GPU-hours.Source: §4 · p. 5
  • Robustness metrics assess detection or message recovery, while image-quality metrics compare images directly or evaluate them without a reference.Zero-bit methods use measures such as p-value and TPR at a specified false-positive rate; multi-bit methods use BER or WER and can be thresholded for unified TPR reporting.Source: §4 · p. 5
  • Quality evaluation combines full-reference measures such as PSNR and SSIM with no-reference and prompt-based metrics, plus FID for distribution similarity.Prompt-based metrics assess alignment with textual descriptions for in-generation watermarking methods.Source: §4 · p. 5

5 Results

WARP evaluates watermark trade-offs and resilience across diverse distortions, adversarial attacks, and re-embedding. Results show that robustness depends on the threat model and embedding design, while attack effectiveness also varies with perceptual cost.

5.1 Clear Performance: Trade-offs in image watermarking

  • Recent post-hoc methods including PixelSeal, ChunkySeal, and InvisMark shift the capacity–imperceptibility frontier, while higher throughput in modern methods relies on GPU acceleration.The observed decoupling between capacity and imperceptibility makes robustness an increasingly important limiting factor.Source: §5.1 · p. 5

5.2 Watermark Robustness: Evaluation in adversarial scenarios

  • Capacity improves robustness only up to a point, and robustness rankings remain largely consistent across five heterogeneous datasets and resolutions.At larger resolutions, robustness scores are generally slightly higher because many algorithms downscale images internally, smoothing finer adversarial perturbations.Source: §5.2 · p. 6
  • Robustness depends on the threat: explicit robustness-oriented training improves the quality–robustness balance, but geometric misalignment and image regeneration remain major vulnerabilities, especially for post-hoc methods.Generative watermarks, particularly Gaussian Shading and MaXsive, retain higher robustness against regeneration; no method is uniformly robust.Source: §5.2 · pp. 6–7

5.3 Like cures Like: Evaluating watermark robustness under re-embedding attacks

  • Post-hoc watermarks are relatively resistant to external re-embedding, but attack effects are asymmetric and most methods show limited resilience to self-re-embedding.Key-based methods can survive when re-embedded with a different key; DCT CAISS is the only method reported to strongly resist self-overlays. Built-in methods are practically immune to post-hoc overlays, with METR as an exception.Source: §5.3 · pp. 7–8, Figure 5 · p. 7

5.4 Adversarial Strength: Evaluating attacks from the attacker’s perspective

  • Watermark-removal methods WMForger, TrustMark-RM, and SADRE combine strong disruption with reasonable image fidelity, while DiffPure is relatively weak and DISCO and DIP are more effective.Traditional geometric transforms such as rotation and cropping are among the most destructive, although pixel-level full-reference metrics do not capture their image quality well.Source: §5.4 · p. 8

6 Conclusion

The conclusion presents WARP as a reproducible benchmark of invisible watermarking methods and attacks, and finds that robustness is threat-model dependent. It recommends choosing watermarking approaches according to expected attacks and deployment context.

6 Conclusion

  • WARP evaluates 32 watermarking methods across 34 attack scenarios and finds no method uniformly robust; persistence varies with embedding mechanism and threat model.The authors favor post-hoc methods with explicit synchronization for conventional processing and in-generation watermarks when pipelines allow them and regeneration is a threat.Source: §6 · p. 8

A Contents •

  • The appendices cover methods and evaluation protocols, robustness and quality analyses, cross-dataset and cross-resolution results, attack hyperparameters, frequency-domain signatures, and benchmark limitations.Section D discusses limitations and future work.Source: A Contents • · p. 11

B Details on employed methods

The benchmark uses a unified protocol and official implementations where available to compare watermarking methods consistently and reproducibly.

B Details on employed methods

  • The benchmark describes its watermarking methods and adversarial attacks, then applies a unified protocol that generates an independent random bit sequence for each input image.Official implementations and released pretrained weights are used where available; preprocessing, post-processing, and evaluation are applied uniformly across methods.Source: B Details on employed methods · p. 11
  • Images are resized to a fixed embedding resolution, and the upscaled watermark residual is added back to the original-resolution input.This strategy, following TrustMark [7], supports spatial alignment across methods and arbitrary image resolutions while preserving the embedded signal’s structure.Source: B Details on employed methods · p. 11

C.1 Further details on extraction success metrics

The section explains how multi-bit detection is evaluated, how payload capacity and attack type shape detection success, and how built-in watermarks affect generated-image quality. It finds that common mild distortions have limited impact and that built-in watermarks generally introduce little quality change.

C.1 Further details on extraction success metrics

  • Theoretical TPR@xFPR estimates use a Bernoulli(0.5) null model to compute very low false-positive rates without massive non-watermarked datasets.The protocol estimates FPR from the number of matching bits and a decision threshold.Source: §C.1 · p. 11
  • For FPR≤1%, theoretical and empirical TPR differ slightly across noise levels, despite extracted bits not being perfectly i.i.d. Bernoulli.The validation used TrustMark with 100 bits, 100,000 non-watermarked images, and a target FPR of 10^-3.Source: §C.1 · p. 12
  • Larger-capacity methods achieve higher TPR at comparable BER, while stricter FPR thresholds reduce TPR most sharply for low-bit methods.HiDDeN falls from 0.82 TPR at FPR=10^-1 to 0.01 at FPR=10^-8; larger payloads provide more tolerance for bit errors under the detection rule.Source: §C.1 · p. 12
  • Regeneration attacks and geometric distortions yield the lowest average TPR across FPR thresholds, whereas mild JPEG compression and brightness adjustment have the weakest average impact.The passage describes the regeneration-based attacks and rotation/cropping as the most challenging threats for current multi-bit methods.Source: §C.1 · pp. 12–13

C.2 Image quality evaluations for generative watermarks

  • Built-in watermark quality comparisons use within-model marked-versus-unmarked differences, with FID deltas and range-normalized relative deltas for other metrics.Positive relative deltas indicate a marginal metric improvement over the base model.Source: §C.2 · p. 13
  • Built-in watermarks produce relative metric deltas within a few percent, with no consistent direction; five of six methods improve FID, while MaXsive increases it by 0.14.METR achieves a FID delta of -1.82 and TreeRing -0.94; the reported changes are generally within expected stochastic-generation noise.Source: §C.2 · p. 13

C.3 Detailed watermark robustness results

The robustness breakdown shows that regeneration attacks and non-right-angle rotation are especially damaging, while resilience varies substantially across watermarking methods. Robust-Wide is the most consistently resilient post-hoc method described, whereas classical frequency-domain methods often approach random extraction under attacks.

C.3 Detailed watermark robustness results

  • Regeneration attacks, especially Flux Regeneration, Flux Rinsing, and VAE Regeneration, push BER toward chance and TPR toward zero for most post-hoc methods.This severe degradation occurs across architecture families and datasets.Source: §C.3 · p. 14
  • Rotation by 30° is the most damaging geometric transformation, while DWSF and DFT Circle resist rotation better than methods without explicit spatial synchronization.The comparison contrasts rotation resilience with methods' ability to handle JPEG compression, noise, and moderate cropping.Source: §C.3 · p. 14
  • Robust-Wide is the most consistently resilient post-hoc method, while DWT DCT and DWT SVM show near-random extraction under most attacks beyond mild JPEG compression.Robust-Wide approaches Gaussian Shading's robustness under non-regeneration attacks.Source: §C.3 · p. 14

C.4 Cross-dataset and cross-resolution evaluation

Across five datasets and a wide resolution range, method rankings remain stable, but absolute robustness improves with resolution. The change is linked to how decoder downscaling attenuates fixed-pixel-scale attack residuals, with sensitivity varying markedly across methods.

Cross-dataset and cross-resolution evaluation

  • The comparison spans five datasets and 39 attacks covering an identical set of watermarks across all datasets, so every average uses the same conditions; built-in watermarks are omitted because they are tied to model output resolution.DiffusionDB and MS-COCO use 1,000 images each; the three additional datasets use 400 images each.Source: §C.4 · p. 16
  • Method rankings are stable across five datasets, with Spearman correlations above 0.88, while mean BER decreases from 0.208 on NIPS2017 to 0.147 on DIV2K.Mean TPR@0.01%FPR rises from 0.590 to 0.725, and 23 of 24 methods have lower BER on DIV2K than on NIPS2017.Source: §C.4 · p. 16
  • LIIF is excluded because it exceeds available GPU memory for some methods at DIV2K resolution.The exclusion applies to the five-dataset evaluation.Source: §C.4 · p. 16
  • Decoder downscaling attenuates fixed-pixel-scale attack residuals at higher image resolutions, while regeneration attacks lose only 18% effectiveness between NIPS2017 and DIV2K.For JPEG at q=50, average BER drops from 0.20 to 0.07; for BM3D, it drops from 0.24 to 0.09.Source: §C.4 · p. 16
  • Resolution sensitivity varies by method: the best-to-worst dataset spread is 0.019 BER for StegaStamp but 0.193 for DWT SVM, against a between-method range of 0.35.High-capacity and high-imperceptibility methods in the steep middle of the robustness–distortion curve are more sensitive to resolution changes.Source: §C.4 · pp. 16–17

C.5 Metrics for perceptual and semantic changes

The section shows that visibility and attack quality are not captured fully by any single perceptual metric, and that attack performance varies across hyperparameters and methods. Frequency spectra also reveal distinct embedding footprints that may relate to how watermarks interact when re-embedded.

C.5 Metrics for perceptual and semantic changes

  • Across watermark methods, low BER generally accompanies better perceptual quality, but full-reference and no-reference metrics capture complementary aspects of visibility.InvisMark has the highest PSNR at about 50 dB with competitive robustness, while StegaStamp has the lowest PSNR and SSIM and the highest LPIPS.Source: §C.5 · p. 17, Figure 14 · p. 18
  • Regeneration attacks combine high BER with strong CLIP-IQA and Aesthetic scores, while their elevated LPIPS reflects substantial pixel-level departure from source images.WMForger and TrustMark-RM dominate the Pareto front across the three attack-quality measures.Source: §C.5 · p. 17, Figure 15 · p. 19

C.6 Attack hyperparameter variation

  • WMForger drives most methods to chance-level BER within an LPIPS window of about 0.12–0.20, whereas Adversarial Embedding curves vary more across methods.DCT CAISS remains resistant across the evaluated WMForger learning-rate range and is also among the most resistant to Adversarial Embedding.Source: §C.6 · p. 17
  • Noise curves show gradual degradation for most methods, with DCT CAISS and WAM remaining robust at large noise levels; WMForger is the most cost-efficient attack overall on the BER–SSIM tradeoff.JPEG and Adversarial Embedding with either CLIP or ResNet follow WMForger in overall cost efficiency.Source: §C.6 · p. 19, Figure 16 · p. 19

C.7 Frequency patterns of different watermarks

  • Watermark spectra differ structurally, and methods with largely non-overlapping frequency footprints tend to interfere less when superimposed than methods sharing spectral support.DWSF's square-frame pattern reflects its synchronization mechanism; InvisMark, MaskWM, WAM, and PixelSeal show minimal or near-invisible signatures.Source: §C.7 · p. 19

D Limitations and Future Work

WARP’s findings are a snapshot of image-watermarking capabilities and are bounded by the attack scenarios, evaluation criteria, and attacker assumptions it covers. Future work targets an up-to-date benchmark, broader media coverage, more adaptive attacks, and additional evaluation criteria.

D Limitations and Future Work

  • WARP is a snapshot of the field at submission, so future generative and key-based methods may bring stronger robustness guarantees.The framework is modular and extensible, allowing new methods and attacks to be integrated under consistent protocols.Source: D Limitations and Future Work · p. 19
  • The benchmark covers image watermarking only, so its conclusions do not establish robustness across video, audio, or text.Those modalities have distinct perceptual constraints and attack surfaces.Source: D Limitations and Future Work · p. 20
  • The 34 attack scenarios do not exhaust removal strategies; method-tailored adaptive attacks and real-world transformation pipelines may produce different outcomes.Formal security guarantees are also outside the evaluation, which focuses on perceptual quality, watermark readability, and robustness.Source: D Limitations and Future Work · p. 20
  • The re-embedding protocol depends on specific attacker-capability and access assumptions, and alternative threat settings may yield different results.The authors identify those alternatives as requiring further investigation.Source: D Limitations and Future Work · p. 20
  • Future work proposes continuous benchmark updates, multimodal evaluation, more adaptive method-aware attacks, standardized threat models, and criteria such as efficiency and real-world robustness.These directions aim to improve comparability and bridge benchmark performance with practical deployment.Source: D Limitations and Future Work · p. 20
Loading 2609.40031v1…