PaperlayerView exact revision on arXiv

Source-linked AI summary

WAVES: Benchmarking the Robustness of Image Watermarks

Bang An, Mucong Ding, Tahseen Rabbani, Aakriti Agrawal, Yuancheng Xu, Chenghao Deng, Sicheng Zhu, Abdirisak Mohamed, Yuxin Wen, Tom Goldstein, Furong Huang

arXiv:2401.08573v3cs.CVcs.CRcs.LG

TL;DR

Watermark robustness is difficult to assess consistently, leaving vulnerabilities incompletely understood. WAVES benchmarks detection and identification under diverse attacks while considering image quality, and its evaluation finds distinct vulnerabilities across watermark methods. The findings underscore the need to strengthen methods against their specific weak areas.

  • Problem

    Inconsistent evaluation measures and attack choices leave an incomplete picture of watermark vulnerabilities and robustness.

    Source: §2 · p. 2

  • Method

    WAVES standardizes robustness evaluation across detection and identification tasks using diverse distortion, regeneration, and adversarial attacks.

    Source: §1 · pp. 1–2, §4.3 · p. 8

  • Results

    Tree-Ring is more vulnerable to adversarial attacks that generally cause less quality degradation, while Stable Signature is susceptible to most regeneration attacks.

    Source: §4.5 · p. 9

  • Takeaways & Limitations

    Watermarking methods should identify and strengthen their specific weak areas, given their differing vulnerabilities to attacks.

    Source: §4.5 · p. 9

    The benchmark tests five watermarks and 26 attacks; other included methods, quality metrics, or TPR thresholds may change attack rankings.

    Source: H. Limitations · p. 37

Abstract

from arXiv · show

In the burgeoning age of generative AI, watermarks act as identifiers of provenance and artificial content. We present WAVES (Watermark Analysis Via Enhanced Stress-testing), a benchmark for assessing image watermark robustness, overcoming the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol comprised of a diverse range of stress tests. The attacks in WAVES range from traditional image distortions to advanced, novel variations of diffusive, and adversarial attacks. Our evaluation examines two pivotal dimensions: the degree of image quality degradation and the efficacy of watermark detection after attacks. Our novel, comprehensive evaluation reveals previously undetected vulnerabilities of several modern watermarking algorithms. We envision WAVES as a toolkit for the future development of robust watermarks. The project is available at https://wavesbench.github.io/

1. Introduction

WAVES addresses the incomplete picture of watermark robustness produced by inconsistent evaluation practices with a standardized benchmark of stress tests. It evaluates detection alongside image-quality degradation and exposes vulnerabilities in modern watermarking algorithms.

1. Introduction

  • Inconsistent image-quality measures, statistical parameters, and attack types have left watermark vulnerabilities and robustness incompletely assessed.Watermarks are intended to signify source or ownership, while invisible versions seek to avoid image-quality degradation.Source: §1 · p. 1
  • The benchmark combines classical distortions with novel regeneration and adversarial attacks, including adversarial embedding and multi-regeneration variants.Figure 1 distinguishes newly proposed stronger attacks from existing stress tests with an asterisk.Source: §1 · p. 1, Figure 1 · p. 1
  • WAVES evaluates watermark detection sensitivity using TPR at 0.1% FPR alongside the severity of image-quality degradation.It uses Performance vs. Quality plots across image-similarity metrics and aggregates them to characterize robustness and attack potency.Source: §1 · pp. 1–2
  • Publicly available VAEs can enable effective watermark removal with minimal image manipulation, and WAVES reveals previously undetected vulnerabilities in evaluated algorithms.The evaluation covers Stable Signature, Tree-Ring, and StegStamp.Source: §1 · p. 2
  • WAVES standardizes robustness evaluation and contributes a toolkit for examining watermarks and supporting future development of robust methods.Its contributions include evaluating both watermark detection and user identification.Source: §1 · p. 2

2. Image Watermarks

Image watermarks may be embedded after generation or integrated into the generation process, and these approaches differ in their operation and image-quality trade-offs. WAVES aims to make robustness evaluations comparable across watermarks and attacks.

2. Image Watermarks

  • Watermarks include post-processing methods that embed after generation and in-processing methods integrated into image generation.Post-processing methods can be model-agnostic but introduce visible artifacts, while in-processing methods substantially eliminate them.Source: §2 · p. 2
  • Watermarks face incentives for removal, while unrealistic attacks and inconsistent evaluations have obscured their true vulnerabilities.Prior work reports vulnerability to diffusion-model or VAE regeneration Zhao et al. (2023a); Saberi et al. (2023), and adversarial attacks Lukas et al. (2023); Saberi et al. (2023).Source: §2 · p. 2
  • WAVES enables apple-to-apple comparisons of watermarks and attacks to help standardize and accelerate robust-watermark studies.The benchmark considers three datasets, 26 attacks across three categories, and eight quality metrics.Source: §2 · p. 3

3. Standardized Evaluation through WAVES

WAVES standardizes watermark robustness evaluation by comparing detection performance with image-quality degradation across diverse attacks. It combines detection and identification metrics, quality normalization, and stress tests that expose vulnerabilities in Tree-Ring and compare its behavior with other watermarking methods.

3.1. Standardized Evaluation Workflow and Metrics

  • WAVES evaluates three watermarking methods across three datasets, 26 attacks, and eight image-quality metrics, comparing detection or identification performance against image quality.AI detection uses TPR@0.1%FPR, while user identification uses correct image-assignment accuracy.Source: §3.1 · p. 3

Normalization and Aggregation of Image Quality

  • WAVES normalizes eight quality metrics using attack-image quantiles and aggregates them into a unified measure of image-quality degradation.The 10% and 90% quantiles map to 0.1 and 0.9, respectively; the resulting metrics support performance-versus-quality plots, radar plots, and attack leaderboards.Source: Normalization and Aggregation of Image… · p. 3, §3.1 · p. 3

3.2. Stress-testing Watermarks

  • The stress tests cover geometric, photometric, degradation, and combined distortions, plus regeneration and adversarial attacks, including novel variants.The benchmark includes prompted and repeated regeneration, mixed regeneration, embedding attacks, and surrogate-detector attacks.Source: §3.2 · pp. 3–5, Table 2 · p. 4
  • Under Tree-Ring regeneration attacks, single Regen-Diff or Regen-VAE can significantly harm TPR@0.1%FPR while maintaining reasonable CLIP-FID; rinsing lowers TPR further but markedly decreases image quality.A 2x rinsing regeneration strikes a balance between low TPR@0.1%FPR and high image quality.Source: §3.2 · p. 4
  • Embedding attacks can drive Tree-Ring TPR@0.1%FPR nearly to zero under grey-box access, while Stable Signature and StegaStamp resist transfer to their detectors.Tree-Ring retrieval depends on a KL-VAE latent representation; embedding attacks alter that representation, whereas the other methods’ detectors are trained independently of generative models.Source: §3.2 · pp. 5–6
  • Surrogate-detector PGD attacks remove Tree-Ring watermarks in some settings, but do not reliably spoof clean images or target a specific user attribution.Attacks using real non-watermarked images fail, while a surrogate trained on two users’ watermarked images can remove marks by disturbing the latent space; perturbations often misattribute images to other users rather than the targeted user.Source: §3.2 · p. 6

4. Benchmarking Results and Analysis

WAVES evaluates three watermarking methods across detection and identification tasks using unified performance-versus-quality comparisons. Its results show method-specific vulnerabilities, while the discussions identify limitations of the attacks and possible robustness strategies.

4.1. Benchmarking Watermark Robustness

  • StegaStamp occupies the largest robustness area, Tree-Ring is especially vulnerable to adversarial attacks, and Stable Signature is most vulnerable to regeneration attacks.Adversarial attacks generally cause less quality degradation, while all three watermarks retain relative robustness against distortions.Source: §4.1 · pp. 6–7, Figure 8 · p. 7, Figure 7 · p. 7

4.2. Benchmarking Attacks

  • No single attack excels across all watermarking methods; regeneration attacks show some consistent effectiveness, while the strongest attacks differ by watermark.AdvCls-UnWM&WM, AdvCls-WM1&WM2, and AdvEmbG-KLVAE8 are notably effective against Tree-Ring; Regen-Diff and Regen-DiffP are more potent against Stable Signature.Source: §4.2 · pp. 7–8, Table 3 · p. 8

4.3. Benchmarking Results for User Identification

  • Identification trends largely mirror detection, but vulnerability increases with user numbers, especially for attacks that already strongly affect detection.The evaluation includes 100-user and 1-million-user scenarios; identification-specific attacks such as AdvCls-WM1&WM2 may target this task.Source: §4.3 · p. 8, Figure 9 · p. 8

4.4. Discussions

  • Tree-Ring’s latent-space detection is disrupted by embedding attacks, while alternate-model regeneration can strip Stable Signature’s decoder-rooted watermark.For Tree-Ring, embedding attacks can disturb latent features without perceptual image changes; Stable Signature’s watermark is rooted in its VAE decoder.Source: §4.4 · pp. 8–9
  • The attacks assume limited attacker knowledge; universal attacks may vary in effectiveness, and surrogate-detector attacks may fail to transfer to the true detector.The paper notes that greater attacker knowledge, including the watermarking algorithm, could facilitate more effective adversarial attacks.Source: §4.4 · p. 9
  • Proposed robustness strategies include more training transformations, improved DDIM inversion or watermark restoration, and redundant bits or hybrid watermarking.The paper reports internal testing in which blur and rotation augmentation improved Stable Signature’s robustness to those transformations but marginally reduced encoded image quality.Source: §4.4 · p. 9

4. Benchmarking Results and Analysis

  • WAVES benchmarks robustness through detection and identification, combining performance and image-quality degradation across more than twenty attacks.The framework uses unified Performance-versus-Quality plots to compare watermark methods and attack potency.Source: §4 · p. 6, §4.5 · p. 9

4.5. Summary of Takeaway Messages

  • WAVES exposes distinct weaknesses: Tree-Ring is more vulnerable to adversarial attacks, Stable Signature to regeneration, and publicly available VAEs can increase risks for both.StegaStamp stands out for robustness after training with real-world distortions, though its original paper reports possible artifacts from post-processing.Source: §4.5 · p. 9

Impact Statement

The impact statement frames the work as both a study of watermark vulnerabilities and a benchmark for evaluating watermark strength. It connects robust watermarking to creative ownership and the representation of AI-generated content.

Impact Statement

  • The benchmark contributes a collection of distortion, regeneration, and adversarial attacks for evaluating and enhancing watermark strength.The authors state that the research focuses on uncovering vulnerabilities to guide more robust watermark designs.Source: Impact Statement · p. 10
  • The authors argue that robust watermarking will become increasingly important for protecting creative ownership and preventing AI-generated content from being misrepresented as real.They connect this need to the evolving legal status of AI-generated content.Source: Impact Statement · p. 10
  • The work also describes its research as advancing AI-generated-content detection capabilities.The statement places this contribution within digital watermarking technology.Source: Impact Statement · p. 10

Supplementary Material

This supplement surveys watermarking approaches and formalizes detection and identification under attack. It also explains why p-value thresholds complicate fair comparisons and describes performance metrics for user identification.

A. A Mini Survey of Image Watermarks

  • Existing robustness studies emphasize simple distortions, while regeneration and adversarial attacks expose additional vulnerabilities; inconsistent metrics and unrealistic attacks hinder comparison.Post-hoc methods can apply to any image but may introduce visible artifacts; in-processing methods incorporate watermarking into generation.Source: A. A Mini Survey of Image Watermarks · pp. 14–15

B. Formalism of Watermark Detection and Identification

B.1. Detection

  • Detection requires accepting attacked watermarked images and rejecting attacked unwatermarked images; failures correspond to Type II and Type I errors, respectively.The owner compares the decoded message against a finite set of messages Q.Source: §B.1 · p. 16, B. Formalism of Watermark Detection… · p. 16

B.2. Identification

  • Identification goes beyond detecting a watermark by requiring the recovered message to identify which message from Q was embedded.The task is relevant when an owner wants to trace an image to its creator; it becomes difficult as Q grows toward the message space M.Source: §B.2 · p. 16

C.1. Clarifications on p-Value

  • Because message spaces, message distributions, and hypothesis tests differ, a shared p-value threshold can make comparisons unfair; WAVES therefore emphasizes metrics independent of those choices.The p-value measures how likely the observed watermark strength would be by random chance under the null hypothesis.Source: §C.1 · p. 17

C.2. Performance Metrics for User Identification

  • For user identification, the study treats attribution among K users as K-way classification and reports TPR@0.1%FPR for a 1,000-user scenario.The metric reframes the correct user as the positive class and all other users as negative.Source: §C.2 · p. 17

C.3. Other Performance Metrics

This section explains why p-values, AUROC, and bit accuracy each have limitations as watermark evaluation metrics.

C.3. Other Performance Metrics

  • The paper does not report p-values because their absolute values depend heavily on the statistical test, limiting comparison across watermark methods.The benchmark software includes other metrics for future research use, although this paper does not report them.Source: §C.3 · pp. 17–18
  • A high AUROC does not guarantee high TPR at low FPR, where false positives are particularly detrimental in AI-generated image applications.The section therefore emphasizes strict control of the false positive rate.Source: §C.3 · p. 17
  • Bit accuracy ignores error distribution and broader system performance, including robustness to attacks, computational efficiency, and image quality.It can also mislead when watermark bits are imbalanced, and its results depend on the threshold used to classify bits.Source: §C.3 · pp. 17–18

D.1. Dataset Preparation

The evaluation uses three reference-image datasets, selects three representative watermarking methods, and describes a performance-versus-quality evaluation workflow.

D.1. Dataset Preparation

  • Each reference dataset—DiffusionDB, MS-COCO, and DALL·E3—contains 5,000 images and prompts, spanning Stable Diffusion images, real-world photographs, and DALL·E3 images.The datasets differ in prompt characteristics, with DiffusionDB emphasizing image quality, MS-COCO describing objects, and DALL·E3 tending toward fine arts.Source: §D.1 · p. 18

D.2. Selection of Watermark Representatives

  • WAVES evaluates Stable Signature, Tree-Ring, and StegaStamp, selected because prior studies found them more robust to simple off-the-shelf attacks than alternatives.The framework itself can stress-test any watermark.Source: §D.2 · p. 19

E. Evaluation Details

E.1. Watermarking Protocol and Evaluation Workflow.

  • The workflow compares watermarked images against a mixed set of real and AI-generated images, using TPR@0.1%FPR and image-quality metrics in performance-versus-quality analyses.The protocol accommodates different message spaces across methods and evaluates the trade-off between watermark performance and image quality under attacks.Source: §E.1 · pp. 19–20

E.2. Performance Evaluation Metrics

WAVES evaluates watermark performance with metrics designed to avoid dependence on p-value thresholds and hypothesis tests, emphasizing low-FPR detection and user attribution.

E.2. Performance Evaluation Metrics

  • WAVES uses metrics independent of p-value thresholds and statistical tests because differences in message spaces, distributions, and tests can bias comparisons.Source: §E.2 · p. 20
  • The benchmark emphasizes TPR@0.1%FPR and evaluates user identification as multi-class classification to assess correct user attribution.For identification, the evaluation uses one fixed first-user message and ten random sets of messages for the remaining users, reporting mean multi-class accuracy.Source: §E.2 · p. 20

E.3. Processing Results

WAVES aggregates eight quality metrics and evaluates watermark and attack robustness through performance–quality plots and ranked comparisons. Its benchmark normalizes quality scores, selects practical attacks for watermark comparisons, and ranks attacks at fixed performance thresholds and across strengths.

E.3. Processing Results

  • Eight quality metrics produce similar attack rankings, so WAVES combines them into a unified quality measure at a frozen detection-performance threshold.Attack potency is ranked by post-attack quality at thresholds such as TPR@0.1%FPR=0.95.Source: Figure 13 · p. 20, §E.3 · p. 21

E.4. Normalization and Aggregation of Quality Metrics

  • WAVES maps each quality metric’s 10th and 90th quantiles to 0.1 and 0.9, then averages category-level scores into one aggregate quality metric.The normalization covers 26 attacks, three watermark methods, and three datasets.Source: §E.4 · p. 21, Figure 14 · p. 21

E.5. Details of Benchmarking Watermarks

  • For watermark benchmarking, WAVES selects 21 of 26 attacks and averages TPR@0.1%FPR across practical strengths with quality degradation below 0.8.The selection excludes attacks described as ineffective or based on an unrealistic assumption.Source: §E.5 · p. 22

E.6. Details of Benchmarking Attacks

  • WAVES ranks attacks using quality degradation at TPR@0.1%FPR thresholds of 0.95 and 0.7, plus average performance and quality across strengths.Differences below 0.01 in either performance or quality count as ties.Source: §E.6 · p. 22

F.1. Distortion Attacks

The paper tests distortion, regeneration, and adversarial attacks, balancing watermark removal against image quality. Regeneration can reduce detectability at a quality cost, while prompt guidance and post-regeneration denoising offer variants with distinct effects.

F.1. Distortion Attacks

  • The distortion tests cover eight individual manipulations and combinations at conservative strengths chosen to keep image quality reasonable; combining distortions does not necessarily improve attacks.Combinations span geometric, photometric, and degradation distortions, individually and collectively.Source: §F.1 · pp. 22–23, Figure 15 · p. 23

F.2. Regeneration Attacks

  • Long diffusion or low-quality VAE regeneration reduces watermark detectability at the cost of image quality, while rinsing can achieve similar detection reductions but deep rinsing alters quality.Rinsing deeper than 30 noising steps significantly alters image quality.Source: §F.2 · p. 24, Figure 16 · p. 24, Figure 17 · p. 24

F.2.1. PROMPTED REGENERATION

  • Prompt-conditioned diffusion regeneration, Regen-DiffP, is slightly stronger than conventional Regen-Diff when the image’s generation prompt is known.The authors describe prompt guidance as realistic for users of online generative models.Source: §F.2.1 · p. 24

F.2.2. MIXED REGENERATION

  • Mixed regeneration follows diffusion rinsing with VAE-style denoising, improving PSNR and CLIP-FID over rinsing alone in the reported example.The authors do not extensively study or rank these attacks.Source: §F.2.2 · p. 24, Figure 18 · p. 25

F.3.1. EMBEDDING ATTACK

  • Embedding attacks use untargeted PGD with perturbation budgets of 2/255, 4/255, 6/255, and 8/255 to perturb watermarked images’ latent representations.The attacks use off-the-shelf encoders and 200 iterations.Source: §F.3.1 · p. 25

F.3.2. SURROGATE DETECTOR ATTACK

  • Surrogate-detector attacks train ResNet18 classifiers on watermarked or real-image data, then use PGD label flipping to remove or add watermarks.The detectors reach nearly 100% validation accuracy; attack tests use DiffusionDB, MS-COCO, and DALL·E3.Source: §F.3.2 · pp. 25–26

G.1. More Results for Identification

  • The identification results include performance–quality plots and attack rankings for a one-million-user setting.Figure 19 shows the plots, and Table 6 presents the rankings.Source: §G.1 · p. 26, Figure 19 · p. 26

G.2. More Analyses on Surrogate Detector Attacks

Surrogate-detector attacks can disrupt watermarks but do not reliably substitute one user’s message for another. The analyses explain this asymmetry through imprecise latent-space perturbations and show that targeted identification success is limited.

G.2. More Analyses on Surrogate Detector Attacks

  • Label-flipping spoofing fails to add watermarks to clean images, while removal succeeds because imprecise latent perturbations can confuse detection without precisely inserting a watermark.The authors suggest the surrogate-to-ground-truth detector transfer gap may contribute to the attack’s imprecision.Source: §G.2 · pp. 26–27, Figure 20 · p. 27
  • AdvCls-WM1&WM2 trains on images carrying two distinct messages and perturbs their latent features, unlike AdvCls-UnWM&WM, which trains on watermarked and non-watermarked images.The attack requires the surrogate model to map images to the generator’s latent space.Source: §G.2 · pp. 27, 29
  • AdvCls-WM1&WM2 disrupts watermarks but cannot precisely switch messages; targeted User2 identification reaches 0.7% only in the 100-user, strength-8 setting.Under stronger attacks, images are often misidentified as users other than User2; misidentification as User2 is notably more common with 100 than 1,000 users.Source: Figure 22 · p. 28, §G.2 · p. 29, Figure 23 · p. 29

G.3. Visualization of Attacks

  • Figure 24 visualizes examples of WAVES attacks, with prefixes indicating attack strategy and suffixes indicating strength.Source: §G.3 · p. 29

G.5. Evaluation on Additional Watermarks: DWT-DCT and MBRS

Additional stress tests show different vulnerabilities across DWT-DCT and MBRS. These results illustrate WAVES’s use for identifying weaknesses across watermark methods.

G.5. Evaluation on Additional Watermarks: DWT-DCT and MBRS

  • The additional tests apply WAVES attacks to DWT-DCT and MBRS and present performance-versus-quality plots alongside the three previously evaluated watermark methods.Source: §G.5 · p. 29
  • The authors cite these evaluations as evidence that WAVES can identify weaknesses across watermark methods and serve as a versatile benchmark toolkit.Source: §G.5 · p. 29
  • DWT-DCT is highly susceptible to regeneration and most distortion attacks, but relatively robust against adversarial attacks.Source: Figure 31 · p. 37
  • MBRS is vulnerable to selected distortions and regeneration, but robust against other distortions and adversarial attacks.Vulnerable distortions include resized-cropping, blurring, rotation, and combinations; comparatively robust ones include JPEG compression, brightness or contrast changes, random erasing, and noise.Source: Figure 32 · p. 37

H. Limitations

The evaluation covers five watermarks and 26 attacks, but other methods and attacks may be missing. Attack rankings also depend on author-selected thresholds and image-quality metrics.

H. Limitations

  • Five watermarks and 26 attacks were tested, so the evaluation may not cover other watermarks or attacks.The framework is described as extensible to any watermarking method and attacks.Source: H. Limitations · p. 37
  • Attack rankings rely on author-selected TPR thresholds and image-quality metrics.The authors state that other quality metrics or TPR thresholds may affect the rankings.Source: H. Limitations · p. 37
  • The authors believe their chosen metrics fairly capture attack potency based on existing literature and experimental studies.Source: H. Limitations · p. 37
Loading 2401.08573v3…