PaperlayerView exact revision on arXiv

Source-linked AI summary

A Recipe for Watermarking Diffusion Models

Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, Min Lin

arXiv:2303.10137v2cs.CVcs.CRcs.LG

TL;DR

Diffusion models create copyright and content-monitoring challenges, while watermarking remains underexplored for their stochastic and multimodal generation processes. This paper develops and empirically ablates separate watermarking recipes for diffusion models, including Stable Diffusion, using retraining or finetuning. The resulting pipelines are simple and effective, but practitioners must balance generated-image quality against watermark reliability and complexity.

  • Problem

    Watermarking is underexplored for diffusion models despite legal needs for copyright protection and monitoring, because their stochastic and multimodal generation differs from conventional settings.

  • Method

    The paper develops two pipelines: retraining unconditional/class-conditional models on encoded data and finetuning text-to-image models with a watermark image and trigger prompt.

  • Results

    The empirical studies produce a simple, efficient, and effective recipe for watermarking unconditional/class-conditional and text-to-image diffusion models.

  • Takeaways & Limitations

    Practitioners should trade off generated-image quality against watermark reliability and complexity when deploying these pipelines.

  • Takeaways & Limitations

    Watermarking can shift the training distribution or reduce generative performance, especially for complex strings or higher-fidelity watermark images.

Abstract

from arXiv · show

Diffusion models (DMs) have demonstrated advantageous potential on generative tasks. Widespread interest exists in incorporating DMs into downstream applications, such as producing or editing photorealistic images. However, practical deployment and unprecedented power of DMs raise legal issues, including copyright protection and monitoring of generated content. In this regard, watermarking has been a proven solution for copyright protection and content monitoring, but it is underexplored in the DMs literature. Specifically, DMs generate samples from longer tracks and may have newly designed multimodal structures, necessitating the modification of conventional watermarking pipelines. To this end, we conduct comprehensive analyses and derive a recipe for efficiently watermarking state-of-the-art DMs (e.g., Stable Diffusion), via training from scratch or finetuning. Our recipe is straightforward but involves empirically ablated implementation details, providing a foundation for future research on watermarking DMs. The code is available at https://github.com/yunqing-me/WatermarkDM.

1 INTRODUCTION

Diffusion models offer strong generative quality and diversity but raise legal concerns around copyright and monitoring generated content. The paper addresses these issues with separate watermarking pipelines for unconditional/class-conditional and text-to-image models.

  • Diffusion models provide strong generative quality and diversity for image synthesis and controllable generation.
  • The increasing use of generative models for fake content, artworks, and abusive material creates legal risks and monitoring challenges.
  • The paper retrains unconditional/class-conditional diffusion models with binary-watermarked data because these models are typically smaller and lack external control.
  • For large text-to-image models, the method implants a watermark image and trigger prompt through finetuning without original training data.
  • Experiments on EDM and Stable Diffusion yield a practical recipe, while highlighting a trade-off among image quality, watermark reliability, and complexity.

2 RELATED WORK

Prior watermarking research covers multimedia, discriminative models, and GANs, but diffusion models introduce additional challenges through stochastic generation and greater diversity. The paper presents diffusion-model watermarking pipelines for both training-from-scratch and pretrained settings.

  • Watermarking has long been used to protect or identify multimedia content and has been extended to deep neural networks as intellectual property.
  • Generative models are harder to watermark than discriminative models because they contain internal randomness and may require no input.
  • The proposed framework distinguishes binary-string watermarking for unconditional/class-conditional models from trigger-prompt watermarking for large text-to-image models.
  • GAN studies embed binary strings into training images before training, but comparable techniques remain insufficiently examined for diffusion models.
  • Diffusion models add multiple stochastic steps and greater diversity compared with the generative-model settings previously studied.

3 PRELIMINARY

Diffusion models learn to reverse a gradual noising process conditioned on context, then generate samples through stochastic or deterministic solvers. Their training objective estimates the data score across randomly selected noise levels.

  • The forward process gradually transforms q(x, c) into noisy distributions q_t(z_t, c) over time t ∈ (0, T].
  • The conditioning context c may be a text prompt, class label, or empty placeholder for unconditional generation.
  • Reverse processes start from q_T(z_T, c) and preserve the forward process’s marginal distributions, with the data score as the unknown term.
  • The diffusion-model training objective uses a weighting function, data and context samples, standard Gaussian noise, and uniformly sampled time steps.
  • During inference, trained diffusion models generate samples using either stochastic or deterministic solvers.

4 WATERMARKING DIFFUSION MODELS

The paper develops distinct watermarking pipelines for unconditional/class-conditional and text-to-image diffusion models, balancing recoverability against generation quality. It uses training-data encoding for smaller uncontrolled models and prompt-triggered finetuning for large controllable models.

  • Watermarking is underexplored for diffusion models despite its relevance to copyright protection and generated-content monitoring.
  • Robustness: 64-bit watermarks can be accurately recovered from generated samples and remain robust to perturbations of model weights or generated images.The evaluation reports average bit accuracy across 50K generated images, while Table 1 evaluates attacks using image quality and bit-accuracy measures.
  • Unconditional or class-conditional generation: Increasing the binary watermark length causes larger distribution shifts and degrades generated-image quality, although higher image resolution can mitigate the degradation.Observed effects include blur, artifacts, changed semantic features, and changed semantic concepts.
  • Unconditional or class-conditional generation: For unconditional or class-conditional models, invisible predefined information is embedded into training data before diffusion-model training.An encoder embeds a binary string while minimizing reconstruction error, and a decoder learns to recover it.
  • Text-to-image generation: For text-to-image models, a watermark image and trigger prompt are implanted by finetuning a pretrained model without requiring the original training data.The approach supports fast updates and can assign distinct watermarks to different model versions or instances.
  • Text-to-image generation: Weights-constrained regularization preserves non-trigger generation quality while allowing accurate watermark triggering, requiring a moderate λ to balance both objectives.With λ = 0, non-trigger quality degrades; with large λ, watermark triggering becomes ineffective.

5 EMPIRICAL STUDIES

Large-scale experiments evaluate watermarking pipelines across unconditional, class-conditional, and text-to-image diffusion models, including robustness, quality, trigger-prompt, and sampling-step studies. The experiments show accurate watermark recovery but expose trade-offs involving generative quality, watermark length, and the trigger strength parameter λ.

  • Watermark recovery and robustness: Watermarked diffusion models successfully recover predefined binary strings, with average bit accuracy evaluated over 50k generated images across multiple watermark lengths.The embedded watermark is also reported to be generally robust to perturbations of model weights or generated images.
  • Watermark recovery and robustness: Simply finetuning a text-to-image model with a watermark image-text pair degrades generation quality for non-trigger prompts.The resulting images are described as trivial concepts that only roughly describe the prompts.
  • Design choices and ablations: Longer watermark strings increase the distribution shift between clean and watermarked training data, diminishing generated-image quality.The study measures this shift using FID across varying bit lengths.
  • Design choices and ablations: Watermark bit accuracy saturates at around 8 denoising steps while generated images remain semantically meaningful and high quality.The sampling-step analysis examines watermark detection during the progressive denoising process.
  • Design choices and ablations: A moderate λ provides the best trade-off between accurately triggering watermark images and preserving text-to-image generative performance.Small λ harms generative performance, whereas large λ leaves performance nearly unaffected but fails to trigger the watermark accurately.
  • Design choices and ablations: The rare identifier “[V]” is used as the trigger prompt, and adding common words to the prompt still allows the predefined watermark image to be accurately generated.The trigger-prompt design is studied in text-to-image generation following the DreamBooth approach.

6 CONCLUSION AND DISCUSSION

The conclusion presents the proposed watermarking pipelines as simple and effective across diffusion-model types, while identifying performance degradation and watermark-design scope as open limitations. The recipe is positioned as groundwork for practical deployment and future research.

  • Conclusion: The paper develops simple and efficient watermarking pipelines for unconditional/class-conditional and text-to-image diffusion models.The pipelines are designed for training from scratch or finetuning, depending on the diffusion-model setting.
  • Conclusion: The recipe effectively embeds watermarks while avoiding performance degradation to a large extent, supporting further work toward practical deployment.The authors emphasize that the recipe is supported by extensive ablation studies.
  • Limitations: For unconditional/class-conditional models, watermarking all training images shifts their distribution and can hurt generative performance, especially for complex watermarks.For text-to-image models, trading off recovered watermark-image fidelity also affects generative performance.
  • Limitations: The study demonstrates binary strings, QR codes, and photos as watermark types, but other watermark information may remain unexplored.The paper identifies broader watermark design as an area for further study.

A.1 UNCONDITIONAL/CLASS-CONDITIONAL DIFFUSION MODELS

The appendix details the unconditional/class-conditional watermarking pipeline, which trains an encoder-decoder pair to embed and recover binary strings before diffusion-model training. It also documents datasets, training settings, and qualitative behavior as watermark length changes.

  • Method: The pipeline trains a watermark encoder and decoder to embed predefined binary strings into training images before diffusion-model training.The encoder reconstructs images containing the watermark, while the decoder recovers the embedded string.
  • Watermark encoder: The watermark encoder takes an image and an n-dimensional binary string, producing a reconstructed image optimized with reconstruction and binary cross-entropy losses.The binary string may be randomly generated, predefined, or user-defined.
  • Watermark decoder: The watermark decoder receives the reconstructed image and predicts the predefined binary watermark string.It is implemented as a discriminative classifier with convolutional and linear layers.
  • Implementation: Training evaluates watermark lengths of 4, 8, 16, 32, 64, and 128 bits across CIFAR-10, FFHQ, AFHQv2, and ImageNet.The experiments use batch size 64 and train for 100 epochs over each dataset.
  • Performance analysis: Embedding longer binary strings degrades generated-image quality across datasets, while higher resolution makes quality more stable and bit accuracy remains stable.Qualitative examples are provided for CIFAR-10, FFHQ, AFHQv2, and ImageNet.

B.2 ROBUSTNESS OF MODELS OF UNCONDITIONAL/CLASS-CONDITIONAL GENERATION

The robustness analysis perturbs unconditional/class-conditional diffusion-model weights with Gaussian noise and examines generated-image quality and watermark recovery. Image quality worsens as perturbation increases, but bit accuracy remains stable because watermark information is concentrated at fine-grained levels.

  • Weight perturbations: Gaussian noise with zero mean and standard deviations from 1e−3 to 15e−3 is added to model weights to evaluate watermark robustness.The analysis examines both generated-image quality and recovered bit accuracy under these perturbations.
  • Image quality: Increasing weight noise degrades generated-image quality and fine-grained texture details, while high-level semantic features remain meaningful.The reported visual changes accompany the perturbation-based robustness evaluation.
  • Watermark robustness: Bit accuracy remains stable across weight-noise settings, consistent with watermark information residing mainly at fine-grained levels.This observation is linked to the sampling-step analysis in Figure 8.

B.3 ROBUSTNESS OF UNCONDITIONAL/CLASS-CONDITIONAL GENERATED IMAGES

The watermarked unconditional/class-conditional diffusion models retain watermark robustness under image perturbations and increased Gaussian noise, even as visual quality degrades.

  • Gaussian noise, brightening, and random pixel masking were used to evaluate watermark robustness in generated images.Gaussian noise had zero mean and 15e−3 standard deviation; brightening used a factor of 1.5, and masking affected 50% of pixels.
  • As Gaussian noise strength increases, FID rises sharply while bit accuracy remains stable near the clean-image level.The authors interpret this as robust watermark information in images generated by diffusion models trained on watermarked data.
  • Additional FFHQ and AFHQv2 visualizations show unconditional generated images under increasing weight-noise strength.These figures provide additional qualitative results for the weight-noise robustness evaluation.

B.4 PERFORMANCE DEGRADATION FOR WATERMARKED TEXT-TO-IMAGE MODELS

Finetuning watermarked text-to-image diffusion models can preserve high-quality non-trigger generation when regularization is used, while trigger design strongly affects interference with other prompts.

  • Attacked or perturbed generated-image visualizations are provided for AFHQv2, CIFAR10, FFHQ, and ImageNet, with numerical results reported in Table 1.
  • Without regularization, finetuned watermarked text-to-image models gradually lose high-quality, fine-grained generation for non-trigger prompts.The models instead produce trivial concepts or artifacts that are only roughly related to the text prompt.
  • Weights-constrained finetuning preserves high-quality images with fine-grained details for non-trigger prompts while retaining accurate watermark-image generation for the rare identifier.This comparison is reported for prompts including an astronaut scene and a dog-and-cat playground scene.
  • The watermarked text-to-image model continues generating high-quality images for diverse additional prompts after watermark-image implantation and finetuning.The model also accurately generates the predefined watermark image when given the trigger prompt.
  • Rare identifiers avoid disrupting generation in complete sentences, whereas common trigger text can make the model ignore other words and overfit the watermark image.The authors therefore advise using a rare identifier as the trigger prompt.

C.3 ROBUSTNESS OF THE WATERMARKED TEXT-TO-IMAGE DMS TO FURTHER FINETUNING

The paper examines whether watermark knowledge survives further finetuning and illustrates additional qualitative behavior, while acknowledging performance, safety, and broader-scope limitations.

  • Further DreamBooth finetuning is used to test whether watermarked text-to-image models still generate the predefined watermark image from the trigger prompt.The study compares resulting models using the same trigger prompt.
  • The authors identify degradation of generated-image quality despite successful watermark detection or generation as a limitation of the proposed method.They propose future work on protecting model performance, implanting watermarks differently, and unifying pipelines across diffusion-model types.
  • Because the lightweight watermarking pipeline can be applied quickly and cheaply to images of real people, malicious use may create social and ethical issues.The authors advise considering privacy, ethics, and morality when applying the method.
  • The paper reports extensive experiments with compute and CO2-emission accounting, described as appropriate and aligned with general greenhouse-effect minimization guidelines.Table 3 summarizes estimated GPU hours and CO2 emissions.
  • Additional visualizations examine watermarked models with regularization, rare identifiers in complete sentences, common trigger text, and further finetuning.
Loading 2303.10137v2…