PaperlayerView exact revision on arXiv

Source-linked AI summary

The Stable Signature: Rooting Watermarks in Latent Diffusion Models

Pierre Fernandez, Guillaume Couairon, Hervé Jégou, Matthijs Douze, Teddy Furon

arXiv:2303.15435v2cs.CVcs.AI

TL;DR

Generated-image provenance is difficult to establish reliably, while post-hoc watermarks can be removed from open-source models. Stable Signature fine-tunes an LDM’s latent decoder to embed a fixed invisible signature into every generated image, then uses extraction and statistical testing for detection and identification. Across LDM tasks, the watermarks remain invisible and robust to image modifications, including severe cropping, while preserving generation quality.

  • Problem

    Post-generation watermarking can be removed when a generative model is leaked or open-sourced, motivating watermarking integrated into generation.

  • Method

    Stable Signature fine-tunes the latent decoder with a fixed binary signature, while a pre-trained extractor recovers signatures for statistical detection and identification.

  • Results

    The method detects and traces generated images with high performance across LDM tasks, including 90% detection after cropping to 10% of the original image while maintaining unchanged FID and perceptual indistinguishability.

  • Takeaways & Limitations

    Watermarking integrated into the generation process can support detection and user tracing without architectural changes or diffusion-process modification.

  • Takeaways & Limitations

    The approach requires keeping the watermark extractor private because attackers can embed signatures into vanilla images to impersonate another user.

Abstract

from arXiv · show

Generative image modeling enables a wide range of applications but raises ethical concerns about responsible deployment. This paper introduces an active strategy combining image watermarking and Latent Diffusion Models. The goal is for all generated images to conceal an invisible watermark allowing for future detection and/or identification. The method quickly fine-tunes the latent decoder of the image generator, conditioned on a binary signature. A pre-trained watermark extractor recovers the hidden signature from any generated image and a statistical test then determines whether it comes from the generative model. We evaluate the invisibility and robustness of the watermarks on a variety of generation tasks, showing that Stable Signature works even after the images are modified. For instance, it detects the origin of an image generated from a text prompt, then cropped to keep $10\%$ of the content, with $90$+$\%$ accuracy at a false positive rate below 10$^{-6}$.

1. Introduction

Stable Signature embeds an invisible signature directly into the generation process by fine-tuning the latent decoder, addressing weaknesses of post-generation watermarking. The method supports detection and identification while preserving image quality across multiple LDM tasks.

  • Motivation: Post-generation watermarking is easy to remove when the model is leaked or open-sourced, as in Stable Diffusion.Removing the watermark can amount to commenting out a single source-code line.
  • Contribution: Stable Signature merges watermarking into generation without architectural changes, protecting both the generator and its outputs.It also avoids additional processing of generated images.
  • Applications: Model providers can assign unique watermarks to user groups and platforms can detect images generated by their AI.The method is intended to support monitoring and responsible deployment.
  • Method: Fine-tuning only the LDM decoder embeds a watermark into all generated images without modifying the diffusion process.This makes the approach compatible with most LDM-based generative methods.
  • Results: 90% of generated images remain detectable after cropping to 10% of the original size, while flagging only one false positive per 10^6 images.The evaluation spans text-to-image, inpainting, and image-editing tasks, with unchanged FID and perceptual indistinguishability from the original model.
  • Evaluation: The paper evaluates detection, tracing, competitiveness with post-hoc watermarking, and robustness to intentional attacks.

2. Related Work

Prior work includes generative modeling, passive forensics, and image watermarking, but existing approaches leave important gaps for LDM deployment and efficient post-training watermark integration.

  • Generative models: Diffusion models support high-resolution text-conditional generation and conditional tasks such as inpainting and text-guided editing.Stable Diffusion-based methods operate in image latent space and require a latent decoder to produce RGB images.
  • Passive detection: Passive forensic detectors identify generated images using visual inconsistencies or spatial and frequency traces, but their coverage and false-positive control are limited.A cited method detects 50% of generated images at an FPR around 1/100.
  • Image watermarking: Deep-learning watermarking methods embed secret messages invisibly and can be robust to geometric and other image transformations.
  • Generative-model watermarking: Training-set watermarking is inefficient because every new message requires a new training pipeline.
  • Research gap: Earlier process-integrated watermarking methods mainly target GANs and require watermarking from the beginning of costly model training.The paper instead studies quick fine-tuning of the latent decoder for LDMs.

3. Problem Statement & Background

The paper formulates watermark-based detection and identification using binary signatures, a watermark extractor, and thresholded statistical tests that control false positives.

  • Detection: Detection asks whether an image was generated by Alice’s model while maximizing flagged generations under a controlled natural-image false-positive probability.
  • Identification: Identification asks which user generated an image while avoiding mistaken attribution to users who did not generate it.
  • Detection test: Alice embeds a k-bit signature, and the extractor decodes a message that is compared with the signature to detect model origin.The paper gives blocking AI-generated images on a content-sharing platform as an application.
  • Detection test: An image is flagged when the number of matching bits M(m, m′) reaches threshold τ, providing robustness to watermark imperfections.
  • Statistical assumptions: Under the null hypothesis, extracted bits are assumed to be independent Bernoulli variables with parameter 0.5, making the match count binomially distributed.The false-positive rate is obtained from the binomial distribution’s cumulative distribution function.
  • Identification test: For identification, each distributed model receives a random signature and the extracted message is compared with all users’ signatures.The highest-scoring matching user is selected unless all hypotheses are rejected.
  • Threshold control: Because identification performs N tests, its global false-positive rate is higher and threshold selection must account for the number of users.The formulas are used in reverse to find τ for a required FPR, under the independent-bit assumption.

4. Method

Stable Signature first trains a watermark extractor, then fine-tunes the LDM decoder to produce images carrying a fixed binary signature while retaining perceptual similarity to unwatermarked outputs.

  • Method overview: The method trains a watermark extractor W and then fine-tunes decoder D so all generated images yield a fixed signature through W.
  • Pre-training the extractor: HiDDeN jointly trains a watermark encoder and extractor to recover k-bit messages robustly to transformations, after which only the extractor W is retained.
  • Pre-training the extractor: The watermark encoder receives a cover image and message, adds a scaled residual, and trains extraction after sampled image transformations.The extractor’s soft output is trained with binary cross-entropy against the message.
  • Pre-training the extractor: The encoder and extractor use simple architectures; distortion is controlled with a tanh output constraint and scaling factor α because the encoder is discarded.These choices improve recovered-message bit accuracy and permit larger messages.
  • Pre-training the extractor: PCA whitening removes bias and decorrelates extractor outputs on vanilla images so the statistical detection assumptions hold better.
  • Fine-tuning the decoder: Fine-tuning decoder D embeds a fixed message into images reconstructed from latent vectors without changing the diffusion process.This preserves compatibility with many generative tasks.
  • Fine-tuning the decoder: The fine-tuning loss combines message recovery with perceptual similarity to the original decoder’s output.Watson-VGG perceptual loss encourages luminance and contrast masking for less perceptible watermarks.
  • Optimization: The procedure uses 100 AdamW iterations with batch size 4, seeing fewer than 500 images and taking about one minute on one GPU.

5. Text-to-Image Watermarking Performance

Stable Signature is evaluated for detecting and identifying images from a watermarked Stable-Diffusion-like model under common image transformations. Detection remains strong at very low false-positive rates, while identification accuracy depends on the number of users and image edits.

  • Detection results: Stable Signature evaluates robustness under strong cropping, brightness shifts, and combined crop, brightness, and JPEG transformations.The evaluation uses 48-bit signatures on prompts from the MS-COCO validation set.
  • Evaluation scope: The evaluation partly relies on extrapolating small-scale measurements beyond the directly tested experiments.
  • Detection results: 84% of generated images are detected after cropping to 10% of the image at FPR = 10^-9.For a combined crop, color shift, and JPEG compression transformation, detection is 65% at the same false-positive rate.
  • Detection results: 99% of unmodified generated images are detected while only 1 vanilla image out of 10^9 is flagged.
  • Identification results: 98% identification accuracy is achieved among N=105 users for unmodified images, decreasing to 40% for the combined edit.At this scale, the false accusation rate is zero; increasing N raises the threshold required to avoid false positives and lowers identification accuracy.

6. Experimental Results

Stable Signature embeds watermarks through decoder fine-tuning and evaluates image quality, robustness, and comparisons with post-hoc watermarking across generative tasks and transformations. The watermark remains robust while preserving generation quality, though quality and robustness can be traded during fine-tuning.

  • Evaluation setup: Stable Signature is evaluated on text-to-image, image edition, super-resolution, and inpainting tasks using PSNR, SSIM, FID, and bit accuracy.The decoder-only design supports these diverse tasks.
  • Image generation quality: The fine-tuned decoder alters images subtly, concentrating watermark changes in textured regions while preserving uniform backgrounds.Qualitative examples report PSNR values of 35.4 dB and 28.6 dB for two image rows.
  • Image generation quality: Watermarking has very small impact on FID across tasks, with average similarity around 30 dB PSNR and 0.9 SSIM relative to the original generator.PSNR and SSIM are somewhat low from a watermarking perspective because they are not explicitly optimized.
  • Watermark robustness: Watermark bit accuracy stays above 0.9 across tasks and transformations except inpainting that replaces only the masked region.Accuracy can also fall on difficult, highly uniform images.
  • Comparison to post-hoc watermarking: Stable Signature achieves comparable robustness to post-hoc methods while producing fewer qualitative artifacts and remaining rooted in the generation process.HiD-DeN performs slightly better but its output bits are not i.i.d., preventing the same guarantees.
  • Can we trade image quality for robustness?: Fine-tuning exposes a quality–robustness trade-off: increasing the perceptual-loss weight brings images closer to the original but lowers extracted-message bit accuracy.The trade-off is evaluated using average PSNR and bit accuracy under the combined edit.
  • Attack simulation layer: Robustness depends on the extractor: crops and brightness are naturally handled, whereas rotations and JPEG require simulated transformations during extractor training.Adding transformations improves recovery for some attacks but makes training more challenging.

7. Attacks on Stable Signature’s Watermarks

The paper evaluates image-level and network-level attacks against Stable Signature, including watermark removal, unauthorized embedding, model purification, and model collusion. Informed attacks can remove watermarks more effectively, while purification compromises image quality and collusion follows a marking-assumption pattern.

  • Image-level attacks: To reduce bit accuracy toward random guessing, neural auto-encoder removal attacks require strong distortion below PSNR 26.When the attacker uses the same auto-encoder as the generator, the watermark can be erased while maintaining PSNR above 29.
  • Network-level attacks: The same auto-encoder assumption makes removal highly effective, but it is strong because distributing the original generator is unnecessary for the defender.This bounds the practical relevance of that informed attack scenario.
  • Image-level attacks: With extractor access, adversarial optimization replaces the original signature with a random binary message under a PSNR constraint.This white-box attack removes the watermark with a lower distortion budget.
  • Image-level attacks: Unauthorized embedding can place a signature into vanilla images, enabling impersonation of another model user.The paper therefore emphasizes keeping the watermark extractor private.
  • Network-level attacks: Model purification makes it difficult to substantially reduce bit accuracy without compromising image quality, because artifacts appear during fine-tuning.The attack removes the message loss and focuses on perceptual reconstruction loss.
  • Network-level attacks: Averaging two user models yields shared bits when their signatures agree and random extracted bits when their signatures disagree.The observed marking assumption also holds when colluders operate at the image level.

8. Conclusion & Discussion

Stable Signature embeds invisible watermarks into all images generated by Latent Diffusion Models through quick decoder fine-tuning, without altering the diffusion process. The method supports detection and user identification, while the experiments involve substantial computational cost and exclude generative-model training.

  • Conclusion: Quick decoder fine-tuning embeds watermarks in all images generated by Latent Diffusion Models without altering the diffusion process.This preserves compatibility with most LDM-based generative models.
  • Conclusion: Stable Signature watermarks are described as robust, invisible to humans, and usable for detecting generated images and identifying their users.These claims summarize the method’s reported application scope.
  • Discussion: The work motivates watermarking as an alternative to relying solely on passive detection before publicly releasing generative models.The authors connect this motivation to the societal impact of public image-generator releases.
  • Reproducibility: The diffusion-based generative model was trained on an internal dataset of licensed images, while the study uses the open-source-compatible KL auto-encoder with compression factor f = 8.Code is publicly available.
  • Environmental impact: Approximately 2000 GPU-days, or 50000 GPU-hours, were used for the experiments, corresponding to about 10 tons of CO2eq excluding generative-model training.The estimate excludes storage, CPU-hours, and hardware production costs.

Supplementary Material

The supplementary material describes the watermark architecture, optimization setup, attack simulation, whitening procedure, and transformation benchmark used to evaluate Stable Signature.

  • Architecture: The watermark encoder uses four convolutional batch-normalization-ReLU blocks, while the extractor uses seven blocks followed by k-bit output layers.The extractor ends with average pooling and a k × k linear layer.
  • Optimization: Training uses 256 × 256 MS-COCO images, 48-bit messages, scaling factor α = 0.3, 300 epochs, and eight GPUs.Optimization uses Lamb with cosine annealing and five warmup epochs.
  • Attack simulation: The attack simulation randomly applies cropping, resizing, or identity before decoding, with crop or resize parameters 0.3 or 0.7.JPEG compression follows these transformations.
  • Whitening: Whitening uses PCA statistics from 10k vanilla images to make thresholded extractor bits independently and identically Bernoulli distributed.The transformation uses the extractor’s covariance eigendecomposition.
  • Image transformations: The robustness benchmark varies crop area, rotation angle, JPEG quality, and image adjustments, including a combined crop 0.5, brightness 1.5, and JPEG 80 transformation.These transformations simulate common image-editing operations.

A.3. Generative tasks

The supplementary experiments cover text-to-image, image editing, inpainting, super-resolution, watermark comparisons, removal attacks, model collusion, and perceptual-loss effects. They specify task protocols, robustness measurements, and qualitative image-quality trade-offs.

  • Generative tasks: Text-to-image evaluation uses MS-COCO validation captions, guidance scale 3.0, and 50 diffusion steps.The protocol selects the first 1000 or 5000 captions depending on evaluation.
  • Generative tasks: Image edition evaluation applies DiffEdit with image and text descriptions, mask-based latent-space generation, and 90% encoding ratio.The experiment computes 5000 images from COCO data.
  • Generative tasks: Inpainting experiments use 5000 thick masks at 512 × 512 resolution, covering 1–50% of each image with a 27% average.Two latent-diffusion variants differ in whether ground-truth pixels are restored after decoding.
  • Generative tasks: Super-resolution evaluation resizes 5000 ImageNet validation images to 128 × 128 and upscales them to 512 × 512 before computing FID.The initial resizing uses bicubic interpolation.
  • Watermark comparisons: Post-generation baselines include DCT-DWT, SSL Watermark, FNNS, and HiDDeN, with distortion controlled for comparison.The implementations are selected for robustness, arbitrary image sizes, or arbitrary bit counts.
  • Attacks: Removal experiments vary perceptual auto-encoders and compression factors to measure the trade-off between image quality degradation and watermark removal.White-box attacks optimize image pixels toward a fixed random message for 10 Adam iterations.
  • Model collusion: Model-collusion experiments average pairs of ten separately fine-tuned 48-bit-key models and analyze 48,000 decoded bits.The results are compared with the two original keys.
  • Perceptual loss: Lower perceptual-loss weights introduce texture artifacts, with problems beginning around PSNR 25 dB.At matched watermark performance, Watson-VGG produces the most eye-pleasing results, followed closely by LPIPS.

B.2. Additional results on watermarks robustness

Additional evaluations show that Stable Signature remains robust under image transformations and decoder network attacks, while whitening improves the statistical basis for false-positive control. The main practical boundary is that robustness depends on retained visual content and image quality.

  • Image transformations: Above 0.9 bit accuracy is obtained most often across tasks and transformations, while resize and JPEG 50 are the most challenging.The watermark is a 48-bit binary key; crop location matters less than the visual content retained.
  • Network-level attacks: Quantization and pruning degrade generation quality faster than watermark robustness.Reducing bit accuracy below 98% requires quantization that lowers PSNR below 25 dB or pruning that lowers PSNR below 20 dB.
  • Statistical validation: The false-positive calculation relies on vanilla-image decoder bits being i.i.d. Bernoulli random variables with parameter 0.5.Because this assumption is not true in practice before whitening, the output is whitened after pre-training.
  • Statistical validation: Whitening substantially changes the decoder-bit covariance matrix toward the behavior of independent Bernoulli bits.The comparison uses 5k vanilla images at 512 × 512 and a Bernoulli simulation of 48-bit messages.
  • Statistical validation: For false-positive rates below 10^-6, the theoretical estimate is slightly higher than the empirical rate, conservatively overestimating observed false positives.At higher false-positive rates, theoretical and empirical values match almost perfectly across 10 keys.

C. Additional Qualitative Results

Qualitative evaluations compare original and watermarked generation across standard image synthesis, inpainting, and super-resolution. The figures use matched settings or latents to inspect visual differences between original and watermarked models.

  • Generated images: Qualitative results compare Stable Signature with Dct-Dwt, SSL Watermark, FNNS, and HiDDeN on generated images at resolution 512.The comparison is presented as a visual evaluation of different watermarking methods.
  • Generated images: Validation-set MS-COCO prompts are generated from the same latents using original or watermarked generative models at resolution 512 with a 48-bit signature.Matched latents support direct visual comparison between the two generative models.
  • Inpainting: ImageNet inpainting results compare original and watermarked generative models under full-image modification and masked-area-only filling.The masked-area-only scenario is more informative for watermarking because the signal is present only in the masked region, though it introduces copy-paste artifacts.
  • Super-resolution: ImageNet super-resolution results compare original and watermarked models when 128 × 128 images are upscaled to 512 × 512 with factor f = 4.The figure reports qualitative results for low-resolution inputs and their upscaled outputs.
Loading 2303.15435v2…