PaperlayerView exact revision on arXiv

Source-linked AI summary

FedOT: Ownership Verification and Leakage Tracing via Watermarks for Federated LDMs

Wenlong Cheng, Yuan Gan, Yunqiu Xu, Jiaxu Miao

arXiv:2606.22875v1cs.CV

TL;DR

Federated LDMs expose models to unauthorized distribution while existing watermarking supports ownership verification but not client-level tracing and can be removed through VAE replacement. FedOT combines chunked watermarks with Latent Vector Transformation to verify ownership, identify leaking clients, and bind the VAE to the U-Net; experiments show reliable verification and tracing.

  • Problem

    Existing LDM watermarking in federated learning does not trace leaks to specific clients and remains vulnerable to VAE replacement attacks.

  • Method

    FedOT embeds client-specific chunked watermarks and applies Latent Vector Transformation to bind the VAE and U-Net latent spaces.

  • Results

    Experiments show that FedOT provides reliable ownership verification and leakage tracing for federated LDMs, with negative transformation selected as the optimal LVT strategy.

  • Takeaways & Limitations

    FedOT offers a federated-LDM protection framework that supports both ownership verification and identification of the client responsible for a leaked model.

  • Takeaways & Limitations

    Embedding robust watermarks can slightly reduce image-generation quality, creating an inherent trade-off between watermark integrity and fidelity.

Abstract

from arXiv · show

Training Latent Diffusion Models (LDMs) within Federated Learning (FL) has attracted increasing attention due to its ability to combine the powerful generative capacity of LDMs with the privacy-preserving properties of FL. However, FL requires sharing the global model with multiple participants, which risks unauthorized model distribution or resale by malicious clients. While an intuitive approach is to adopt existing VAE-based watermarking techniques for LDMs in FL, this strategy falls short in addressing such threats due to two fundamental challenges: (1) Existing methods support ownership verification but lack the ability to trace model leakage to a specific malicious client; (2) VAE-based watermarks are vulnerable, as they can be removed simply by replacing the decoder with a clean counterpart. In this paper, we propose FedOT, the first framework for ownership verification and leakage tracing in federated LDMs. Specifically, to address the first challenge, we design a chunked watermark, where the first part is for ownership verification, and the second part is used for client identification. Furthermore, to overcome the second challenge and secure the model against VAE replacement attack, we introduce Latent Vector Transformation (LVT), which strengthens the connection between the VAE and U-Net latent spaces by modifying the original latent distribution of the VAE. Consequently, any attempt to replace the VAE for watermark removal leads to significant image quality degradation, making the LDM model unusable. Extensive experiments demonstrate that FedOT achieves superior performance in both ownership verification and traceability. Project page: https://spyzixuan.github.io/FedOT/.

1 Introduction

FedOT addresses unauthorized leakage of federated latent diffusion models by combining chunked watermarks for ownership verification and client tracing with Latent Vector Transformation (LVT) to resist VAE replacement. Experiments identify negative transformation as the preferred LVT strategy and show reliable verification and tracing under common watermark-removal attacks.

  • Problem: FedLDMs expose shared global models to malicious clients who may distribute or resell fine-tuned models without authorization.Such leakage creates intellectual-property and ethical concerns.
  • Problem: Existing federated copyright and traceability methods target classification, making them inapplicable to generative LDMs.The paper therefore turns to watermarking techniques designed for LDMs while considering federated training.
  • FedOT: FedOT embeds client-specific chunked binary watermarks into each client’s VAE decoder, using the first r bits for group ownership and remaining n−r bits for client identification.This enables both ownership verification and leakage tracing for FedLDMs.
  • FedOT: LVT binds the VAE and U-Net latent spaces so replacing the watermarked VAE removes the watermark only at the cost of severe image-quality degradation.The transformation is designed to deter zero-cost VAE replacement attacks.
  • Evaluation: Among translation, mirror, and negative LVT strategies, negative transformation is optimal for balancing generation performance and component-binding strength.FedOT is evaluated against VAE replacement, model purification, and image attacks, showing reliable ownership verification and leakage tracing.

2 Related Work

Federated Learning enables privacy-preserving collaborative training by keeping raw data local while aggregating client updates into a global model. Applied to Latent Diffusion Models, it reduces communication by sharing only U-Net parameters but exposes the global model to participating clients, creating copyright-protection challenges.

  • Federated Learning: Federated Learning lets multiple clients collaboratively train models without sharing raw data, preserving privacy.
  • Federated Learning: In a client-server architecture, each client trains locally and the server aggregates updates with FedAvg to form a global model.
  • Federated Learning: For federated Latent Diffusion Model training, only U-Net parameters are uploaded and aggregated each round, reducing communication costs.
  • Federated Learning: Because participating clients can access the global model, Federated Learning introduces copyright-protection challenges.

3 Methodology

FedOT combines Latent Vector Transformation (LVT) with client-specific watermarking in federated Stable Diffusion to support ownership verification and leakage tracing. LVT couples the VAE and U-Net through a modified latent distribution, making VAE replacement cause severe image-quality degradation.

  • Framework Overview: FedOT applies LVT to the global VAE, embeds unique watermarks into model replicas, and federates local U-Net updates while aggregating only U-Net parameters.The server performs LVT before creating replicas, distributes them for private-data fine-tuning, and verifies watermarks from generated images.
  • Watermark Design: The chunked n-bit watermark uses the first r bits for ownership verification and the remaining n−r bits for identifying the leaking client.The server verifies the shared prefix first, then compares the client-specific suffix to trace the source, reducing identification overhead.
  • Watermark Training: Watermarks are embedded once per client in the VAE decoder using an extractor-guided objective that balances watermark accuracy with Watson-VGG image fidelity.Because only the U-Net is updated during federated fine-tuning, this procedure preserves the watermark despite subsequent U-Net optimization.
  • Latent Vector Transformation: LVT modifies the VAE encoder’s latent distribution so the U-Net adapts to a transformed latent space, establishing dependency between the VAE and U-Net.The transformation pipeline trains the encoder to produce transformed latents and adapts the decoder to reconstruct from them.
  • Latent Vector Transformation: Replacing the watermarked VAE with a clean counterpart creates a latent-space mismatch that severely degrades image synthesis quality.Random Gaussian transformations impair reconstruction, while deterministic shifts can cause color inversion, blurring, and texture distortions.

4 Experiments

Experiments evaluate FedOT’s reconstruction and generation quality, watermark reliability, ownership verification, client tracing, and resistance to VAE replacement and purification attacks. Results show that LVT improves attack resistance while introducing quality trade-offs, and FedOT maintains robust verification and tracing across federated settings.

  • Experimental Setup: FedOT trains on 10,000 COCO2017 images for watermark embedding and 10,000 LAION image-text pairs for federated Stable Diffusion fine-tuning across K = 5 clients.The watermark contains n = 48 bits, with r = 16 bits for ownership verification and 32 bits for client tracking.
  • VAE Replacement Attack: After VAE replacement, Stable Signature* and FedOTw/o LVT lose watermark detection while preserving image quality, whereas LVT causes noticeable quality degradation that prevents low-cost watermark removal.This contrast demonstrates LVT’s effectiveness against replacement attacks.
  • LVT Strategy Comparison: FedOTrand, FedOTtran, and FedOTmir resist replacement by +37.225, +70.070, and +49.147, respectively, while their generation quality differs substantially.FedOTrand has FID: 35.585, FedOTtran has FID: 22.427, and FedOTmir has FID: 21.475; larger latent shifts improve binding but can reduce fidelity or semantic consistency.
  • Ownership Verification and Tracing: Stable Signature* verifies ownership but cannot trace clients, while FedOT supports both ownership verification and client tracing, with detection rates above 0.932 and bit accuracy over 0.91.FedOTw/o LVT performs best on these metrics, while LVT variants remain robust despite a slight decrease.
  • Robustness Analysis: Across 5, 10, and 20 clients, generation quality remains relatively stable, watermark robustness is largely unaffected, and detection rates exceed 0.941.Purification for 300 epochs raises all three LVT methods’ FID from below 22 to above 26, showing that watermark removal affects generated-image quality.

5 Conclusion … A.1 Federated LDMs and Threat Model

FedOT introduces ownership verification and client-level tracing for federated latent diffusion models through chunked watermarks. Its federated setup keeps the VAE frozen while clients update the U-Net, and accounts for malicious clients attempting to evade detection or redistribute models.

  • 5 Conclusion: FedOT is presented as the first framework for ownership verification and tracing in federated Latent Diffusion Models.It targets both model ownership and identification of the client responsible for a leak.
  • A.1 Federated LDMs and Threat Model: A chunked watermark identifies the responsible client for a leaked model while supporting ownership verification and traceability.The watermark is embedded into each distributed LDM as an n-bit message.
  • 5 Conclusion: FedOT uses Latent Vector Transformation to strengthen the connection between VAE and U-Net latent spaces against VAE replacement attacks.The conclusion states that existing VAE-based watermarking methods are vulnerable to removal attacks, motivating this design.
  • A.1 Federated LDMs and Threat Model: A global federated LDM contains a pretrained VAE that encodes images into latent space and a latent-space diffusion model using a U-Net architecture.The global model is maintained on a central server.
  • A.1 Federated LDMs and Threat Model: During federated training, clients update only U-Net parameters while keeping the VAE frozen.Malicious clients may store intermediate global models and redistribute them illegally.
  • A.1 Federated LDMs and Threat Model: Ownership verification confirms whether a given image originates from the federated LDM.Traceability determines which client is responsible for a leaked model.
  • A.1 Federated LDMs and Threat Model: The trusted server embeds unique watermarks, aggregates client updates, and performs watermark-based verification when suspicious images appear.These responsibilities define the server-side assumptions in the threat model.
  • A.1 Federated LDMs and Threat Model: Malicious clients may evade watermark detection through image manipulation, model fine-tuning, or direct modification of watermark-related parameters.The client group may contain malicious clients.

A.2 Local Training under FedOT

FedOT trains federated Stable Diffusion by optimizing and aggregating only U-Net parameters while freezing the VAE and text encoder to preserve watermark integrity. Because LVT transforms the latent space, image quality improves initially, then degrades, with FID and CLIP-Score stabilizing around epoch 15.

  • Local training: Each client locally updates only U-Net parameters, while the VAE and text encoder remain frozen to preserve watermark integrity.After local updates, the server aggregates U-Net parameters and redistributes the aggregated model for the next round.
  • Federated aggregation: The server repeatedly aggregates client U-Nets and redistributes the aggregated model, producing the final global model after federated training.The algorithm distributes complete watermarked models initially and uploads the complete model in the final round.
  • Training dynamics: Around epoch = 15, FID and CLIP-Score stabilize; further federated training deteriorates generated image quality.This non-monotonic learning dynamic results from the LVT-induced transformation of the latent space.

B Watermark Design and Training Details · B.1 Additional Watermark Design Details

FedOT constructs client-specific n-bit watermarks by combining a shared ownership-verification prefix with Hamming-optimized tracing suffixes, then embeds them into replicated global-model copies. Genetic-algorithm optimization increases inter-client separability, enabling reliable tracing at large client scales while reducing collisions.

  • B Watermark Design and Training Details: FedOT takes the global model M, client count K, watermark length n, and ownership-verification length r as inputs, then replicates M K times.The resulting copies are used to create client-specific watermarked models.
  • B Watermark Design and Training Details: The first r watermark bits support ownership verification, while the remaining n −r bits are optimized separately for each client.The two parts are concatenated into K complete n-bit watermarks before embedding into corresponding model copies.
  • B Watermark Design and Training Details: The complete watermark-generation and embedding procedure is specified in Algorithm 3.This procedure produces the set of watermarked models { ˆ Mi}K.
  • B.1 Additional Watermark Design Details: To reduce source-client confusion, FedOT maximizes the minimum Hamming Distance between the tracing suffixes of different client watermarks.The objective compares bits from positions r+1:n across every pair of clients.
  • B.1 Additional Watermark Design Details: A Genetic Algorithm (GA) approximately optimizes the watermark assignment to maximize distinction across clients.The optimization targets separability among client-identification suffixes.
  • B.1 Additional Watermark Design Details: 0.303% collision probability occurs at 102 clients under random assignment, whereas Hamming-optimized assignment maintains 0% collision probability up to 103 clients.These simulations evaluate tracing scalability beyond the client scale used in the main experiments.

B.2 Additional Watermark Training Details … C.1 Fine-tuning Details

FedOT trains client-specific watermarked decoders while preserving the VAE encoder, and defines watermark accuracy and detection metrics. Its LVT fine-tuning couples the VAE and U-Net through transformed latent spaces and combines reconstruction, perceptual, regularization, and generation losses.

  • B.2 Additional Watermark Training Details: The server generates a unique binary watermark for each client and trains a decoder with the Stable Signature extractor E to embed it.Public images are encoded by a frozen VAE encoder, decoded by the trainable decoder, and passed to E for watermark prediction.
  • B.2 Additional Watermark Training Details: The resulting client-specific decoders invisibly embed watermarks into generated outputs, enabling ownership verification without reducing visual fidelity or model utility.
  • B.3 Definitions of Bit Accuracy and Detection: Bit Accuracy measures the proportion of correctly extracted watermark bits relative to the ground-truth watermark.It is computed as the average indicator of per-bit equality between m and m′.
  • B.3 Definitions of Bit Accuracy and Detection: Detection measures the proportion of generated watermarked images whose extracted watermark reaches a bit-accuracy threshold τ.A higher detection rate indicates more robust and reliable watermark extraction.
  • C.1 Fine-tuning Details: During LVT fine-tuning, the encoder learns the forward transformation T and the decoder learns its inverse T^-1, adapting latent vectors and coupling the VAE with the U-Net.For negative transformation, the negative image serves as ground truth for pixel-wise mapping; the decoder then reconstructs images in the new latent space.
  • C.1 Fine-tuning Details: The LVT objective uses MSE for faithful reconstruction, perceptual loss for high-level feature similarity, KL divergence for latent regularization, and generator loss for image realism.The composite VAE loss sums these terms with adaptive weighting for the KL and generator components.
  • C.1 Fine-tuning Details: The KL divergence uses λ_KL = 10^-8, while the adaptive loss weight is clamped to a maximum value of 10^4 to maintain training stability.The adaptive weight balances perceptual and generator losses using decoder-gradient norms and a small denominator constant δ.

C.2 Analysis of LVT · D Watermark Robustness

LVT preserves latent-space structure through deterministic transformations, whereas random Gaussian perturbations disrupt local continuity and substantially impair decoder reconstruction. Translation and mirroring retain variance and neighborhood geometry, enabling stable reconstruction with high visual quality while embedding detectable latent perturbations.

  • C.2 Analysis of LVT: LVT samples encoder latents as z = µ + σ·ε, with ε drawn from a standard Gaussian reparameterization noise distribution.The latent vector follows z ∼ N(µ, σ^2).
  • C.2 Analysis of LVT: Random Gaussian perturbations increase latent variance from σ^2 to σ^2 + 1 and disrupt local continuity between neighboring samples.The perturbed latent distribution is z′ ∼ N(µ, I(σ^2+1)), while independent noise alters neighborhood distances.
  • C.2 Analysis of LVT: The resulting neighborhood disruption makes it difficult for the decoder to learn a stable inverse mapping, causing inaccurate reconstruction and noticeable quality degradation.The noise-dominated distance term breaks local latent geometry and prevents accurate image reconstruction.
  • C.2 Analysis of LVT: Translation and mirroring are deterministic, globally consistent linear mappings that transform the latent mean while preserving variance σ^2.Translation uses A = 1 and C ≠ 0, whereas mirroring uses A = −1 and C = 0.
  • C.2 Analysis of LVT: These structured transformations preserve pairwise distances and therefore maintain the local geometric structure of the latent space.For neighboring vectors, the transformed distance remains ||z_i − z_j||.
  • C.2 Analysis of LVT: By preserving local relationships without changing variance, structured transformations support stable decoder reconstruction, high visual quality, and globally detectable latent perturbations.Unlike random noise, they introduce a consistent global shift while maintaining image quality.
  • C.2 Analysis of LVT: Negative transformations preserve the latent space’s Gaussian property while imposing a structured, pixel-wise negative relationship across latent dimensions.The encoder maps inputs toward negative counterparts, and the decoder adapts accordingly.

D.1 Generated Image Attacks … D.4 Mirror Recovered Attack

FedOT maintains strong watermark robustness under common generated-image attacks, with worst-case bit accuracy of 0.772 under combined cropping and brightness changes. Recovered VAE-replacement attacks remain ineffective for negative, translation, and mirror LVT transformations, even when attackers know the recovery process or translation coefficient.

  • D.1 Generated Image Attacks: FedOT remains robust against cropping, brightness, JPEG compression, contrast adjustment, text overlay, and 50% resizing attacks on generated images.These perturbations simulate common real-world image-editing scenarios.
  • D.1 Generated Image Attacks: 0.772 worst-case bit accuracy remains under the combined Crop and Brigh. attack, despite FedOT’s slight decrease relative to Stable Signature*.The result demonstrates strong overall watermark performance under image-level perturbations.
  • D.2 Negative Recovered Attack: Negative-like artifacts appear after VAE replacement, and applying an additional negative transformation only partially restores the degraded images.The learned latent negative mapping approximates rather than perfectly replicates a pixel-level negative transformation.
  • D.2 Negative Recovered Attack: Recovered images retain high FID after attackers apply an additional negative transformation, indicating that watermark removal still produces substantial quality degradation.Clean denotes unattacked watermarked generation, Attack denotes VAE replacement, and Recovered denotes the additional negative operation.
  • D.3 Translation Recovered Attack: Translation-based VAE replacement causes noticeable color shifts, while recovery remains ineffective even when the attacker fully knows the translation coefficient = 5.Recovered image quality remains low and can be worse than direct VAE replacement.
  • D.4 Mirror Recovered Attack: Mirror recovery remains ineffective even when attackers fully know the mirror transformation process and attempt to restore image quality.The mirror transformation is harder to identify as the source of degradation than the negative and translation transformations.

D.5 Collusion Attack · E More Experimental Results

FedOT is evaluated against collusion in which malicious clients average watermarked VAE parameters, and against VAE replacement using Flickr30K generation-quality comparisons with Stable Signature*. Collusion substantially degrades tracing accuracy while leaving ownership accuracy largely unaffected, indicating that the ownership watermark persists.

  • D.5 Collusion Attack: FedOT is evaluated against collusion attacks in which multiple malicious clients average their watermarked VAE parameters to remove the embedded watermark.The evaluation is inspired by SCA, Byzantine, and Fools-Gold attack methods.
  • D.5 Collusion Attack: 0.216 tracing chunk bit accuracy degradation occurs when 2 clients collude, while ownership chunk bit accuracy remains largely unaffected.The passage reports that ownership accuracy even slightly improves under collusion.
  • D.5 Collusion Attack: 0.321 tracing chunk bit accuracy degradation occurs when 3 clients collude, while ownership chunk bit accuracy remains largely unaffected.These results indicate that collusion degrades the tracing watermark but fails to eliminate the ownership watermark.
  • D.5 Collusion Attack: The chunked watermark design preserves ownership information because ownership bits are shared identically across all clients.The supplied passage attributes the differing collusion effects to this chunked watermark design.
  • D.5 Collusion Attack: Table 14 compares generation quality on Flickr30K using 256×256 images and 48-bit watermarks against Stable Signature*.The left table reports results before VAE replacement, while the right reports performance after the attack.

E.1 Results on Flicker30K Dataset … F.1 Secrecy of LVT

FedOT maintains watermark imperceptibility and enables reliable client attribution while making VAE replacement attacks degrade generated-image quality. Its LVT remains difficult to infer because the transformation is confidential and learned parameters contain deviations from the designed mapping.

  • E.1 Results on Flicker30K Dataset: On Flicker30K, VAE replacement leaves image quality largely unchanged for Stable Signature* and FedOTw/o LVT but completely removes their embedded watermarks.The experiments use the main-paper configuration with translation coefficient 5.
  • E.1 Results on Flicker30K Dataset: LVT-based FedOTrand, FedOTmir, and FedOTneg suffer noticeable image-quality degradation after VAE replacement, whereas FedOTtran shows a slight improvement on Flicker30K.The translation operation introduces mild latent-space perturbations that can occasionally produce marginally better images.
  • E.2 VAE Image Reconstruction Quality: FedOTneg achieves the best LVT reconstruction performance, with the lowest FID and the best PSNR and SSIM among the compared methods.Learning random noise can substantially harm reconstruction, producing an FID as high as 14.865.
  • E.3 Image Generation Quality: All FedOT variants preserve high watermarking fidelity, with PSNR above 31 dB and SSIM above 0.94; FedOTtran reaches PSNR 33.611 and SSIM 0.968.These results indicate minimal perceptual distortion from watermark embedding.
  • E.4 Selection of Detection Threshold τ: FedOT selects τ = 0.69 as its operating detection threshold, achieving an FPR as low as 0.1% while maintaining strong discrimination across variants.The threshold is chosen from ROC curves recording TPR and FPR under varying τ.
  • E.5 End-to-End Attribution Accuracy.: End-to-end attribution across 5 clients yields per-client accuracy from 96.80% to 98.70%, overall accuracy of 98.12%, and a false accusation rate of 1.88%.The evaluation samples 1,000 generated images per client.
  • E.6 Additional Visualizations: FedOT makes high-quality image generation infeasible after VAE replacement, deterring unauthorized model use and supporting ownership verification and model tracing.The visualizations use translation coefficient 5 and include animal, landscape, oil-painting, and human-related images.
  • F.1 Secrecy of LVT: The server keeps transformation T confidential, while learned VAE parameters contain deviations that make precise inference and reversal difficult; recovering the latent space still produces low-quality reconstruction.These findings remain true even under favorable conditions for the attacker.

F.2 Limitations · F.3 Communication Overhead · F.4 Scalability

FedOT incurs a slight image-quality trade-off from robust watermark embedding, while reducing repeated communication by exchanging only U-Net parameters after initialization. Its scalability is limited by linearly growing storage for per-client watermarked VAEs and current hardware constraints.

  • F.2 Limitations: Watermark embedding can slightly decrease image generation quality, reflecting an inherent trade-off in preserving robust watermark integrity.The method focuses on minimizing this degradation, but the trade-off remains when embedding robust watermarks into generative models.
  • F.2 Limitations: The image-quality degradation is described as a common limitation of watermarking techniques, including FedOT.The passage characterizes the degradation as minor while emphasizing that it cannot be fully eliminated.
  • F.3 Communication Overhead: The complete LDM model is transmitted only once during the initial federated-training distribution.This avoids repeatedly transmitting components other than the U-Net in later communication rounds.
  • F.3 Communication Overhead: Subsequent communication rounds exchange only U-Net parameters between the server and clients.Other components therefore incur communication overhead only during initial distribution, reducing the repeated communication burden.
  • F.4 Scalability: Each client requires one uniquely watermarked VAE, occupying approximately 335MB of server storage.Supporting more clients consequently increases storage requirements linearly.
  • F.4 Scalability: Hardware limitations prevent the method from scaling to millions of clients.The scalability constraint arises alongside the linearly increasing storage requirement for per-client watermarked VAEs.
Loading 2606.22875v1…