Source-linked AI summary
SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models
Feifei Wang, Zhentao Tan, Tianyi Wei, Yue Wu, Qidong Huang
TL;DR
Diffusion-based customization can enable unauthorized use of portraits and other private images, while existing anti-customization methods may optimize ineffectively at some timesteps. SimAC adjusts timestep selection and feature optimization to diffusion-model properties, and experiments report greater identity disruption on facial benchmarks.
Problem
Unauthorized customization can use private portraits to generate fake images, while existing anti-customization methods may spend attack steps on near-zero gradients.
Source: §1 · pp. 1–2Method
SimAC combines adaptive greedy timestep selection with a feature interference loss targeting high-frequency features during denoising.
Source: §1 · p. 2, §3.4 · p. 5, §3.5 · p. 6Results
On two face datasets, SimAC achieves more obvious identity disruption and better privacy protection than the existing approach described.
Source: §1 · p. 3Takeaways & Limitations
The experiments support using diffusion timestep and feature properties to improve anti-customization protection of facial images.
Source: §1 · p. 3, §9 · p. 13
Abstract
from arXiv · showhide
Despite the success of diffusion-based customization methods on visual content creation, increasing concerns have been raised about such techniques from both privacy and political perspectives. To tackle this issue, several anti-customization methods have been proposed in very recent months, predominantly grounded in adversarial attacks. Unfortunately, most of these methods adopt straightforward designs, such as end-to-end optimization with a focus on adversarially maximizing the original training loss, thereby neglecting nuanced internal properties intrinsic to the diffusion model, and even leading to ineffective optimization in some diffusion time steps. In this paper, we strive to bridge this gap by undertaking a comprehensive exploration of these inherent properties to boost the performance of current anti-customization approaches. Two aspects of properties are investigated: 1) We examine the relationship between time step selection and the model's perception in the frequency domain of images and find that lower time steps can give much more contributions to adversarial noises. This inspires us to propose an adaptive greedy search for optimal time steps that seamlessly integrates with existing anti-customization methods. 2) We scrutinize the roles of features at different layers during denoising and devise a sophisticated feature-based optimization framework for anti-customization. Experiments on facial benchmarks demonstrate that our approach significantly increases identity disruption, thereby protecting user privacy and copyright.
1. Introduction
The introduction identifies ineffective adversarial optimization at some diffusion timesteps as a privacy-protection gap. It proposes timestep selection and feature-based optimization that account for diffusion-model properties.
1. Introduction
- Large randomly sampled timesteps can yield near-zero gradients, making many attack steps ineffective and leaving user details available for customized images.The paper attributes this weakness to anti-customization methods not accounting for properties inherent in diffusion models.Source: §1 · p. 2
- Smaller timesteps emphasize high-frequency image components, motivating an adaptive greedy search for effective timestep intervals.The search iteratively updates the selection range.Source: §1 · p. 2
- During denoising, deeper UNet decoder layers emphasize texture while lower layers emphasize structure, so SimAC targets high-frequency features with a feature interference loss.The loss is integrated with the diffusion denoising loss.Source: §1 · p. 2
- The paper frames its contribution as aligning adversarial-gradient optimization with diffusion models’ timestep and intermediate-layer properties.It presents adaptive greedy time selection and feature interference loss as improvements that can be integrated with existing anti-customization frameworks.Source: §1 · pp. 2–3
- On two face datasets, SimAC achieves more obvious identity disruption and better privacy protection.The method is designed to generalize across anti-customization frameworks.Source: §1 · p. 3
2. Related work
The related work reviews diffusion generation and customization, then describes privacy-protection methods and the limitations that motivate this paper.
2.1. Generative Models and Diffusion Models
- Diffusion models support prompt-conditioned generation, while latent diffusion operates in a lower-dimensional latent space and uses cross-attention for conditioning.The section contrasts these models with earlier generative approaches.Source: §2.1 · p. 3
2.2. Customization
- DreamBooth fine-tunes Stable Diffusion using a few concept images and an identifier, whereas textual inversion freezes the U-Net and optimizes the identifier embedding.Related approaches include Custom Diffusion and DreamBooth with LoRA.Source: §2.2 · p. 3
2.3. Privacy Protection for Diffusion Models
- Privacy methods include adversarial attacks on diffusion features or denoising objectives, while current approaches are also described as having ineffective optimization and poor identity disruption.PhotoGuard instead targets unauthorized inpainting.Source: §2.3 · p. 3
3. Method
SimAC analyzes how diffusion timesteps and U-Net features shape adversarial optimization, then uses adaptive timestep selection and feature interference loss to improve anti-customization perturbations.
3.1. Preliminaries
- Diffusion models progressively denoise noisy latents, while adversarial perturbations seek to hinder image reconstruction under a bounded noise budget.The paper describes PGD as iteratively updating perturbations using the loss gradient and projecting them into an η-ball.Source: §3.1 · pp. 3–4
3.2. Overview
- SimAC combines analysis of diffusion-model properties with adaptive greedy timestep selection and feature interference loss in its optimization pipeline.The method targets temporal and feature dimensions of existing anti-customization approaches.Source: §3.2 · p. 4, Figure 3 · p. 4
3.3. Analysis on Properties of LDMs
- For t ∈ [0, 100], reconstruction differences are mainly high-frequency, whereas at t ∈ [700, 800], low-frequency differences dominate; deeper decoder features capture more high-frequency information.The analysis also reports that larger timesteps are associated with more near-zero gradients and less high-frequency response.Source: §3.3 · pp. 4–5, Figure 4 · p. 5
3.4. Adaptive Greedy Time Interval Selection
- The adaptive greedy strategy samples five timesteps, removes an interval around the minimum-gradient timestep, and adds noise at the maximum-gradient timestep.It repeats until the interval is at most 100 steps or the training-step limit is reached; the analysis states that rescaling can raise expected gradient magnitude when some timesteps yield zero gradients.Source: §3.4 · pp. 5–6
3.5. Feature Interference Loss
- Feature interference loss penalizes layer-wise Euclidean distances between clean and perturbed features and is combined with the vanilla training loss.The authors identify small time intervals as better for this loss because increasing timesteps reduce high-frequency feature response.Source: §3.5 · p. 6
4. Experiments
Experiments compare SimAC-enhanced anti-customization against prior methods on two face datasets, then test transfer across prompts, model versions, and customization strategies. Across the supplied comparisons, SimAC improves protection, though transfer performance weakens in some mismatch settings.
4.1. Setup
- Experiments use Celeb-HQ and VGGFace2 portraits, mainly Stable Diffusion v2.1, assessing identity resemblance with Identity Score Matching, face-detection failure with FDFR, and image quality with BRISQE and SER-FIQ.The dataset comprises about 50 individuals, each with at least 15 clear face images for customization; all methods use the same noise budget, η = 16/255.Source: §4.1 · p. 6
4.2. Comparison with Baseline Methods
- SimAC+Anti-DreamBooth improves Anti-DreamBooth and outperforms the other baselines across prompts, with stronger identity disruption on CelebA-HQ and VGG-Face2.The reported effects include higher face-detection failure rates and lower identity-matching scores; the VGG-Face2 comparison describes greater concealment of facial features.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · pp. 7–8
4.3.1 Quantitative Results
- Under prompt mismatch, performance declines, but ISM remains consistent; under model-version mismatch, overall performance declines while ISM remains considerable.The prompt test changes the rare identifier from “sks” to “t@t”; the model test evaluates training and customization across Stable Diffusion v1.4 and v2.1.Source: §4.3.1 · p. 8, Table 3 · p. 9, Table 4 · p. 9
4.3.2 Qualitative result
- When model versions differ, protection weakens in some cases, but strong distortion remains in most prompts; transfer to Custom Diffusion is less pronounced than to Lora.The qualitative results also report that SimAC improves the transfer performance of Anti-DreamBooth noise for both customization strategies.Source: Figure 10 · p. 9, Figure 11 · p. 10, §4.3.2 · p. 10
5. Anti-Customization Efficiency
SimAC improves Anti-DreamBooth’s training efficiency: 20 epochs with SimAC match the performance of 50 epochs without it. The authors link this gain to stronger gradients from adaptive time selection.
5. Anti-Customization Efficiency
- The efficiency study compares SimAC+Anti-DreamBooth and Anti-DreamBooth at 10, 20, and 50 epochs.The authors report that SimAC+Anti-DreamBooth outperforms Anti-DreamBooth even when both use the same number of epochs.Source: §5 · p. 10
- 20 epochs of SimAC+Anti-DreamBooth match the performance of Anti-DreamBooth trained for 50 epochs.The comparison averages metrics across four prompts; the authors describe this as a significant efficiency boost.Source: §5 · p. 10
- For models trained for 20 or 50 epochs, integrating adaptive time selection noticeably enhances gradients during training, significantly improving training efficiency and protection effectiveness against malicious customization.Source: §5 · pp. 10–11
6. GPT-4V(ision) Evaluation
The evaluation uses GPT-4V to assess identity similarity in generated portraits, complementing ArcFace-based measurements. Among detectable-face images, SimAC+Anti-DreamBooth receives the lowest similarity scores and shows the strongest identity disruption.
6. GPT-4V(ision) Evaluation
- The study uses GPT-4V to simulate human judgment of identity similarity alongside the conventional ArcFace measure.Selected generated images are grouped with the clean portrait and outputs from AdvDM, Anti-DreamBooth, and SimAC+Anti-DreamBooth for comparison.Source: §6 · p. 11
- Prompts asked the model to assess similarity based on a person’s facial features and give a 0-to-10 rating with explanations or justifications; of 50 images assessed, 35 yielded valid responses.Source: §6 · p. 11
- Among images with detectable faces, GPT-4V judged SimAC+Anti-DreamBooth to have the lowest identity similarity and the most profound identity disruption.Its face-detection failure rate was also higher than AdvDM and Anti-DreamBooth, so the similarity comparison applies to the detectable-face subset.Source: §6 · p. 11
7. Generalizing to Other Anti-Customization Methods
The authors test whether SimAC also improves AdvDM. It generally improves face-detection failure rates across prompts, with less effective performance on the prompt also used during training.
7. Generalizing to Other Anti-Customization Methods
- The authors focus on AdvDM because PhotoGuard’s VAE-encoder attack does not involve selecting timesteps for noise addition during training.Source: §7 · p. 12
- For the latter two prompts, SimAC improves AdvDM’s ISM and FDFR protection metrics as well as BRISQUE and SER-FIQ image-quality degradation.Its performance is comparable to AdvDM on the second prompt.Source: §7 · p. 12, Table 9 · p. 12
- Adding SimAC to AdvDM consistently improves face-detection failure rates across four prompts and improves protection metrics on the latter two prompts.On the training prompt, “a photo of sks person,” the combination is less effective; the authors note that unseen prompts better reflect real-world inference.Source: Table 9 · p. 12, §7 · p. 12
8. Ablation Study
The ablations favor decoder layers 9, 10, and 11 for feature interference loss, while increasing the noise budget strengthens defense. The authors choose 16/255 to balance identity disruption and input-image quality.
8. Ablation Study
- UNet decoder layers 9, 10, and 11 perform best among tested layer combinations across four prompts, consistent with deeper features being more sensitive to adversarial noise.Table 10 evaluates layer combinations on CelebA-HQ under four prompts.Source: §8 · p. 12, Table 10 · p. 13
- Increasing the noise budget improves anti-customization performance, but the authors select 16/255 to balance disruption against image degradation at higher budgets.The table caption states that generated-image deterioration increases with the noise budget.Source: §8 · pp. 12–13, Table 11 · p. 13
- For protected user portrait images after customization, a 4/255 noise budget yields minimal protective effect, while 16/255 makes the effect highly pronounced and meets the authors’ user identity privacy objective.Figure 14 depicts protected user portrait images after customization under different noise budgets.Source: §8 · p. 12
9. Conclusion
SimAC addresses ineffective optimization in adversarial anti-customization methods by examining diffusion-model properties. Its temporal and feature-based improvements increase identity disruption in experiments.
9. Conclusion
- The paper identifies overlooked diffusion-model properties as a weakness of adversarial anti-customization methods, which can lead to ineffective optimization.Source: §9 · p. 13
- SimAC combines adaptive greedy timestep selection, informed by frequency-domain perception, with feature-based optimization informed by denoising-layer roles.Source: §9 · p. 13
- Experiments on facial benchmarks show increased identity disruption, supporting privacy protection within the evaluated setting.Source: §9 · p. 13