PaperlayerView exact revision on arXiv

Source-linked AI summary

No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation

Lizhi Xiong, Jun Li, Ziqiang Li, Weiwei Jiang, Zhangjie Fu

arXiv:2610.19717v1cs.CVcs.CR

TL;DR

Facial-reference personalization creates privacy risks, while existing defenses struggle to generalize from training-based to training-free methods. IDGuardian disrupts identity extraction and injection, and experiments report protection across diverse pipelines and robustness to post-processing and adaptive attacks.

  • Problem

    Facial-reference personalization raises privacy concerns because images may be misused for identity reconstruction or impersonation, and existing defenses generalize poorly to training-free pipelines.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    IDGuardian perturbs identity extraction and injection using cross-encoder feature confounding and guidance-flow identity deflection.

    Source: §1 · p. 2

  • Results

    Experiments report identity protection across diverse personalization pipelines and diffusion- and flux-based generators, with transfer to unseen black-box systems and robustness to post-processing and adaptive attacks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework is designed to protect identity across both training-based and training-free personalization paradigms.

    Source: §1 · p. 2

    The threat model assumes the identity holder can proactively perturb facial images before sharing them.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Recent advances in diffusion models have enabled high-fidelity, identity-preserving image generation for personalized applications such as digital avatars and virtual try-on systems. However, their reliance on sensitive facial reference images raises growing privacy concerns. Existing defense mechanisms are primarily designed for training-based personalization and struggle to generalize to emerging training-free approaches, due to fundamental differences in their identity integration paradigms. To bridge this gap, we propose IDGuardian-the first generalizable and model-agnostic identity protection framework capable of defending against both training-based and training-free personalization methods. IDGuardian abstracts the personalization process into two critical stages: identity extraction and identity injection. It then introduces crafted adversarial perturbations to simultaneously disrupt both stages. Specifically, it degrades the identity features extracted by external encoders and establishes an adversarial conceptual bridge that misdirects the generative trajectory away from the target identity. Extensive experiments show that IDGuardian effectively protects identity across various personalization pipelines and model architectures, while remaining robust to post-processing, adaptive attacks, and cross-dataset generalization. Our code is available at https://github.com/OpenAscent-L/IDGuardian.git.

1. Introduction

Personalized generation relies on facial references, creating privacy risks that existing defenses designed around training-based methods do not address well in training-free settings. IDGuardian disrupts identity extraction and injection, with experiments reporting protection across diverse personalization pipelines and model architectures.

1. Introduction

  • Diffusion models generate identity-preserving portraits from facial references, while training-based personalization fine-tunes models and training-free methods use pretrained identity encoders at inference.Training-based methods require computational resources, tuning, and GPU support; training-free approaches are described as efficient and scalable.Source: §1 · p. 1
  • Existing defenses lose effectiveness in training-free personalization because identity is injected through encoders rather than internalized by fine-tuning, and fusion strategies vary across pipelines.Figure 1 illustrates the reported ineffectiveness; these differences complicate generalizable defense design.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • IDGuardian abstracts personalization into identity extraction and injection, then disrupts both with cross-encoder feature confounding and guidance-flow identity deflection.The latter redirects the diffusion trajectory away from the clean identity manifold.Source: §1 · p. 2
  • IDGuardian protects identity across training-based and training-free methods, generalizes from one white-box training setup to unseen black-box systems, and remains robust to post-processing and adaptive attacks.Experiments include diffusion- and flux-based generators; human and perceptual evaluations confirm reliable protection without compromising usability.Source: §1 · p. 2
  • IDProtector [39] addresses training-free identity leakage through multiple encoders, but its architectural assumptions limit applicability to training-based and encoder-agnostic personalization.Its comparison motivates a defense that spans both personalization paradigms.Source: §1 · p. 2

2. Related Work

Personalized generation uses either fine-tuning or zero-shot identity injection, with training-free methods varying in how they fuse identity features. Existing proactive defenses largely target fine-tuning, leaving a generalization gap across both paradigms.

2.1. Identity-Preserving Personalized Generation

  • Training-based methods fine-tune models on reference images, whereas training-free methods extract identity features with external encoders and inject them through varied attention and embedding strategies.Examples include PhotoMaker's Direct Stack, IP-Adapter and InstantID's Cross-Attention, Infinite-ID's Mixed Attention, and KV-Edit's Self-Attention.Source: §2.1 · p. 2

2.2. Proactive Defense for Personalized Generation

  • Proactive defenses include perturbation-based methods targeting fine-tuning, backdoor triggers, and frameworks jointly optimizing encoder, reverse-process, and fine-tuning errors.Examples include Anti-DreamBooth [43], AdvDM [19], SimAC [44], PersGuard [22], and ACE.Source: §2.2 · p. 3
  • Existing defenses diminish in training-free settings, while IDProtector [39] relies on multiple encoder-specific features and has limited generalization to training-based pipelines.The passage identifies a need for more efficient, generalizable, and model-agnostic defenses.Source: §2.2 · p. 3

3. Methods

IDGuardian models personalized generation as identity extraction followed by identity injection, then perturbs the reference image to disrupt both stages. Its threat model assumes the defender can modify only that image and lacks knowledge of the attacker’s specific generation technique.

3.1. Threat Model

  • The defender perturbs a victim’s public facial image without knowing the attacker’s specific personalization technique, seeking to reduce identity consistency in generated outputs while keeping the perturbation imperceptible.The threat model assigns roles to an identity holder, generative user, and defender; perturbation magnitude is constrained to preserve imperceptibility.Source: §3.1 · p. 3

3.2. Overall Method

  • IDGuardian independently disrupts identity extraction by making protected-image features dissimilar to the original and identity injection by redirecting diffusion sampling away from the clean identity manifold.The framework treats personalization as a two-stage process and jointly optimizes an imperceptible perturbation across both stages.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Cross-Encoder Identity Field Confounding

  • Cross-Encoder Identity Field Confounding minimizes clean-versus-perturbed identity similarity using complementary FaceNet and CLIP embeddings.The encoders combine facial identity features with broader semantic information from visual data.Source: §3.3 · p. 4

3.4. Guidance-Flow Identity Deflection

  • Identity conditioning shifts the score from an unconditional source toward the target manifold; IDGuardian subtracts clean-identity guidance from adversarial-identity guidance to deflect sampling away from that manifold.The adversarial conceptual bridge steers samples toward lower-likelihood regions under the clean identity condition.Source: §3.4 · p. 5

3.5. Perturbation Optimization

  • Perturbation optimization combines normalized identity-loss and score-based bridge gradients, then updates the perturbation with a projected gradient step controlled by α.The bridge gradient is upsampled and channel-aligned to match the pixel-domain perturbation before the gradients are combined.Source: §3.5 · p. 6

4. Experiments

Experiments evaluate IDGuardian across identity-personalization pipelines and generators, testing identity suppression, image quality, robustness, generalization, and design choices. The reported results show suppression across diverse settings, with ablations examining component and optimization effects.

4.1. Implementation Details and Metrics

  • Primary experiments use VGGFace2 images resized to 224 × 224, an SDXL-based IP-Adapter surrogate, PGD with ϵ = 8/255 and α = 0.005, and 200 optimization steps.The evaluation uses identity similarity, visual quality, and recognition utility metrics; CelebA-HQ images at 512 × 512 support cross-dataset evaluation.Source: §4.1 · p. 6

4.2. Overall Performance

  • IDGuardian more effectively suppresses identity cues while better preserving perceptual quality and maintaining competitive time efficiency than other defense methods.The reported measures include PSNR, SSIM, and time cost; its low time cost is attributed to training-free inference.Source: §4.2 · pp. 6–7, Table 2 · p. 7
  • IDGuardian achieves the lowest average identity similarity across ArcFace, FaceNet, and VGGFace, while substantially reducing FaceQNet image-quality scores across seven personalization pipelines.The comparison spans training-free and training-based methods and Stable Diffusion 1.5, SDXL, and FLUX.1-dev; Figure 4 visualizes suppression across settings.Source: §4.2 · p. 6, Figure 4 · p. 6, Table 1 · p. 7
  • Identity suppression remains effective under JPEG compression, Gaussian blur, Gaussian noise, and the adaptive Impress attack.The paper notes that mild JPEG compression can slightly improve some baselines’ defense performance by also perturbing recognition of the original identity features.Source: §4.2 · p. 7, Table 3 · p. 8
  • Experiments report transfer to unseen black-box systems, generalization across surrogate backbones and datasets, and reduced perceived identity similarity in human and GPT-based evaluations.The evaluations include commercial APIs and CelebA-HQ and LFW datasets.Source: §4.2 · p. 7

4.3. Ablation Study

  • Combining CLIP and FaceNet losses leverages semantic and low-level features, and adding the adversarial conceptual bridge further enhances identity suppression.The ablations compare CLIP loss, CLIP plus FaceNet losses, and the same losses with the bridge.Source: §4.3 · pp. 7–8
  • The proposed guidance direction models the shift explicitly and yields more stable identity suppression than guidance only toward the adversarial distribution or away from the clean distribution.Both one-sided alternatives are described as often unstable because of distributional ambiguity.Source: §4.3 · p. 8, Table 4 · p. 8
  • A perturbation magnitude of ϵ = 8/255 with step size α = 0.005 gives the best reported balance of identity suppression and visual fidelity; optimization loss stabilizes after 200 steps.Smaller perturbations yield insufficient suppression, larger ones introduce artifacts and degrade quality, and loss decreases rapidly during the first 100 iterations.Source: §4.3 · p. 8

5. Conclusion

IDGuardian protects identity in diffusion- and flux-based personalized image generation by combining two complementary adversarial strategies. The paper reports that it suppresses identity information while remaining imperceptible and robust to post-processing and adaptive removal attacks.

5. Conclusion

  • IDGuardian combines Cross-Encoder Identity Field Confounding and Guidance-Flow Identity Deflection to reduce identity similarity and steer diffusion trajectories away from the target identity manifold.The framework is presented for proactive identity protection in diffusion- and flux-based personalized generation.Source: §5 · p. 8
  • Experiments show that IDGuardian suppresses identity consistency in generated outputs, mitigating identity leakage in personalized image synthesis.Source: §5 · p. 8
  • IDGuardian remains imperceptible while suppressing identity information and is robust to common post-processing and adaptive removal attacks such as Impress.Source: §5 · p. 8

A. Methodological and Evaluation Details

IDGuardian jointly optimizes perturbations to disrupt identity extraction and injection. Its optimization combines identity and diffusion-trajectory signals, while its DreamBooth evaluation averages similarity over four reference images and generated images.

A. Methodological and Evaluation Details

  • IDGuardian jointly optimizes perturbations through Cross-Encoder Identity Field Confounding and Guidance-Flow Identity Deflection, targeting identity extraction and injection, respectively.The section supplies optimization objectives, loss composition, and implementation settings omitted from the main text.Source: A. Methodological and Evaluation Details · p. 12

A.1. Optimization Process of IDGuardian

  • Each optimization step combines FaceNet/CLIP identity-loss gradients with a diffusion-space score gradient, then updates the perturbation using projected gradient descent under an L-infinity budget.The score gradient is bilinearly upsampled and its first three channels retained to match the perturbation tensor; the protected image is formed as xadv = x + δ.Source: §A.1 · p. 12

A.2. DreamBooth Fine-Tuning and Similarity Evaluation

  • For DreamBooth evaluation, similarity to four original identity images is averaged for each generated image, then averaged across all generated images.This aggregates performance using all four reference images rather than a single reference.Source: §A.2 · p. 12

B.1. Comparative Evaluation with Baselines

IDGuardian outperforms baseline methods in identity suppression while preserving perceptual quality across the reported metrics. The section also reports stable results across repeated runs and ablations supporting the complementary roles of identity supervision and the conceptual bridge.

B.1. Comparative Evaluation with Baselines

  • Tables 7 and 8 show IDGuardian’s consistent superiority in identity suppression and effectiveness in preserving perceptual quality, across identity-similarity metrics ArcFace, FaceNet, and VGGFace and perceptual-quality metrics FaceQNet and MagFace.Source: §B.1 · p. 13
  • Repeated-run analysis finds stable performance gains; on IP-Adapter, IDGuardian has consistently lower identity similarity with small variance than baseline methods.Source: §B.1 · p. 13
  • Figure 6 shows protection remaining effective under stronger JPEG compression, Gaussian blur, and Gaussian noise distortions.Source: Figure 6 · p. 13

B.2. Ablation Study on Identity Similarity and Perceptual Quality Metrics

  • Ablations support complementary roles for identity supervision and the adversarial conceptual bridge in identity suppression, with visual degradation treated as an intended defense effect.Tables 9 and 10 report identity-similarity and perceptual-quality metrics across encoder combinations and bridge configurations.Source: §B.2 · p. 13

B.3. Robustness Evaluation Settings.

Robustness tests cover common image degradations and adaptive attacks, with IDGuardian retaining low identity similarity under the tested perturbations. The section also reports that perceptual quality can decline under heavy input distortion.

B.3. Robustness Evaluation Settings.

  • Tests use JPEG quality factors 85, 70, and 50; Gaussian blur kernels 5 × 5, 7 × 7, and 11 × 11 at σ = 1.5; and noise variances 100, 400, and 900.The reported protection remains comparable to undistorted inputs for common distortions and is maintained under the stronger settings; Table 5 reports IP-Adapter Plus XL results as an example.Source: §B.3 · p. 13, Table 5 · p. 14
  • Figure 6 compares protected-image perceptual quality across three post-processing strengths and reports that quality remains largely preserved as distortions increase.Source: §B.3 · p. 14

B.4. Robustness Evaluation on Identity Similarity and Perceptual Quality Metrics

  • Across common degradations and adaptive attacks such as Impress, IDGuardian maintains low identity similarity, although perceptual quality metrics decline under heavy input distortion.Qualitative examples also show identity-relevant features remaining suppressed under post-processing transformations.Source: §B.4 · p. 14

C.1. Visualization of the Main Ablation Results

The appendix visualizes the identity-loss and guidance ablation, then tunes perturbation settings and optimization length. It selects a perturbation configuration balancing identity suppression and image quality, and uses 200 iterations based on loss convergence.

C.1. Visualization of the Main Ablation Results

  • IDGuardian achieves superior identity protection while maintaining high visual quality across the tested identity-loss combinations and guidance directions.The visualization complements the quantitative ablation results in Table 4 of the main text.Source: §C.1 · p. 14

C.2. Perturbation Budget and Learning Rate

  • Below 0.15 identity consistency across ArcFace, FaceNet, and VGGFace, with SSIM exceeding 0.8419 and PSNR surpassing 32.186 dB, for α = 0.005, ϵ = 8/255.The authors report this configuration as the best balance between identity suppression and perceptual quality.Source: §C.2 · p. 14, Figure 7 · p. 15
  • At ϵ = 4/255 identity suppression is insufficient, while ϵ = 16/255 compromises visual fidelity; α = 0.001 under-optimizes and α = 0.01 introduces instability and artifacts.The findings motivate jointly tuning perturbation magnitude and learning rate.Source: §C.2 · p. 14

C.3. Number of Optimization Steps

  • 200 iterations are used by default because identity loss saturates around that point and further gains become marginal.Identity loss declines rapidly during the first 100 steps; the figure’s shaded band is a heuristic stability indicator, not a confidence interval.Source: §C.3 · pp. 14–15

D.1. Evaluation on Commercial Face Personalization APIs

IDGuardian provides a degree of defense on commercial face-personalization APIs and maintains identity suppression across surrogate models and datasets. Evaluations on CelebA-HQ and LFW also report cross-dataset generalization.

D.1. Evaluation on Commercial Face Personalization APIs

  • IDGuardian provides a degree of defense under black-box commercial API conditions, demonstrating transferability beyond the training environment.The evaluation covers Seedream 4.0, Owen-Image, Kling-AI, and Ihuiwa.Source: §D.1 · p. 15, Figure 9 · p. 16

D.2. Flexibility in Surrogate Selection

  • Identity suppression remains strong across distinct surrogate generative models when attack configurations are held constant.The comparison includes SDXL and SD1.5 backbones and their corresponding IP-Adapter variants.Source: §D.2 · p. 16

D.3. Dataset Generalization

  • On CelebA-HQ, IDGuardian maintains identity suppression and reduced perceptual quality across metrics, resolutions, and demographic compositions; LFW results are consistent with VGGFace2 and CelebA-HQ.The reported comparisons use identity-similarity and perceptual-quality measures.Source: §D.3 · p. 16

E. Human and GPT-Based Evaluation of Identity Protection Effectiveness

Human and GPT evaluations report lower identity similarity for IDGuardian outputs than for comparison methods. Their judgments favor the method in most tested comparisons, while the study reports participant consent and adherence to recognized ethical principles.

E. Human and GPT-Based Evaluation of Identity Protection Effectiveness

E.1. Human Study

  • Human participants rated IDGuardian-generated images as having lower perceived identity similarity than images from state-of-the-art baseline methods.The subjective assessment complements quantitative metrics by capturing perceptual differences.Source: §E.1 · p. 16

E.2. GPT-Based Evaluation

  • GPT compared pairs of original and generated images by identifying which generated image had the greatest discrepancy in identity features; its findings agreed with the user study.The authors describe this automated assessment as a scalable and efficient complement to human judgment.Source: §E.2 · pp. 16–17

E.3. Ethics Statement

  • Participants were informed about the study’s purpose and provided consent, and the study followed recognized ethical research principles.Source: §E.2 · p. 17

Related papers

Loading 2610.19717v1…