PaperlayerView exact revision on arXiv

Source-linked AI summary

IDProtector: An Adversarial Noise Encoder to Protect Against ID-Preserving Image Generation

Yiren Song, Pei Yang, Hai Ci, Mike Zheng Shou

arXiv:2412.11638v2cs.CV

TL;DR

Encoder-based identity-preserving generation makes customization convenient, but leaves a gap in protection against unauthorized use of portraits. IDProtector generates adversarial noise with a noise encoder to protect portraits across encoder-based methods. The authors report reduced generated-face similarity, robustness to distortions, and effectiveness on closed-source models, while noting a visibility trade-off.

  • Problem

    Prior protection research targeted fine-tuning methods, leaving a gap in protection against encoder-based identity-preserving generation.

    Source: §1 · p. 2

  • Method

    IDProtector trains a noise encoder to generate adversarial perturbations for portraits, targeting multiple encoder-based identity-preserving methods.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    IDProtector protects an image within 0.2s and achieves significantly lower face similarity in generated images than existing protection methods.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Protection remains effective under affine transformations and JPEG compression and works against multiple ID-preserving models, including closed-source Midjourney.

    Source: §6 · p. 8

    The adversarial noise is not fully invisible, requiring a trade-off between attack effectiveness and imperceptibility.

    Source: §6 · p. 8

Abstract

from arXiv · show

Recently, zero-shot methods like InstantID have revolutionized identity-preserving generation. Unlike multi-image finetuning approaches such as DreamBooth, these zero-shot methods leverage powerful facial encoders to extract identity information from a single portrait photo, enabling efficient identity-preserving generation through a single inference pass. However, this convenience introduces new threats to the facial identity protection. This paper aims to safeguard portrait photos from unauthorized encoder-based customization. We introduce IDProtector, an adversarial noise encoder that applies imperceptible adversarial noise to portrait photos in a single forward pass. Our approach offers universal protection for portraits against multiple state-of-the-art encoder-based methods, including InstantID, IP-Adapter, and PhotoMaker, while ensuring robustness to common image transformations such as JPEG compression, resizing, and affine transformations. Experiments across diverse portrait datasets and generative models reveal that IDProtector generalizes effectively to unseen data and even closed-source proprietary models.

1. Introduction

Encoder-based identity-preserving generation simplifies customization but heightens privacy risks, while prior protection research has focused on fine-tuning methods. IDProtector targets encoder-based methods with feed-forward noise generation designed for universality, efficiency, robustness, and imperceptibility.

1. Introduction

  • IDProtector addresses the gap in protection against encoder-based ID-preserving generation, which prior work had not addressed.Earlier protection such as Anti-DreamBooth targeted fine-tuning methods including LoRA and DreamBooth.Source: §1 · p. 2
  • The method targets four challenges: universal protection, efficient generation, robustness to transformations and preprocessing, and imperceptible noise.It uses affine-matrix noise during face alignment as augmentation to improve robustness without the learning burden of post-hoc differentiable augmentations.Source: §1 · p. 2
  • IDProtector is a feed-forward method that protects portraits from state-of-the-art encoder-based identity-preserving techniques.A ViT-based encoder predicts adversarial noise, and the composite loss targets InstantID, IP-Adapter, IP-Adapter Plus, and PhotoMaker.Source: §1 · p. 2
  • IDProtector is evaluated on CelebA and tested on unseen VGG Face data and proprietary models, with reported generalization to unseen data and closed-source models.The authors describe the approach as combining a ViT-based noise encoder, joint learning objectives, and affine-based data augmentation.Source: §1 · p. 2

2. Related Works

Related work covers customized generation through fine-tuning or pretrained encoders, alongside defensive perturbations developed primarily for fine-tuning-based customization. Encoder-based identity protection remains an unaddressed gap in that defensive literature.

2.1. Tuning-based Customized Generation

  • Fine-tuning approaches adapt diffusion models through parameter or word-vector optimization, but require substantial computation and multiple images.The passage cites DreamBooth [17], Custom Diffusion, LoRA [7], and Textual Inversion [3].Source: §2.1 · p. 2

2.2. Encoder-based ID-Preserving Generation

  • Encoder-based methods enable zero-shot generation, evolving from CLIP features to ArcFace conditioning for stronger identity consistency and broader feature preservation.IP-Adapter-Plus uses patch features; StoryMaker combines ArcFace and CLIP features for faces, clothing, hairstyles, and body features.Source: §2.2 · pp. 2–3

2.3. Defensive Perturbations for Diffusion Models.

  • Prior defensive perturbations address style appropriation or fine-tuning-based customization, while protection against encoder-based methods remains unexplored.The passage describes Glaze, AdvDM, Anti-DreamBooth, and further defenses [21] [25] [28] [31].Source: §2.3 · p. 3

3. Problem Definition

The study frames portrait protection as Alice adding perturbations to hinder Bob’s encoder-based identity-preserving customization. The objective is to reduce identity similarity across generators while keeping perturbations bounded.

3. Problem Definition

  • The protector’s goal is to add perturbations that hinder exploitation while minimizing changes to the original portrait.Protection is expected to work across encoder-based methods and withstand natural distortions such as resizing, JPEG compression, and blurring.Source: §3 · p. 3
  • The exploiter may choose open-source or proprietary encoder-based models, making protection across different models a central challenge.The scenario identifies Alice as the image protector and Bob as the image exploiter.Source: §3 · p. 3
  • The formal objective trains a noise encoder to produce perturbations that lower the sum of identity-similarity values across multiple feature spaces.The comparison is between generation from the original portrait and from its perturbed version; S is cosine similarity in ArcFace and CLIP feature spaces.Source: §3 · p. 3

4. Method

IDProtector uses an adversarial noise encoder to add protective perturbations to portrait images, with training objectives targeting identity features used by encoder-based generation methods. The method also regularizes perturbations and augments face alignment during training to address image-quality and robustness concerns.

4.1. Overall Method

  • The method adds predicted adversarial noise to a portrait image, then uses feature-extraction pipelines during training to shape the protection objective.The input is resized to 224×224 before noise prediction; the perturbation is resized back to the original dimensions before being added.Source: §4.1 · pp. 3–4

4.2. Noise Encoder

  • A ViT maps a 224×224 RGB image to three-channel perturbations, while a face-location mask supplies an additional input channel.Outputs are projected to [−1, 1], denormalized to [−ϵ, ϵ], resized to the original dimensions, and added to the image.Source: §4.2 · p. 4

4.3. Adversarial ID Protection

  • IDProtector targets identity features used by InstantID, IP-Adapter, IP-Adapter-Plus, and PhotoMaker to disrupt the information available to generation models.For InstantID, the attack minimizes cosine similarity between ArcFace features of the original and protected images; other methods use selected CLIP embeddings as attack targets.Source: §4.3 · p. 4

4.4. Imperceptibility

  • An ℓ1 penalty limits predicted noise, and an auxiliary penalty discourages values beyond the ϵ-ball boundary.The final loss combines adversarial loss and regularization.Source: §4.4 · p. 5

4.5. Robustness

  • Adding Gaussian noise to the affine alignment matrix during training addresses misalignment, and the augmentation improves robustness across other branches and common image transformations.The augmentation is applied only in the InstantID branch.Source: §4.5 · p. 5

5. Experiments

Experiments compare IDProtector with protection baselines across four identity-preserving generators, then test generalization to unseen data and models and robustness to image distortions. Results report larger identity shifts, faster protection, and effectiveness beyond the training setting and under tested distortions.

5.1. Experimental Setup

  • Training used 190,000 uncropped CelebA portraits; evaluation included 50 CelebA and 50 VGG Face images, with a maximum pixel perturbation of 9/255.The study used a three-stage curriculum; lower ISM indicates greater identity shift, FDR records face detection, and SER-FIQ measures generated-face perceptual quality.Source: §5.1 · p. 5, Table 2 · p. 5

5.2. Comparison with Baselines

  • The SER-FIQ scores did not consistently match qualitative assessments, motivating the need for a more reliable generated-image quality metric.IDProtector had a noticeable impact on generated-image quality.Source: §5.2 · p. 6
  • Per-image PGD tuning reduced ISM more substantially across most methods, but required over an hour on average with IDProtector’s objective.The authors attribute the stronger reductions to per-image tuning leveraging individual image characteristics.Source: §5.2 · p. 6
  • 0.1–0.14 ISM reduction against IP-Adapter, IP-Adapter-Plus, and PhotoMaker exceeded baselines’ average 0.02 (maximum 0.06); on InstantID, IDProtector exceeded 0.4 versus baselines below 0.1.The comparison covers both datasets and four generators.Source: §5.2 · p. 6, Table 3 · p. 7
  • 0.173 seconds per image was IDProtector’s average protection time, below 1% of the fastest baseline; its PSNR and SSIM were also higher than baselines.Using PGD with IDProtector’s objective took over an hour on average.Source: Table 4 · p. 7, §5.2 · p. 7

5.3. Generalization to Unseen Data and Models

  • On unseen VGG Face, ISM reductions were approximately 0.1 for IP-Adapter and around 0.55 for InstantID on both datasets; unseen generators averaged 0.31, including 0.10 on Midjourney and 0.54 on Jing Gou.Figure 4 also shows altered facial shapes, reduced prompt-following, and decreased resemblance to the reference identity.Source: Table 5 · p. 7, §5.3 · p. 8

5.4. Robustness Evaluation

  • Under JPEG compression, cropping, added noise, and affine transformation, ISM remained comparable to undistorted results, and the visual evaluation showed dissimilar faces or artifacts across four models.The tested cropping removed up to 20% of the region; JPEG quality was 85.Source: §5.4 · p. 8, Figure 5 · p. 8

6. Conclusion and Limitation

IDProtector uses a noise encoder to protect images from encoder-based identity-preserving generation, with fast inference and protection across models and distortions. Its adversarial noise is not fully invisible, creating a trade-off between attack effectiveness and imperceptibility.

6. Conclusion and Limitation

  • IDProtector adds adversarial perturbations through a noise encoder, protecting images in 0.2s and remaining effective under affine transformations and JPEG compression.The conclusion also reports lower face similarity in generated images than existing protection methods and effectiveness against closed-source models such as Midjourney.Source: §6 · p. 8
  • The adversarial noise is not fully invisible, requiring a trade-off between attack effectiveness and imperceptibility.Future work will focus on improving imperceptibility while maintaining robustness.Source: §6 · p. 8

Image Generation

Figure 6 visualizes the face localization prior as brightness differences over the original RGB images, distinguishing pixels visible to ArcFace from those discarded.

Image Generation

  • Bright regions mark pixels retained by ArcFace, while dark regions mark pixels discarded, in an overlay of the face localization prior on the original RGB image.Source: Figure 6 · p. 11

A. Details of Prior Channels

IDProtector uses additional prior channels alongside RGB to reduce the noise encoder’s learning burden.

A. Details of Prior Channels

A.1. Face Localization Prior

IDProtector supplies face-localization and center-crop priors to communicate which image regions survive preprocessing. The reported priors reduce training losses or speed early convergence for the corresponding generation methods.

A.1. Face Localization Prior

  • The face localization prior is a binary 0/1 mask identifying facial regions preserved by InstantID preprocessing, so the network need not learn face localization independently.Source: §A.1 · p. 11
  • The face region is derived by detecting five landmarks, aligning them to ArcFace’s target format, and mapping the aligned crop’s corners back to the original image to form the mask.The alignment produces a 112×112 face image for ArcFace.Source: §A.1 · p. 11
  • The face localization prior significantly reduces PhotoMaker and InstantID losses; without it, InstantID cosine similarity plateaus around 0.5 and effective ID protection is not achieved.Source: §A.1 · p. 11, Figure 9 · p. 12

A.2. Aspect Ratio Prior

  • The center crop prior helps IP-Adapter and IPAdapter-Plus converge more rapidly during early training by marking regions retained after CLIP preprocessing.It is a binary 0/1 channel that communicates CLIP’s preprocessing to the network.Source: §A.2 · p. 12

B. Ablation on Alternative Target Embeddings

The ablation compares the current attack target with the penultimate hidden state of CLIP Vision. The alternative retains nearly comparable protection on IP-Adapter and IP-Adapter-Plus, though its facial-similarity drop is less pronounced.

B. Ablation on Alternative Target Embeddings

  • The target-selection principles include blocking potential information pathways and choosing an embedding with semantic abstraction to facilitate the attack.The authors note that their selected target is not the only viable choice.Source: B. Ablation on Alternative Target… · p. 12
  • The penultimate hidden state of CLIP Vision exhibits the two target properties to some extent and is evaluated as an alternative.Figure 8 marks this alternative in blue and the current target in red.Source: B. Ablation on Alternative Target… · p. 12
  • IP-Adapter and IP-Adapter-Plus protection remains nearly comparable with the hidden-state target, despite a less pronounced ISM facial-similarity drop.Table 7 compares the second-to-last hidden state with the current target.Source: Table 7 · p. 12, B. Ablation on Alternative Target… · p. 12
  • The current embedding performs better in this comparison, but the attack embedding is not uniquely constrained.The alternative target still achieves nearly comparable protection on IP-Adapter and IP-Adapter-Plus.Source: B. Ablation on Alternative Target… · p. 12

C. More Visualizations on Protecting against Unseen Models

Figure 10 provides visualizations of ID protection against customization models unseen during training.

C. More Visualizations on Protecting against Unseen Models

Related papers

Loading 2412.11638v2…