PaperlayerView exact revision on arXiv

Source-linked AI summary

Reverse Personalization

Han-Wei Kung, Tuomas Varanka, Nicu Sebe

arXiv:2512.22984v1cs.CV

TL;DR

Prompt-based anonymization may depend on a subject being represented in the pretrained model or require identity-specific fine-tuning, while existing methods offer limited control over facial attributes. The paper uses conditional diffusion inversion and identity-guided conditioning for image-only reverse personalization. It reports a balance between identity removal, attribute preservation, and image quality, with controllable facial attributes.

  • Problem

    Prompt-based identity modification may fail for subjects absent from a model’s learned feature space or require identity-specific fine-tuning, and anonymization methods offer limited attribute control.

    Source: §1 · p. 2

  • Method

    The framework combines conditional diffusion inversion with identity-conditioned guidance to manipulate input faces without text prompts or model retraining.

    Source: §1 · p. 2

  • Results

    The method claims an optimal balance among identity obfuscation, attribute preservation, and visual quality, with controls for facial attributes.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The framework supports user-controlled anonymization that can selectively preserve or remove facial attributes while maintaining realism and utility for identity-agnostic tasks.

    Source: §1 · p. 2

    The current framework anonymizes single images and lacks temporal consistency when applied to videos.

    Source: §8 · p. 14

Abstract

from arXiv · show

Recent text-to-image diffusion models have demonstrated remarkable generation of realistic facial images conditioned on textual prompts and human identities, enabling creating personalized facial imagery. However, existing prompt-based methods for removing or modifying identity-specific features rely either on the subject being well-represented in the pre-trained model or require model fine-tuning for specific identities. In this work, we analyze the identity generation process and introduce a reverse personalization framework for face anonymization. Our approach leverages conditional diffusion inversion, allowing direct manipulation of images without using text prompts. To generalize beyond subjects in the model's training data, we incorporate an identity-guided conditioning branch. Unlike prior anonymization methods, which lack control over facial attributes, our framework supports attribute-controllable anonymization. We demonstrate that our method achieves a state-of-the-art balance between identity removal, attribute preservation, and image quality. Source code and data are available at https://github.com/hanweikung/reverse-personalization .

1. Introduction

The introduction frames face anonymization as removing identity-specific features while retaining useful attributes and realism, gaps that prompt-based methods and prior anonymization approaches do not fully address. It presents reverse personalization as an image-only, attribute-controllable approach that repurposes personalization techniques to suppress identity.

1. Introduction

  • Prompt attention adjustment can have negligible effect for noncelebrities whose identities are absent from the model’s learned feature space.The introduction identifies this as a limitation of modifying attention weights on celebrity names.Source: §1 · pp. 1–2
  • Conditional diffusion inversion and identity-conditioned face embeddings support manipulation of input images without text prompts or model retraining.The approach extracts identity features from arbitrary input images and allows high-level facial-attribute control.Source: §1 · p. 2
  • The authors find that increasing classifier-free guidance amplifies identity-defining features, motivating a reverse-personalization mechanism to suppress them.They describe generation as first producing a neutral face and then progressively injecting characteristic traits.Source: §1 · p. 2
  • The framework claims an optimal balance of identity obfuscation, attribute preservation, and visual quality, with controls for facial attributes such as age, gender, and ethnicity.It is presented as applicable to arbitrary individuals without prior model exposure or identity-specific fine-tuning.Source: §1 · p. 2

2. Related Work

  • The paper repurposes personalization techniques, usually aimed at preserving identity or appearance, toward attribute-controllable face anonymization.The framework supports selective preservation or removal of facial attributes while maintaining realism and utility for identity-agnostic tasks.Source: §2 · p. 3
  • Anonymization methods face a utility trade-off: traditional blur, pixelation, and masking degrade contextual cues, while newer approaches still struggle with utility and attribute control.Identity-agnostic cues such as expressions, pose, and background can matter for downstream tasks.Source: §1 · p. 2, §2 · p. 3

3. Method

The method uses identity-conditioned diffusion inversion to modify facial identity without prompt-based reweighting, then supports user control over which semantic facial attributes are retained or changed.

3.1. Preliminary

  • DDPM inversion recovers stepwise noise from predicted means and variance, while IP-Adapter adds weighted image-prompt attention without retraining.Source: §3.1 · p. 3

3.2. Motivation

  • Prompt reweighting depends on the model knowing the subject, while personalization can move beyond prior knowledge but requires fine-tuning and may fail with few reference images.Source: §3.2 · p. 4

3.3. Reverse personalization

  • With one image, prompt substitution is inapplicable, so null-identity conditioning guides inversion; DPM-Solver++ speeds inversion and is said to ensure perfect input reconstruction.Source: §3.3 · p. 4
  • Classifier-free guidance combines identity-conditioned and null-identity predictions; a negative scale steers generation away from identity-defining features.Source: §3.3 · p. 4

3.4. Attribute-controllable anonymization

  • Users specify desired facial attributes in a new text prompt while reusing the original noise trajectory, enabling selective attribute control during anonymization.Source: §3.4 · p. 5

4. Experiments

Experiments compare reverse personalization with existing anonymization methods, test its hyperparameters and attribute control, and evaluate alternative implementations. The results examine identity removal alongside attribute preservation and image quality.

4.1. Hyperparameter analysis

  • More-negative classifier-free guidance lowers re-identification rates but excessively negative scales worsen image quality; increasing IP-Adapter scale improves anonymization by strengthening identity conditioning.The analysis covers CelebA-HQ and FFHQ; quality changes include higher FID and lower face-specific IQA scores.Source: §4.1 · pp. 5–6, Figure 5 · p. 6

4.2. Comparison with existing methods

  • Reverse personalization and NullFace show balanced performance, while NullFace’s lower re-identification rates come with weaker attribute preservation and image quality.The comparison covers CelebA-HQ and FFHQ.Source: Table 1 · p. 6, §4.2 · p. 6

4.3. Attribute-controllable anonymization

  • Attribute-controlled anonymization preserves all three evaluated attributes most accurately on CelebA-HQ and two of three on FFHQ, where DP2 slightly outperforms it for race.The evaluated attributes are age, sex, and race.Source: Table 2 · p. 7, §4.3 · p. 7

4.4. Ablation study

  • DDIM inversion underperforms across metrics, with reconstruction and guidance-scale weaknesses associated with poorer anonymization, attribute preservation, and image quality.Qualitative results also show DDIM outputs failing to preserve expression, gaze, and pose and introducing artifacts.Source: §4.4 · pp. 7–8, Table 3 · p. 8
  • InstantID achieves lower re-identification rates than SDXL but has reduced attribute preservation and higher FID; its ID embedding entangles identity with other facial attributes.Neither alternative implementation matches SDXL in high-level attribute preservation.Source: §4.4 · pp. 7–8, Table 4 · p. 8

5. Conclusion

The conclusion presents reverse personalization as a way to anonymize faces without prior exposure to the subject or model fine-tuning. It reports a balance among identity obfuscation, attribute preservation, and image quality, with ablation visuals indicating realism and attribute preservation.

5. Conclusion

  • The framework requires neither prior subject exposure in the diffusion model nor model fine-tuning, supporting use with arbitrary individuals.It is presented as a face-anonymization method that removes identity-specific features while preserving non-identity attributes.Source: §5 · p. 8
  • The ablation study’s qualitative results show the method maintaining realism and accurate attribute preservation.Source: Figure 9 · p. 8
  • The method aims for an optimal balance among identity obfuscation, attribute preservation, and image quality.The conclusion presents this balance as part of its flexible anonymization capability.Source: §5 · p. 8

6. Balanced performance across privacy and utility

Across five privacy–utility plots on CelebA-HQ and FFHQ, the method is reported closest to the plotted optimal balance. The plots compare re-identification with utility measures for facial attributes and image quality.

6. Balanced performance across privacy and utility

  • The method consistently lies closest to the privacy–utility “sweet spot” across the five plots on CelebA-HQ and FFHQ.The authors report that the other methods compromise either privacy or utility.Source: §6 · p. 13, Figure 10 · p. 13
  • The plots compare re-identification rates against expression, gaze, pose, FID, and Face IQA utility measures.Source: §6 · p. 13
  • The optimal region is the lower-left plot corner, except for Face IQA, where it is the upper-left corner.The gradient background marks that region.Source: §6 · p. 13, Figure 10 · p. 13

7. Privacy-utility trade-off across guidance scales

More negative classifier-free guidance scales lower re-identification rates but also worsen attribute preservation and image quality. The authors attribute this trade-off to excessively negative guidance imposing the synthetic identity too strongly.

7. Privacy-utility trade-off across guidance scales

  • More negative guidance scales lower re-identification rates while increasing expression, gaze, and pose distances and worsening image quality.Image quality worsens as FID rises and Face IQA falls.Source: §7 · p. 13, Figure 11 · p. 13
  • The authors attribute these costs to excessively negative guidance imposing the synthetic identity too strongly and altering non-identity attributes.Source: §7 · p. 13
  • The comparison covers guidance scales -20, -15, -10, and -5, with re-identification measured using AdaFace and SwinFace.Source: §7 · p. 13

8. Challenging cases and limitations

The method and baselines struggle to preserve rare or extreme expressions, which the authors link likely to limited training data. Image quality is bounded by the underlying diffusion model, and the current framework processes single images without video temporal consistency.

8. Challenging cases and limitations

  • Both the method and baselines struggle to preserve extreme or uncommon expressions, likely because training data for them is limited.Figure 12 shows examples of these failures.Source: §8 · p. 14, Figure 12 · p. 14
  • Unlike prior methods that struggle with extreme face angles or occlusions, this approach performs reliably in those cases.Source: §8 · p. 14
  • Image quality is bounded by the underlying diffusion model, SDXL in these experiments.The authors suggest that more advanced diffusion models could improve quality.Source: §8 · p. 14
  • The framework currently anonymizes single images; applied to video, it lacks temporal consistency.Video anonymization using video diffusion models is identified as future work.Source: §8 · p. 14

9. Identity recovery test

The paper tests whether negative classifier-free guidance can reverse anonymization and recover the original identity. It reports that recovery attempts fail to restore the original identity.

9. Identity recovery test

  • Anonymization depends on interactions among model architecture, inversion, and hyperparameters, which the authors say prevent reversal by negative classifier-free guidance.The concern is whether applying a negative guidance scale to anonymized images can recover the original identity.Source: §9 · p. 14
  • Recovered images remain visibly different from the original inputs after negative classifier-free guidance.The authors report that this visual result aligns with their quantitative findings.Source: §9 · p. 14
  • Re-identification rates remain low and comparable to anonymized images in the negative-guidance recovery test.The table caption states that these low rates indicate the original identity cannot be recovered through this method.Source: §9 · p. 14

10. Societal risks of AI-generated human faces

The paper describes risks from AI-generated human faces, including convincing fake identities, biased representations, and non-consensual sexual exploitation. It calls for coordinated action by policymakers and technologists to address these harms.

10. Societal risks of AI-generated human faces

  • Convincing fake identities on social media can foster manipulation, fraud, and disinformation.The paper notes that AI-generated faces are now nearly indistinguishable from real ones.Source: §10 · p. 14
  • Face-generation models trained on imbalanced datasets can reinforce societal prejudices and produce skewed, exclusionary representations.The passage also identifies risks of amplifying cultural, racial, and gender biases.Source: §10 · p. 14
  • The paper cites non-consensual deepfake pornography as a form of sexual exploitation and urges coordinated action by policymakers and technologists.Source: §10 · p. 14

11. Additional qualitative results

The paper presents qualitative comparisons of its anonymization results with six state-of-the-art approaches across CelebA-HQ and FFHQ. The supplied passages identify the comparison datasets and figures but do not report comparative outcomes.

11. Additional qualitative results

Loading 2512.22984v1…