Source-linked AI summary
Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models
Abhishek Basu, Fahad Shamshad, Karthik Nandakumar
TL;DR
Few-shot LVLM personalization can let an adversary bind a person’s identity from a few images and detect them through natural-language queries. Anti-Persona disrupts shared identity features with image-level perturbations, and is reported to protect across evaluated settings while preserving visual fidelity.
Problem
A few publicly available photos can enable unauthorized identity binding in personalized LVLMs, whose natural-language interface supports detection in new images.
Source: §1 · p. 1Method
Anti-Persona aggregates multiple reference-image embeddings into an identity prototype and perturbs images to disrupt prototype alignment, with spatial and low-frequency refinements.
Source: §1 · pp. 2–3Results
Anti-Persona is reported strongest overall among evaluated methods while preserving visual fidelity, covering reactive protection on Yo’LLaVA and MyVLM and proactive protection on Yo’LLaVA.
Source: §1 · p. 3, §6 · p. 12Takeaways & Limitations
These results motivate stronger privacy safeguards for personalized multimodal systems.
Source: §6 · p. 12The threat model assumes a non-adaptive adversary using an off-the-shelf personalized LVLM and excludes personalization changes, purification, adversarial training, and explicit perturbation removal.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Few-shot personalization enables large vision--language models (LVLMs) to learn user-specific visual concepts for applications such as personalized retrieval and subject-aware querying. However, it also creates a privacy risk: an adversary can bind a target identity from a few reference images and subsequently detect that identity in new images through natural-language queries. We introduce Anti-Persona, an image-level defense against unauthorized identity binding and recognition in personalized LVLMs. Our key insight is that identity personalization relies on visual features shared across multiple reference images. We aggregate these features into an identity prototype and optimize visually subtle perturbations that disrupt prototype alignment in the vision-encoder space. Spatial smoothing and low-frequency preservation further promote visual fidelity and practical resilience to image compression. The resulting protection does not depend on a specific prompt and supports both proactive anti-personalization and reactive image protection. Experiments on two representative personalized LVLMs demonstrate protection rates of up to $95.0\%$ while preserving visual fidelity. The method remains stable across prompt variations and evaluated identity-query tasks, and improves black-box transfer under encoder mismatch.
1 Introduction
Few-shot personalization lets LVLMs recognize user-specific identities, but it also creates a privacy risk of unauthorized identity binding and prompt-based detection. Anti-Persona targets shared identity cues with image-level perturbations and reports strong protection while preserving visual fidelity.
1 Introduction
- A few public photos can let an adversary personalize an LVLM to detect a person in new images through natural-language queries, without that person’s consent.Unlike conventional face-recognition systems, the prompt-based interface lowers the barrier to unauthorized identity detection.Source: §1 · pp. 1, 3
- Existing image defenses primarily target generative misuse, whereas personalized LVLMs can expose identity through recognition without synthesizing the target.This difference motivates protection specifically for identity binding and prompt-based recognition.Source: §1 · p. 2
- Anti-Persona aggregates reference-image representations into an identity prototype and perturbs images to weaken alignment with recurring identity cues.This targets identity-consistent structure across images rather than only image-specific features.Source: §1 · pp. 2–3
- Spatial coherence and low-frequency perturbation energy are designed to reduce visible artifacts and improve resilience to JPEG compression.These refinements preserve the identity-disruption objective under realistic image sharing.Source: §1 · pp. 2–3
- Anti-Persona achieves the strongest overall protection among evaluated methods while preserving visual fidelity, with stability across prompts, VQA-style identity queries, and encoder mismatch.The evaluation covers proactive and reactive protection on representative personalized LVLMs.Source: §1 · p. 3
2 Related Work
Prior work studies personalization in LVLMs and image-level defenses against unauthorized use, but generative and recognition-based personalization pose different protection objectives.
2 Related Work
- LVLM personalization learns to recognize and interact with user-specific concepts, using approaches such as Yo’LLaVA’s identifier tokens and MyVLM’s concept head.These systems acquire identity-specific representations from few examples for personalized retrieval and subject-aware querying.Source: §2 · p. 3
- Image-protection methods such as PhotoGuard [23], Glaze, Anti-DreamBooth, MIST [13], Anti-Diffusion, and AdvDM [14] target generative misuse such as editing, style imitation, or subject-driven synthesis.Their protection objective is to prevent faithful editing, mimicry, or reproduction of protected content.Source: §2 · p. 3
- Recognition-based LVLM personalization instead requires disrupting identity features that support prompt-based detection, a goal generative defenses are not directly optimized to meet.The distinction motivates image-level protection tailored to identity binding and recognition.Source: §2 · pp. 3–4
3 Preliminaries
Personalized LVLMs learn lightweight subject-specific parameters from a few images while retaining a shared vision-encoder interface. The threat model considers adversarial identity recognition and protection applied either before personalization or to later query images.
3.1 Personalization in LVLMs
- Personalization learns lightweight subject-specific parameters from same-identity reference images while keeping the pretrained vision-language backbone frozen.Yo’LLaVA uses latent identifier tokens, whereas MyVLM uses a concept head; both rely on encoder representations.Source: §3.1 · p. 4
3.2 Threat Model
- The threat is an adversary learning a target identity from references and using natural-language prompts to test for that person in new images.The considered adversary is non-adaptive and uses an off-the-shelf personalized LVLM; modifying personalization, purification, adversarial training, and explicit perturbation removal are outside scope.Source: §3.2 · p. 4
- The defender aims to prevent reliable recognition while preserving visual fidelity, using the target encoder in gray-box settings or a surrogate encoder without target parameters or gradients in black-box settings.Protection may be proactive, applied to references before personalization, or reactive, applied to new queries after clean-reference personalization.Source: §3.2 · pp. 4–5
- The protection framework aggregates reference-image encoder representations into an identity prototype, then optimizes and refines the protected image.Its combined loss disrupts prototype alignment, reduces similarity to the clean image, and anchors toward a neutral target before spatial and frequency refinements.Source: Figure 2 · p. 5
4 Method
The method protects images by perturbing identity-relevant vision-encoder features, targeting structure shared across reference images while refining perturbations for visual unobtrusiveness and compression resilience.
4 Method
- Perturbations disrupt identity-relevant encoder features, target identity-consistent structure shared across reference images, and are designed to remain visually unobtrusive and robust to realistic image processing.The method operates in the vision-encoder representation space, the shared interface for visual identity information used by downstream personalization modules.Source: §4 · p. 5
4.1 Identity Prototype
- The identity prototype is the centroid of reference-image embeddings, capturing shared identity features while averaging out pose, lighting, and background variation.Moving protected-image representations away from this prototype targets features consistent across the identity rather than features specific to individual images.Source: §4.1 · p. 6
4.2 Protection Objective
- The protection objective combines prototype-level identity disruption, instance-level repulsion from the clean image, and target anchoring to stabilize optimization.The instance-level term suppresses residual image-specific cues, complementing disruption of the shared identity prototype.Source: §4.2 · p. 7
- A fixed target image provides a consistent optimization direction, while projected gradient descent under an ℓ∞ constraint enforces imperceptibility.The target is a gray image used as a neutral anchor that does not correspond to a specific identity.Source: §4.2 · p. 7
4.3 Perturbation Refinement
- Gaussian smoothing suppresses abrupt local perturbation variations, encouraging spatial coherence in visually sensitive regions while reducing potentially visible artifacts.The refinement is applied after optimization steps and preserves the perturbation's alignment with the identity-disruption objective.Source: §4.3 · pp. 7–8
- Patch-wise DCT filtering retains low-frequency perturbation components that are more likely to survive JPEG compression than attenuated high-frequency components.The low-frequency refinement is designed to improve compression robustness while preserving identity disruption.Source: §4.3 · p. 8
5 Experiments
Experiments span reactive and proactive protection, black-box transfer, prompt and query variation, JPEG compression, and reference-set size. Strict black-box protection is 30.7%, compared with 29.1% for MI-FGSM.
5.1 Experimental Setup
- Evaluation covers Yo’LLaVA and MyVLM, ten identities, shared baseline protocols, and reactive clean-reference versus proactive protected-reference testing.Protection rate measures identity-recognition failure; PSNR and SSIM quantify visual fidelity.Source: §5.1 · pp. 8–9, Table 1 · p. 9, Table 2 · p. 9
5.2.1 Main Protection Results
- Figure 3 shows protected images retaining local detail with fewer artifacts and appearing closer to clean images than TI-FGSM and PGD.The visual comparison is consistent with higher PSNR and SSIM than those baselines.Source: §5.2.1 · p. 9, Figure 3 · p. 10
5.2.2 Black-Box Transfer
- 30.7% average protection under strict black-box transfer exceeds MI-FGSM’s 29.1%, although performance remains below gray-box results.The target encoder is never used during protection generation.Source: Table 3 · p. 10, §5.2.2 · p. 11
- Across five equivalent subject-presence prompts, protection ranges from 86.0% to 93.0%, indicating limited sensitivity to prompt wording.Source: §5.2.2 · p. 10, Table 4 · p. 11
5.2.3 Robustness and Sensitivity Analyses
- 45.8% mean protection on MyVLM VQA-style identity queries exceeds TI-FGSM’s 40.0% by 5.8 percentage points.Anti-Persona records the best result for three of four identities.Source: Table 5 · p. 11, §5.2.3 · p. 11
- At Q = 95, protection is 86.7%, 27.8 percentage points above MI-FGSM; it is 37.8% at Q = 85 and 13.3% at Q = 75.At Q = 65, all methods fall to 10.0% or below, with Anti-Persona at 8.9%.Source: §5.2.3 · p. 11, Table 6 · p. 12
- Protection rises from 87.8% with one or two references to 90.0%, 91.1%, and 93.3% with four, six, and nine references, respectively.The 5.5-point improvement is consistent with aggregating identity information across multiple images.Source: §5.2.3 · p. 11, Figure 4 · p. 12
6 Conclusion
Anti-Persona is an image-level defense against unauthorized identity recognition in personalized LVLMs. It disrupts identity-consistent encoder features while refining perturbations for visual quality and compression resilience.
6 Conclusion
- Anti-Persona protects against unauthorized identity recognition in personalized LVLMs.It aggregates reference embeddings into an identity prototype and disrupts identity-consistent encoder features.Source: §6 · p. 12
- Spatial and frequency-domain refinements are designed to preserve visual quality and improve compression resilience.Source: §6 · p. 12
- Experiments show reactive protection on Yo’LLaVA and MyVLM and proactive protection on Yo’LLaVA, with stability across prompts, VQA-style queries, and improved black-box transfer.Source: §6 · p. 12
A Ablation Studies
The ablation studies use reactive Yo’LLaVA under a victim-aligned gray-box setting, with protection rate and PSNR as evaluation measures.
A Ablation Studies
- Ablations use reactive Yo’LLaVA in the victim-aligned gray-box setting with CLIP ViT-L/14@336, ϵ=4/255, and seed 42; PR is measured over 90 pooled queries across 10 identities.PSNR is computed between the clean and protected images.Source: A Ablation Studies · p. 15
- Protection rate is measured over 90 pooled queries across 10 identities.Source: A Ablation Studies · p. 15
- PSNR is computed between clean and protected images.Source: A Ablation Studies · p. 15
A.1 Protection Objective and Perturbation Refinements
The component ablation varies active loss terms and perturbation refinements on reactive Yo’LLaVA. The full configuration achieves the highest observed protection rate with high PSNR.
A.1 Protection Objective and Perturbation Refinements
- 93.3% protection rate and 39.30 dB PSNR are achieved by the full configuration, the highest observed protection rate.Source: §A.1 · p. 15, Table A1 · p. 15
- The ablation varies active loss terms and perturbation refinements, with loss-only rows omitting both refinements and refinement rows using the complete objective.Source: §A.1 · p. 15
- Table A1 reports component ablations on reactive Yo’LLaVA in the victim-aligned gray-box setting, with PSNR in dB.Source: Table A1 · p. 15
A.2 Refinement Hyperparameters
The refinement hyperparameter sweeps vary Gaussian smoothing width and the DCT keep parameter. The selected setting gives the highest observed protection rate, while a nearby setting trades some protection for PSNR.
A.2 Refinement Hyperparameters
- The selected setting (σ, b) = (0.8, 5) provides the highest observed protection rate in the sweeps.Source: §A.2 · p. 15
- Setting b = 4 gives slightly higher PSNR with a small reduction in protection rate.Source: §A.2 · p. 15
- Table A2 varies one parameter at a time while the other remains at its selected value on reactive Yo’LLaVA at ϵ = 4/255.Source: §A.2 · p. 15
B Computational Cost
The computational-cost analysis compares Anti-Persona with PGD under the default 500-step configuration and reports the scope and setup of its measurements.
B Computational Cost
- PGD and Anti-Persona have the same dominant encoder computation per step because they use the same vision encoder.The comparison uses the default 500-step configuration.Source: B Computational Cost · p. 16
- Prototype features are computed once per identity and reused when protecting that identity’s images.Source: B Computational Cost · p. 16
- Wall-clock time and peak memory include method-specific loss and refinement operations, while encoder FLOPs are analytically estimated.The table reports a single-run measurement on one NVIDIA RTX A6000 with CLIP ViT-L/14@336 and ϵ = 4/255.Source: Table A3 · p. 16, B Computational Cost · p. 16