Source-linked AI summary
The Nearest Target Is the Wrong One: Target Separation in Arc2Face Identity Unlearning
Zeynel Tok
TL;DR
Identity unlearning in a face-conditioned generator can fail when a numerically different redirection target remains recognisably close to the forgotten person. The paper audits Arc2Face with an ArcFace-based projection adapter and controlled hard-neighbour target policies, finding that least-similar targets improve clean forgetting while retention stays nearly unchanged. It concludes that target separation is a measurable design variable for identity unlearning.
Problem
Identity-conditioned generation requires recognition-based auditing because visual change or numerical embedding difference may leave residual identity leakage.
Method
A projection adapter redirects Arc2Face’s identity conditioning before generation, and four policies select among the same hard-retain candidate set.
Results
Clean forgetting improves from 9/30 groups under nearest-hard targeting to 30/30 under least-sim-hard targeting, while mean retention remains essentially unchanged.
Takeaways & Limitations
Target separation should be specified, reported, and stress-tested as a practical design axis in identity-pathway unlearning.
Takeaways & Limitations
The method is adapter-based redirection rather than base-model erasure and does not provide a persistent deletion guarantee for Arc2Face’s weights.
Abstract
from arXiv · showhide
Unlearning an identity from a face-conditioned generator by redirecting its conditioning embedding can silently fail if the redirected output is still verified as the original person. We show that this failure depends on a controllable choice of how far the redirection target lies from the forget identity in recognition space, and that the most intuitive target, the nearest neighbour, is the one most likely to cause it. We audit Arc2Face with a locked ArcFace protocol and a projection adapter that redirects identity conditioning before generation. On a hard-neighbour stress test built from the hardest 0.5% of eligible identities, four target-selection policies show a monotonic response: clean forgetting rises from 9/30 groups under the nearest hard target to 30/30 under the least similar one. Mean forget-identity re-identification falls from 51.9 to 0.0 while mean retention stays flat. This reflects successful redirection rather than outputs becoming unverifiable: 710 of 720 least-sim-hard generations arrive at the chosen target, with no leakage to unrelated identities. Re-verifying identical images with an independent recogniser (AdaFace) preserves that trend, correlating at r=0.94, arguing against a verifier artefact. Target separation is thus a first-order, reportable design variable for identity unlearning.
1 Introduction
Identity unlearning can silently fail when a redirected embedding remains verifiably close to the forgotten identity. The paper isolates target geometry with an identity-pathway adapter and shows that farther targets improve forgetting without reducing retention.
- Motivation: A redirected conditioning embedding can remain close enough to the forget identity for generated faces to re-verify as that person.The paper argues that visual or prompt-level change is insufficient without recognition-based verification.
- Research question: The study asks whether redirection geometry, rather than merely redirection itself, determines success under hard-neighbour evaluation.
- Approach: The projection adapter intervenes directly in Arc2Face’s identity-conditioning pathway before generation while leaving generator weights unchanged.
2 Related Work
The paper situates identity unlearning within identity-conditioned face generation, verification-focused unlearning evaluation, and destination-sensitive boundary shifts. It adopts identity replacement while changing where the intervention occurs.
- Identity-conditioned generation: Arc2Face generates photorealistic faces from ArcFace identity embeddings, making it a case study for removing a reproducible person’s identity.
- Identity unlearning: Unlearning audits can miss residual recognition similarity, so evading a gallery match does not necessarily establish that an identity was forgotten.
- Identity replacement: The paper adopts identity replacement framing but intervenes upstream in the identity-conditioning pathway rather than fine-tuning identity-sensitive cross-attention layers.
- Identity replacement: A faithful PIU reproduction did not converge to a usable forget/retain point in the authors’ codebase, so it is not reported as a comparative baseline.The authors note that this may reflect reimplementation or unspecified original details rather than PIU itself.
- Related perspective: Boundary Unlearning provides an analogue in which the destination of redirected samples affects whether a decision-boundary shift registers as forgetting.
3 Preliminaries
The benchmark uses split-faithful identity references, closed-set threshold verification, and separate forget and retain metrics. It deliberately stresses the hardest 0.5% of eligible identities rather than estimating population-average performance.
- Problem setup: Each group contains one forget identity and six retain identities, with disjoint fitting references for conditioning and held-out references for verification.
- Identity verification: Verification predicts the closest gallery identity by cosine similarity and requires the predicted identity’s score to exceed fixed threshold τ.
- Evaluation protocol: Detected-face exclusions are applied before metrics, and detection coverage is high and essentially constant across policies.
- Metrics: FA measures residual forget-identity re-identification with lower values better, while RA measures intended retain-identity matching with higher values better.
- Metrics: IdentityLeak@8 records whether at least one of eight fixed-seed outputs for a forget-reference verifies as the forget identity.
- Benchmark scope: The benchmark is an adversarial stress test of local identity entanglement, not an estimate of population-average unlearning performance.It focuses on the extreme hard-neighbour tail and does not extend the main benchmark to milder 1–2% cohorts.
- Benchmark construction: The cohort contains 30 identities from the hardest 0.5% of 5,956 eligible non-public identities, with 168 generated images per group.
4 Method
The method learns a low-rank adapter that redirects forget embeddings while preserving retain and background embeddings, then compares target-selection policies within the same hard candidate sets. The benchmark cohort is selected from the extreme hard-neighbour tail.
- Identity-pathway projection adapter: The rank-8 residual adapter modifies Arc2Face’s 512-dimensional identity conditioning while leaving generator weights untouched.It uses D ∈R8×512 and U ∈R512×8, adding 8,192 learned parameters.
- Benchmark cohort: The selected hard-neighbour cohort lies in the low-distance tail of mean distance to the three nearest identities.
- Training objective: The adapter objective redirects forget references toward a selected target while retaining retain identities and preserving a background batch of embeddings.
- Training objective: Training uses λr = 5.0, λp = 2.0, and λreg = 0.001 for 1,000 AdamW steps at learning rate 0.03.
- Inference: The adapter is applied uniformly to every identity embedding without an identity-specific gate, so the method is identity-pathway redirection rather than base-weight erasure.
- Target selection: Nearest-hard, random-hard, median-hard, and least-sim-hard choose among the same three hard-retain candidates using their similarity ranking to the forget identity.
5 Experimental Setup
The experiment fixes generation settings across four target-selection policies and trains a separate projection adapter for each forget identity and policy. Checkpoints are selected using training-pool forget, retain, and preserve losses, with no evaluation references used for selection.
- Experimental Setup: 168 attempted generations per group result from three fit references and eight fixed seeds across the fixed Arc2Face configuration.Generation uses fp16 precision, 25 inference steps, guidance scale 3.0, and 512×512 resolution.
- Experimental Setup: Each forget identity and target-selection policy receives a separately trained projection-adapter checkpoint.The checkpoint is selected using the highest Sckpt from forget, retain, and preserve losses evaluated only on the training pool.
- Experimental Setup: The checkpoint score combines weighted forget, retain, and preserve objectives as Sckpt = (1-Lforget) + 0.5(1-Lretain) + 0.5(1-Lpreserve).No evaluation-reference image contributes to checkpoint selection.
- Experimental Setup: Results use the complete canonical n = 30 hard-neighbour benchmark across all four target-selection policies and the RAtarget/RAhard/RArandom retention split.The benchmark is reported unless a result is explicitly stated otherwise.
6 Results
Across a four-policy hard-neighbour benchmark, increasing target separation monotonically improves forgetting while aggregate retention remains nearly unchanged. The improvement reflects redirection to the selected target, persists under AdaFace re-verification, and is reproduced in nested cohort sub-cuts.
- 6.1 Hard-neighbour Benchmark: 9/30 groups reach FA = 0 under nearest-hard targeting, compared with 30/30 under least-sim-hard targeting.Both policies use the same benchmark, adapter architecture, training objective, and hard-retain candidate set.
- 6.2 Retention Split by Retain Role: Mean RA remains 88.75–88.86 across policies, while target-identity RA rises from 53.89 under nearest-hard to 94.28 under least-sim-hard.Hard-retain RA remains near 80 and random-retain RA near 98 across policies, so aggregate retention masks a target-specific swing.
- 6.3 Effect of Target Separation: Target arrival among detected forget generations rises from 48.0% to 100.0% as mean Leak@8 falls from 67% to 0%.Under least-sim-hard, 710/710 detected generations verify as the selected target, with 710/720 = 98.6% target arrival when no-face attempts count as non-arrivals.
- 6.1 Hard-neighbour Benchmark: Mean target similarity falls from 0.8689 to 0.2874 while mean FA falls from 51.94 to 0.00 and mean RA remains mostly unchanged.Random-hard and median-hard lie between the endpoints, forming a monotonic sweep rather than a single endpoint comparison.
- 6.3 Effect of Target Separation: Median ArcFace similarity to the forget identity falls from 0.74 under nearest-hard to 0.23 under least-sim-hard, while mean FA falls from 51.94% to 0%.This image-level collapse accompanies successful target redirection rather than merely unverifiable outputs.
- 6.4 Second-recogniser Robustness: AdaFace preserves the trend on identical generated images, with group-policy FA agreement at r = 0.94 and FA = 0 on 84/120 group-policy pairs.The result argues against an ArcFace-verifier-only artefact but does not establish representation-independent generalisation.
- 6.4 Second-recogniser Robustness: On nested hardest-10 and hardest-20 sub-cuts, FA = 0 groups rise from 5/10 to 10/10 and from 7/20 to 20/20, respectively.Mean RA stays within about 1.5 percentage points of the full-cohort value; these are sensitivity analyses, not independent replications.
7 Discussion
The discussion shows that target separation is a decisive design variable for identity-pathway unlearning, while the controlled benchmark leaves important generalization and fidelity questions open.
- Mean FA falls from 51.94 to 0.00 as target separation increases, while RA remains essentially unchanged.The sweep changes only the redirection target under an identical objective and training budget.
- Nearest-hard targeting leaves the most residual source verification, whereas least-similar targeting eliminates it under the same experimental setup.
- RAtarget improves from approximately 54 to 94 as separation increases, while aggregate RA stays flat.The improvement is concentrated in the selected target identity’s retention rather than reflecting a forgetting/retention trade-off.
- The hard-neighbour benchmark is designed for failure discovery and does not establish the reported effect size as average-case identity unlearning.
- Future evaluation should search larger identity pools under separation, retain-risk, and bystander-leakage constraints and measure perceptual and semantic fidelity directly.The paper specifically names pose, expression, attributes, and demographic presentation as fidelity dimensions for future evaluation.
8 Limitations
The study is constrained to ArcFace-based, adapter-based redirection in a controlled hard-neighbour benchmark, with limited verification and image-quality coverage.
- The intervention and target-selection rule depend on ArcFace geometry, despite AdaFace re-verification arguing against an ArcFace-only verifier artefact.
- The method learns an identity-specific adapter before generation rather than erasing Arc2Face’s base weights, so it provides no persistent deletion guarantee.
- Each adapter targets one forget identity, and policies select among three hard-retain candidates instead of searching the full eligible identity pool.
- The study primarily evaluates identity verification and does not establish preservation of pose, expression, attributes, or demographic presentation.Two training-free quality scores show no material policy difference, but neither is a calibrated standalone benchmark and perceptual checks were not reported.
- The main protocol uses a seven-identity gallery; large-population verification is a distinct operating regime, with 4.5–4.6% retain-role bystander leakage.Forget-role bystander leakage is 0.0% across all four policies in the larger 5,964-identity check.
- Using real hard-retain identities as targets is a controlled benchmark choice, not a deployment recommendation, because it may transfer privacy or misuse risk.
9 Conclusion
The audit identifies target separation as a practical design axis for Arc2Face identity redirection, while transfer to other conditioning representations remains unresolved.
- Least-sim-hard targeting raises clean-forgetting success from 9/30 to 30/30 groups relative to nearest-hard targeting while leaving retention nearly unchanged.The result remains stable across nested 10-, 20-, and 30-identity sub-cuts and reproduces with an independently trained recogniser.
- Target separation should be reported and stress-tested as a measurable design axis in identity-unlearning methods.
- Whether the target-separation effect transfers to other conditioning representations remains an open question.