PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Attacks and Identity Leakage in De-Identification Systems: An Empirical Study

Felix Rosberg, Cristofer Englund, Eren Erdal Aksoy, Fernando Alonso-Fernandez

arXiv:2609.27022v1cs.CV

TL;DR

The paper tests whether adversarial attacks transferred from surrogate identity encoders can make a realistic de-identification system leak identity. It evaluates encoder fine-tuning and low-pass filtering as mitigations, finding that transfer attacks cause leakage and the proposed mitigations alleviate it.

  • Problem

    The study asks whether attacks on identity encoders can cause identity leakage in de-identification, where identity is intended to be obscured while attributes are preserved.

    Source: §I · p. 1

  • Method

    The authors evaluate surrogate-generated attacks on FIVA and test adversarial-example fine-tuning and low-pass filtering as mitigations.

    Source: §I · p. 1

  • Results

    All tested surrogate models cause identity leakage, with ElasticFace, CosFace, and AdaFace strongest; FaceNet and ImageNet ResNet50 have minor effects.

    Source: A. Quantitative Results · p. 5

  • Takeaways & Limitations

    The proposed mitigations alleviate leakage, diminishing the practicality of attacks against the de-identification system.

    Source: §I · p. 1

    The paper leaves real-world patch-based attacks and robustness against an adversary adapting to the mitigation for future study.

    Source: B. Qualitative Results · p. 8

Abstract

from arXiv · show

In this paper, we investigate the impact of adversarial attacks on identity encoders within a realistic de-identification framework. Our experiments show that the transferability of attacks transfers from an external surrogate model to the system model (e.g., CosFace to ArcFace) allows the adversary to cause identity information to leak in a sufficiently sensitive face recognition system. We present experimental evidence and propose strategies to mitigate this vulnerability. Specifically, we show how fine-tuning on adversarial examples helps to mitigate this effect for distortion-based attacks (i.e., snow, fog, etc.), while a simple low-pass filter can attenuate the effect of adversarial noise without affecting the de-identified images. Our mitigation results in a de-identification system that preserves its functionality while being significantly more robust to adversarial noise.

I. INTRODUCTION

The paper asks whether adversarial attacks on identity encoders can make realistic de-identification systems leak identity, and tests black-box transfer attacks and mitigation strategies. Prior work covers de-identification vulnerabilities and practical attacks on facial-recognition and image-translation systems.

II. RELATED WORK

  • Prior work reports that target-oriented de-identification methods can be reversed in black-box attacks when enough victim-system data is collected.Related research also examines adversarial attacks on facial-recognition and image-translation systems, including face swapping.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • The study tests whether identity encoders used in de-identification can be exploited through adversarial transfer from surrogate models unavailable to the attacker’s victim system.Its experiments target FIVA, with ArcFace as the victim encoder, and compare six identity encoders plus an ImageNet-pretrained ResNet50.Source: §I · p. 1
  • Adversarial transfer can make ArcFace extract identity after de-identification when CosFace generates the attack, while FaceNet has less impact.The study defines identity leakage as a successful biometric face-recognition match and uses FIVA as a measurement tool for the de-identification effect.Source: §I · p. 1
  • The proposed mitigations are adversarial-example fine-tuning of the identity encoder and low-pass filtering to dampen high-frequency noise.The authors report that these measures alleviate identity leakage and diminish the attack’s practicality.Source: §I · p. 1

II. RELATED WORK

  • Wearable perturbations have been shown to make facial identity encoders misjudge identity, motivating study of those encoders as components of de-identification systems.Many target-oriented de-identification methods use facial-recognition models as priors for controlling synthetic identity.Source: §II · p. 2

III. METHOD

The method varies the gradient-generating encoder and distance objective, then applies generated adversarial examples to FIVA with three ArcFace variants.

A. Gradient Calculation Models

  • Seven gradient models span the victim ArcFace, four other cosine-space face encoders, FaceNet’s L2 space, and an ImageNet-pretrained ResNet50.This selection probes whether transfer depends on similarity between the surrogate and victim embedding spaces.Source: A. Gradient Calculation Models · p. 2

B. Attack Objectives

  • Cosine distance is the primary attack objective for compatibility with ArcFace, while FaceNet uses L2 loss and both objectives are tested across surrogates.The experiments assess whether matching the victim’s embedding metric affects attack transfer.Source: B. Attack Objectives · pp. 2–3

C. De-Identification Identity Encoders

D. Adversarial Attacks

The study adapts a diverse attack set to embedding-based identity encoders and evaluates leakage in de-identified faces, alongside the performance of fine-tuned encoders.

D. Adversarial Attacks

  • The attack suite adapts elastic transform, fog, Gabor, Frank-Wolfe, snow, JPEG, and PGD attacks to encoder-specific embedding losses.Attack strengths are selected qualitatively based on gradient loss and visual effect; learned attacks are used to fine-tune victim ArcFace.Source: D. Adversarial Attacks · p. 3

E. Evaluation

  • Identity retrieval on de-identified faces, rather than the framework’s attack-accuracy robustness metric, measures leakage across six identity encoders and FAR values 10^-3, 10^-4, and 10^-5.Experiments use FaceForensic++ and aggregate configurations across gradient models, evaluation models, attacks, thresholds, and loss functions.Source: E. Evaluation · p. 3
  • ArcFace fine-tuned with adversarial examples is evaluated for false rejection rate and accuracy at FARs 10^-3, 10^-4, and 10^-5 on FaceForensic++ [49] and LFW to assess performance reduction.Source: E. Evaluation · p. 3

F. Robustness Tuning

Robustness tuning distills the original ArcFace teacher into a student trained on clean and adversarial images, using separate student embedding layers for the two inputs.

F. Robustness Tuning

  • The student starts as a clone of pretrained ArcFace and receives clean and adversarial images through separate cloned output and batch-normalization layers.One branch matches clean-image embeddings for adversarial inputs; the other preserves information for clean images.Source: F. Robustness Tuning · p. 3
  • The tuning objective combines robustness, consistency, and union cosine-distance losses between teacher and student embeddings, weighted by separate coefficients.The robustness term compares clean teacher embeddings with adversarial student embeddings; consistency compares clean teacher and student embeddings, while union compares the student’s clean and adversarial embeddings.Source: F. Robustness Tuning · p. 4

G. Explainable Inspection

Experiments find that transferable attacks can cause identity leakage in FIVA, while decoded embeddings provide a visual inspection of matched and failed retrieval cases.

G. Explainable Inspection

  • DeepFaceDecoder reconstructs faces from identity embeddings, enabling visual inspection of attack effects without decoding each attack with its generating encoder.ArcFace embeddings were used for the visualizations.Source: G. Explainable Inspection · p. 4

A. Quantitative Results

  • Table I reports chosen noise strengths and gradient-iteration counts for each attack, including learned-attack perturbation strength.Source: Table I · p. 4
  • Evaluation varied gradient-calculation and evaluation encoders across six face encoders, three FAR thresholds, and a no-attack baseline.The FAR thresholds were 10−3, 10−4, and 10−5; Table III covers results after tuning ArcFace on learned adversarial examples.Source: A. Quantitative Results · p. 4, Table II · p. 5, Table III · p. 5
  • All tested surrogate encoders caused identity leakage; ElasticFace, CosFace, and AdaFace had the strongest effects, while access to victim ArcFace was more effective.The results support transferability, especially for surrogates with a similar embedding metric space.Source: A. Quantitative Results · p. 5, Table II · p. 5
  • Matching the victim’s distance metric was slightly more effective, although FaceNet and ImageNet-pretrained ResNet50 still produced minor leakage.Cosine or L2 distance had similar effects overall.Source: A. Quantitative Results · p. 5

1) Robustness Tuning:

Robustness tuning combines adversarial-example fine-tuning with Gaussian low-pass filtering. Filtering sharply reduces measured identity leakage, while fine-tuning alleviates it to an extent.

1) Robustness Tuning:

  • Combined fine-tuning examples apply a Learned perturbation, then randomly apply remaining attacks up to three times; the model is fine-tuned on VGGFace2.The combined setup is the model evaluated in Table IV.Source: 1) Robustness Tuning · pp. 6–7, Table IV · p. 6
  • Fine-tuning ArcFace on learned adversarial examples alleviated leakage, and tuning on learned examples alone worked better than combining them with other attacks.The combined training set included learned perturbations and distortion-based attacks.Source: Table IV · p. 6, 1) Robustness Tuning · p. 7
  • The Gaussian filter is applied before identity extraction to dampen high-frequency adversarial noise while keeping the image interpretable to the encoder.The de-identification generator conditions on the embedding rather than the blurred image, so the filter does not affect its input image.Source: 1) Robustness Tuning · p. 7
  • TAR at FAR 10−5 identity leakage drops to almost 0 with Gaussian low-pass filtering.Figure 5 aggregates retrieval for MagFace, ElasticFace, and CosFace attacks as blur-kernel σ increases.Source: Figure 5 · p. 6, 1) Robustness Tuning · p. 7

2) Performance After tuning:

Robustness tuning has minimal effect on ArcFace performance, with small losses at stricter FAR values and a gain at FAR 10−3.

2) Performance After tuning:

  • ArcFace tuning minimally affects performance: it loses some performance at FAR 10−4 and 10−5 but gains some at FAR 10−3.The authors consider the effect negligible in the context of face recognition and their goal of more robust de-identification.Source: 2) Performance After tuning · p. 7

B. Qualitative Results

The experiments show that attacks can leak identity through de-identification, while proposed filtering and tuning mitigate the vulnerability. Qualitative examples show that most attacks retain convincing synthetic faces, with JPEG L-∞ degrading realism.

B. Qualitative Results

  • Decoded ArcFace embeddings make leakage more visible in examples where CosFace-generated Frank Wolfe, PGD L2, or PGD L-∞ attacks match.The examples also show low-pass filtering preventing leakage; ArcFace embeddings are used for visualization regardless of which encoder generated the attack.Source: B. Qualitative Results · p. 7
  • Table V reports tuned-model EER, FRR, and accuracy for LFW's 10-fold test protocol and FaceForensics++.The caption specifies FRR and accuracy at FAR 10−3, 10−4, and 10−5.Source: Table V · p. 8
  • Most attacks still produce convincing de-identified faces, but JPEG L-∞ significantly degrades synthetic-face realism.The authors note that PGD L-∞ does not cause significant identity leakage.Source: B. Qualitative Results · p. 8
  • The authors identify real-world patch attacks and testing against attackers who adapt to the mitigation as future directions.Source: B. Qualitative Results · p. 8

V. CONCLUSION

  • 2.45% of trials matched using AdaFace with ElasticFace at FAR 10−5, and 2.98% using ImageNet ResNet50 with MagFace at FAR 10−4.The conclusion also reports that Gaussian low-pass filtering can almost completely nullify leakage at FAR 10−5 or lower, while adversarial-example distillation addresses remaining distortion-based attacks.Source: §V · p. 8

APPENDIX A NON-AGGREGATED RESULTS

The appendix provides attack-specific, surrogate-specific results without the aggregation used in the main manuscript, including results for tuned identity encoders.

APPENDIX A NON-AGGREGATED RESULTS

  • The appendix reports identity-retrieval success rates by attack and surrogate model rather than only the main manuscript's aggregated results.The listed detailed tables span attacks using multiple surrogate models and include tuned identity encoders.Source: Appendix A · p. 9
  • The table series covers FaceForensics++ attacks with untuned encoders, learned-adversarial-example tuning, and tuning that also includes distortion attacks.Individual captions identify different surrogate models; the supplied passages do not report table cell values.Source: Appendix A · p. 9, Table X · p. 11, Table XIII · p. 11

DISCLOSURE STATEMENT

The authors disclose a co-authorship relationship with a guest editor on a separate manuscript and request handling by editors without professional or personal connections to the authors.

DISCLOSURE STATEMENT

  • The disclosed co-authorship with the guest editor concerns a separate manuscript submitted elsewhere; the authors request unconnected editors handle this manuscript.Source: DISCLOSURE STATEMENT · p. 16
Loading 2609.27022v1…