Breaking High Confidence: Practical Face Impersonation under High-Security Thresholds

Changjin Kim; Seunghun Paik; Dongsoo Kim; Jae Hong Seo

Source-linked AI summary

Breaking High Confidence: Practical Face Impersonation under High-Security Thresholds

Changjin Kim, Seunghun Paik, Dongsoo Kim, Jae Hong Seo

arXiv:2608.20884v1Aug 21, 2026cs.CV

TL;DR

High-security FRS deployments remain insufficiently studied, especially under practical score-query limits. The paper develops a black-box score-based impersonation attack by analyzing and reducing errors across its pipeline, achieving high ASR under stringent thresholds. The authors also identify digital-evaluation and surrogate-target alignment boundaries for real-world applicability.

Problem
Prior FRS security analyses mainly considered medium-security thresholds, leaving high-security vulnerability under practical query budgets insufficiently explored.
Method
The paper proposes a black-box score-based attack using template-space geometry, mathematical error analysis, and inverse-model reconstruction.
Results
The attack achieves high ASR across evaluated systems, including 92.8% on LFW against AWS Rekognition at confidence threshold 99 using at most 100 queries.
Takeaways & Limitations
The findings demonstrate vulnerability in open-source and commercial FRSs even at low-FMR settings under black-box score-based attacks.
Takeaways & Limitations
The evaluation uses digital API submissions, while physical presentation may introduce deformation and sensor-level defenses may be employed.

Abstract

from arXiv · show

Face recognition systems (FRSs) are increasingly deployed in critical real-world services for authentication, such as banking applications and airport identity checks, necessitating stringent security configurations. Consequently, the security vulnerabilities of FRSs have garnered significant attention. While existing studies have extensively explored FRS security, prior analyses have primarily focused on medium-security threshold settings, which are not directly applicable to FRSs operating under high-security constraints. In this paper, we propose the first successful impersonation attack against FRSs under high-security threshold settings. Among various threat models, we focus on a practical and challenging scenario: score-based impersonation attacks under strict rate limits. To precisely evaluate the feasibility of such attacks, we provide a principled mathematical analysis characterizing the gaps in each stage of the attack pipeline. Our method significantly enhances impersonation capabilities in score-based attacks, even under elevated decision thresholds. On the LFW benchmark, with a budget of only 100 confidence score queries per identity, our attack achieves an impersonation success rate exceeding 92\% against Amazon Rekognition at a confidence score threshold of 99-recommended setting for law enforcement scenarios. We further observe consistently robust performance across multiple open-source FRSs evaluated at similarly stringent decision thresholds.

1 Introduction

The paper targets an underexplored security regime: practical impersonation against face recognition systems using high-security thresholds and limited queries. It proposes techniques that achieve high attack success rates across commercial and open-source systems.

FRS deployments increasingly support high-stakes authentication, including airport identity checks and financial onboarding.
Prior studies mainly analyze medium-security settings, leaving high-security regimes such as FMR = 10^-6 less explored.
Existing attacks either require 4,000–300,000 queries or experience sharply reduced ASR at high-security thresholds.
The paper conducts the first high-security analysis of practical black-box score-based impersonation attacks.
The proposed techniques use geometric properties of template space and are supported by theoretical and experimental analyses.
92.8% ASR is achieved on LFW against AWS Rekognition at confidence threshold 99 using at most 100 queries.

2 Related Work

Prior score-based impersonation attacks use optimization, generative models, or template-space geometry, but query-intensive methods conflict with rate limits and low-query methods fail at stringent thresholds.

Gradient-free attacks include hill-climbing, genetic algorithms, and gradient estimation.
Generative latent-space attacks can recover high-fidelity faces but require 4,000–300,000 score queries.
Adaptive query trajectories may leave structured footprints that enable query filtering or request rejection.
A prior geometry-based attack achieved non-trivial ASR with 100 non-adaptive queries on commercial APIs at moderate thresholds.
That prior attack’s ASR dropped to 0 at high-stakes thresholds.
Template inversion attacks commonly assume access to target templates and use inverse models to reconstruct faces.

3 Backgrounds and Problem Formulation

FRSs map faces to fixed-length templates and compare template similarity against a decision threshold. The threat model considers a black-box adversary that submits its own faces and receives confidence scores.

3.1 Face Recognition: FRSs use feature extractors to map input faces into fixed-length face templates.
3.1 Face Recognition: Templates from the same identity are expected to be close, while templates from different identities are expected to be distant.
3.1 Face Recognition: The system recognizes two faces as the same person when their template similarity exceeds a decision threshold.
3.1 Face Recognition: Commercial APIs allow users to enroll a face and query another face to obtain a confidence score against the enrolled face.
3.2 Score-based Face Impersonation Attack: The adversary queries its own face to the target FRS and receives a confidence score for the enrolled target face.
3.2 Score-based Face Impersonation Attack: The black-box adversary lacks the target FRS’s parameters, architecture, training data, and decision threshold.

4 Revisiting Prior Low-Query Score-based Attacks

The revisited attack interprets score queries geometrically, estimates the enrolled template through a linear system, and reconstructs a face through inversion. Experiments show that approximation errors substantially reduce performance at high-security thresholds.

4.1 High-Level Idea of the Attack: Each score query reveals the target template’s relative position in a surrogate embedding space.
4.1 High-Level Idea of the Attack: The score model is s_i = ⟨x_i, y⟩ + ϵ_i, where ϵ_i captures surrogate-target embedding mismatch.
4.1 High-Level Idea of the Attack: After Q queries, the adversary forms Ay = s − ϵ and estimates y with the pseudoinverse while ignoring unknown errors.
4.1 High-Level Idea of the Attack: The estimated template is inverted into a candidate facial image using inverse models for the surrogate system.
Key Techniques to Instantiate the Attack: The orthogonal face set reduces error effects, while numerical methods estimate cosine similarity from commercial confidence scores.
4.2 Investigating Errors from Approximations: The pipeline contains errors from score interpretation, pseudoinverse solving, and template inversion.
4.2 Investigating Errors from Approximations: The experiment sequence isolates the error sources, with PCA-subspace selection minimizing the remaining projection gap in Exp. 4.
4.2 Investigating Errors from Approximations: 97%+ of samples exceed the FMR = 10^-6 threshold after sequentially removing the errors, compared with approximately 0.2% in the baseline.

5 Reducing Errors to Surpass High Thresholds

The attack reduces three pipeline errors using a correction matrix, PCA-based OFS, and SPNet, with techniques selected according to black-box or white-box access. These methods improve projection and inversion quality without additional queries.

5.1 (E1): Correction Matrix to Tame Metric Distortion: The correction matrix reduces metric-distortion error E1 by replacing A†s with AT S−1s.S contains cosine similarities among OFS elements, and the correction matrix is interpreted through metric distortion.
5.1 (E1): Correction Matrix to Tame Metric Distortion: The corrected estimate projects onto the row space of A under the metric defined by Σ, improving the target-embedding estimate.The interpretation depends on approximating target-model metric distortion with a positive definite matrix Σ.
5.2 (E2): PCA-Based OFS: PCA-based OFS reduces subspace-projection error E2 by selecting the optimal principal-component subspace rather than merely enforcing orthogonality.With a sufficiently large dataset, the adversary sets A = W using faces corresponding to principal components.
5.3 (E3): Improved Inverse Model: SPNet reduces inversion error E3 through an inverse-model design intended to improve reconstructed-face sharpness and inversion precision.The reported modifications remove the final tanh, replace ReLU with GELU, and apply pixel-wise normalization in the deconvolution block.
Training Losses: The final SPNet training loss combines pixel loss and identity loss, including the adversary’s surrogate model when computing identity loss.The loss is LTrain = λ1LPixel + λ2LID.
Experimental Analysis: Black-box attacks enable the correction matrix, PCA-based OFS, and SPNet, whereas white-box attacks use only PCA-based OFS and SPNet.The distinction reflects that error E1 is not engaged in the white-box setting.

6 Experimental Analysis

The experiments evaluate the proposed attack on open-source and commercial face recognition systems using standard benchmark datasets and stringent thresholds. AWS CompareFace is tested at confidence thresholds of 80, 90, and 99, while open-source systems use estimated FMR = 10−6 thresholds.

Target Systems: The evaluation covers various open-source and commercial face recognition systems.The study explicitly conducts extensive experimental analyses across both system types.
Target Systems: Table 2 summarizes the target systems, with Thres. denoting each open-source system’s estimated threshold at FMR = 10−6.The table organizes the system specifications and corresponding threshold information.
Target Systems: The target systems include ViT-KPRPE, TopoFR, SphereFace-R, and AWS CompareFace, with ArcFace used as the adversary’s surrogate model.The open-source systems are denoted F1, F2, and F3; AWS CompareFace is the commercial target.
Evaluation Protocol: The experiments measure attack success rate as the number of reconstructed faces authenticated as the target face.Open-source thresholds are set at FMR = 10−6, while AWS confidence thresholds are 80, 90, and 99.
Evaluation Protocol: The evaluation uses LFW, CFP-FP, and AgeDB, enrolling the first face from same-identity pairs and measuring authentication success.These datasets are described as widely used benchmarks for face-recognition accuracy.

Threshold Estimation for FMR = 10−6

The evaluation estimates stringent FMR = 10^-6 thresholds and tests the attack across open-source and commercial face recognition systems under limited-query conditions. Ablations show that the proposed techniques substantially improve attack success, including at high confidence thresholds.

Threshold estimation: FMR = 10^-6 thresholds are estimated using large-scale face datasets because the benchmark datasets lack enough false pairs for direct measurement.The estimate is intended to align thresholds with the evaluation datasets rather than the IJB-C protocol.
Ablations: 0.60% is the baseline ASR at threshold 99, increasing to up to 63.50% when the correction matrix and PCA-based OFS techniques are enabled.The comparison is reported for the ablation against FA.
Attack results: 92.80% ASR is achieved after replacing NbNet with SPNet at AWS CompareFace’s high-stakes threshold of 99.The reproduced baseline results may differ from the originally reported results because of API/model version differences.
Ablations: 50 queries suffice for non-trivial ASR across the tested threshold regimes, while ASR at threshold 99 rises sharply after approximately 50 queries.At thresholds 80 and 90, ASR approaches 100% with 60 queries.
Ablations: The correction matrix, PCA-based OFS, and SPNet each independently improve ASR under strict thresholds.The improvements hold with both the baseline NbNet and the proposed SPNet.

7 Discussion

The discussion considers defenses and deployment boundaries, including model-level metric distortion, inverse-model resolution, digital-only evaluation, and surrogate-target misalignment. These factors delimit how directly the reported attack results transfer to other settings.

Potential mitigation: Stronger metric distortion can substantially weaken the attack, with one defense producing 0.13% ASR at FMR = 10^-6.The discussion contrasts this with a 41.33% ASR for another defense under the same setting.
Potential mitigation: The proposed inverse model produces 128×128 images, but combining the attack with Arc2Face retains 53.92% ASR at threshold 99.The paper states that improving high-resolution inverse models could further enhance ASR.
Physical attacks: The commercial-API evaluation uses direct digital submission, whereas physical presentation may introduce deformation and presentation-attack detection defenses.The paper leaves physical realization as a future direction involving possible combination with spoofing attacks.
Open questions: Surrogate-target misalignment significantly affects attack efficiency, but its systematic characterization across architectures, datasets, and losses remains unexplored.The paper identifies this characterization as important future work.

8 Conclusion

The paper finds that black-box score-based attacks can expose vulnerabilities in open-source and commercial face recognition systems under low-FMR settings with practical query budgets. It attributes this capability to embedding-space geometry and a high-precision inverse model, while noting that experiments were conducted in controlled settings.

Conclusion: The method combines a geometric interpretation of score queries over embedding space with an inverse model designed for high inversion precision.The conclusion presents these components as enabling the reported attack.
Scope: The experiments were conducted in a strictly controlled setting and did not target real-world FRS deployments.This scope boundary qualifies the empirical conclusions.

Supplementary Material of Breaking High Confidence: Practical Face Impersonation under High-Security Thresholds

The supplementary material details a two-phase black-box score-based impersonation pipeline, including offline preprocessing and online target recovery. It formalizes confidence-score processing, surrogate embedding construction, correction, and attack-image synthesis.

Pipeline overview: The supplement provides the complete AWS CompareFace pipeline, omitted algorithms, and analyses of confidence-score conversion and query cost.The attack is organized into offline preprocessing and online impersonation phases.
Formalization: The formal FRS model maps feature-space cosine similarity to a confidence score and permits direct queries against an enrolled target face.The adversary may also query other image pairs when constructing attack components.
Offline preprocessing: Offline preprocessing constructs a PCA-based orthogonal face set and correction matrix from pairwise score queries independent of the target identity.The correction matrix is obtained by applying an estimated inverse confidence-to-cosine mapping to the score matrix.
Online impersonation: Online recovery queries target scores for orthogonal-face components, computes a corrected candidate template, and inverts it into an attack image.The candidate template is formed from the estimated cosine-similarity vector, correction matrix, and orthogonal-face feature matrix.
Score inversion: AWS confidence scores are normalized to [0, 1] before cosine-similarity estimation using an adopted inverse mapping and specified parameters.The reported parameters are (L, d0, k, b) = (0.99, 0.75, −25, 0.00004).

A.3 Query Cost

The supplementary analysis quantifies query costs and examines the mathematical and architectural components underlying the attack. It also reports dataset, threshold, and open-source evaluation details, including performance caveats and additional ASR results.

Query cost: k(k−1)/2 offline pairwise queries and k online target queries are required when the orthogonal-face set has size k.Because correction-matrix construction is offline, the online impersonation phase requires only k queries.
Query cost: Using AWS search_faces_by_image, k = 100 calls can construct the correction matrix while each call returns up to 4,096 confidence scores at a $0.001 fee.This provides a practical way to reduce AWS API usage costs.
Mathematical analysis: The correction-matrix formulation recovers a weighted projection solution, while the white-box case reduces to the earlier attack without correction when Σ = I.The proposition characterizes the reconstructed vector through S−1s and the row space of A.
SPNet: The supplementary material specifies SPNet as a Style Mapping Network plus five Block Layers using AdaIN-based style modulation and progressive feature upsampling.SPNet stands for Self-Perceptual Network and uses the network itself alongside an external pretrained network for perceptual loss.
Evaluation: Thresholds and TMRs are reported across datasets and FMR levels, but benchmark-derived thresholds may be conservative and inflate ASR estimates.This caveat applies when thresholds are measured on datasets following the same distribution as LFW, CFP-FP, and AgeDB.
Evaluation: 32.06% to 74.43% ASR improvement is reported against open-source FRSs, although the correction matrix can degrade performance in those systems.The authors attribute this difference to the correction matrix not reducing metric distortion for these open-source FRSs.

E.3 Additional Ablation Studies on Open-Source FRSs

Additional ablations show that the attack remains effective across open-source FRSs and stringent FMR levels with moderate query counts. The PCA-based OFS and SPNet provide the strongest reported improvements, while competing attacks perform poorly under the same budget.

Query sweep: 50–60 queries are sufficient for achieving non-trivial ASRs across open-source attack scenarios and FMR levels.The query sweep covers FMR = 10−4, 10−5, and 10−6.
Ablation study: The PCA-based OFS and SPNet sharply improve ASR as techniques are added, whereas the correction matrix has a marginal effect across open-source settings.The ablation progressively adds techniques for reducing errors E1–E3.
Attack visualization: All recovered faces surpass the target authentication threshold with SPNet, despite barely resembling the targets visually.The attack enforces embedding-space closeness rather than human-perceived image-space similarity.
Baseline comparison: 1.22 ± 0.02% ASR is achieved by one baseline at FMR = 10−4 under a 100-query budget, while another records 0.49 ± 0.02%.A third baseline remains clearly below 10% even with around 3,000 queries at its best-accuracy threshold.
Baseline comparison: The compared attacks are not competitive with the proposed attack under a 100-query budget at FMR = 10−6.The supplementary material notes that public implementations are available for two of the compared methods.