Steering the Flow: Inverting Face Recognition Models via Gradient-Guided Flow Matching

Ye Lu; Shen Wang; Zhaoyang Zhang; Yihan Yan; Li Liu; Runze Liu; Fanghui Sun

Source-linked AI summary

Steering the Flow: Inverting Face Recognition Models via Gradient-Guided Flow Matching

Ye Lu, Shen Wang, Zhaoyang Zhang, Yihan Yan, Li Liu, Runze Liu, Fanghui Sun

arXiv:2608.16791v1Aug 17, 2026cs.CVcs.AIcs.CRcs.MM

TL;DR

Face model inversion remains difficult because pixel and GAN-latent optimization can be unstable and produce artifacts. SFMI steers a Flow Matching prior with progressive target-specific gradients, achieving average attack accuracy above 90% with favorable perceptual similarity across diverse targets.

Problem
Pixel-space and GAN-latent inversion can suffer non-convex optimization, unstable gradients, visual artifacts, and poor semantic coherence when recovering high-fidelity target-identity images.
Method
SFMI uses a two-stage Flow Matching prior and progressive target-aware gradient guidance to steer a smooth face-generation flow toward identity-specific reconstructions.
Results
Average attack accuracy exceeds 90% with favorable perceptual similarity across diverse face-recognition targets under identity-disjoint evaluation.
Takeaways & Limitations
SFMI provides a deterministic trajectory-steering formulation that mitigates optimization instability while delivering competitive attack accuracy and visual fidelity under the evaluated white-box protocol.
Takeaways & Limitations
The method involves relatively many hyperparameters, making tuning difficult.

Abstract

from arXiv · show

Model Inversion Attacks (MIAs) aim to reconstruct representative training samples of target identities from face recognition models, exposing critical security vulnerabilities. Existing methods typically rely on indirect guidance or highly stochastic guidance, making it difficult to stably optimize generation trajectories toward target facial images. In this paper, we propose Steering Flow Model Inversion (SFMI), a novel two-stage white-box model inversion method that reformulates inversion as a trajectory-steering task. Specifically, Step I, Learning a Generic Flow Matching Prior, pre-trains a generic unconditional Flow Matching model to encode the manifold of human faces as a robust prior. Step II, Attacking with Progressive Guidance Scheduler (PGS), injects time-dependent target-specific gradients during sampling. By backpropagating through the target model to obtain gradients from intermediate generated states, PGS progressively injects adaptive guidance signals into the vector field. This process effectively steers the current generative flow from random noise toward the high-density regions of the target class. Under an identity-disjoint cross-evaluation setting using the CelebA dataset, SFMI achieves an ACC of 0.9248, an FID of 22.61, and an LPIPS of 0.3874 on the ArcFace target. Extensive experiments on multiple target models demonstrate that SFMI achieves competitive state-of-the-art performance in attack success and visual fidelity under the evaluated white-box protocol.

I. INTRODUCTION

Face recognition models can leak identity information through memorized training data, while existing inversion approaches struggle with unstable optimization and stochastic guidance. SFMI addresses these limitations by steering Flow Matching ODE trajectories with adaptive target-model gradients and achieves competitive identity-disjoint performance across diverse targets.

Motivation: Face recognition DNNs can unintentionally memorize training data, creating a privacy threat through identity leakage.Model inversion attacks reconstruct representative facial images of target identities from information encoded in trained models.
Limitations of Existing Methods: High-dimensional, non-convex inversion is difficult because GAN priors yield non-smooth latent-to-image mappings and unstable gradient backpropagation.Diffusion-based attacks further face challenges from stochastic noise injected during SDE sampling.
Proposed Approach: SFMI reformulates white-box model inversion as ODE trajectory steering using Flow Matching as a generative prior.Flow Matching provides relatively smooth and stable velocity fields compared with diffusion models’ stochastic differential equations.
Evaluation: Above 90% average attack accuracy with favorable perceptual similarity was achieved under identity-disjoint evaluation across ResNet, ArcFace, CosFace, ViT, and MobileFaceNet targets.The paper reports strong and competitive performance relative to state-of-the-art optimization-based and generative model inversion methods, particularly in attack accuracy and visual fidelity.
Proposed Approach: PGS dynamically modulates the ODE vector field with adaptive gradients from the target model to provide fine-grained identity-recovery control.The scheduler integrates target-model feedback into the ODE solver during inversion.

II. RELATED WORK · A. Deep Face Recognition · B. Model Inversion Attacks

The related work traces deep face recognition’s shift toward identity-discriminative deep embeddings and model inversion attacks’ evolution from direct pixel optimization to generative priors. It identifies instability in existing generative attacks and motivates SFMI’s deterministic Flow Matching trajectory steering with progressive target feedback.

A. Deep Face Recognition: CNNs, ResNets, and ViTs progressively advanced visual representation learning through hierarchical features, stable deep training, and global self-attention.
A. Deep Face Recognition: Face recognition shifted from handcrafted features to end-to-end identity-discriminative embeddings, with objectives progressing from metric learning to margin-based softmax losses.
B. Model Inversion Attacks: MIAs threaten privacy by reconstructing sensitive training data or inferring private attributes, progressing from confidence-based attacks to explicit white-box optimization.
B. Model Inversion Attacks: Direct gradient descent in high-dimensional, non-convex pixel space often becomes trapped in poor local optima, producing semantically meaningless high-frequency noise.
B. Model Inversion Attacks: Related IoT privacy studies examine secure face identification, sparse low-rank face recognition, and feature inference attacks in vertical federated prediction.
B. Model Inversion Attacks: Generative priors addressed direct pixel optimization’s ill-posedness by constraining attacks to natural-image manifolds, beginning with GAN-based Generative Model Inversion.
B. Model Inversion Attacks: Existing defenses strengthen training or pre-training, but face recognition’s necessary identity-discriminative features expose a trade-off between recognition utility and privacy security.
B. Model Inversion Attacks: GAN-based latent optimization is unstable because latent-to-image mappings are nonlinear and nonsmooth, while diffusion approaches typically use SDE-based formulations.

III. METHOD … 1) Method Overview:

SFMI formulates model inversion as target-identity reconstruction under a white-box threat model and solves it with two coupled stages: a Flow Matching facial prior and progressively scheduled target-model guidance. The prior constrains sampling to realistic face structure, while intermediate-state gradients steer trajectories toward the target class.

A. Preliminaries: Given a target identity, the attack seeks an image capturing that identity’s private visual characteristics from a pre-trained face recognition model.The target model maps images to probabilities over K identities, and inversion reconstructs an image from information encoded in the trained model.
A. Preliminaries: The conventional inversion objective combines a target-class classification loss with a semantic-plausibility prior weighted by λ.Linv penalizes deviations from the target class, while R(x) constrains reconstructed images to remain semantically plausible.
B. Steering Flow Model Inversion (SFMI): SFMI replaces explicit prior-loss optimization with an implicitly learned facial prior and progressive target-model gradients within the learned face manifold.The Flow Matching model supplies the prior, while gradients steer sampling toward the target class.
A. Preliminaries: The white-box threat model gives the adversary the target architecture and trained parameters, enabling explicit gradient computation through backpropagation.The adversary also uses independently separated public data to learn transferable facial priors while private data remain inaccessible.
1) Method Overview:: Stage I trains a Flow Matching velocity field that transports Gaussian-prior samples to the human-face manifold, producing stable trajectories and a generic facial prior.The learned prior captures broad facial structure and appearance statistics, constraining later attack trajectories to realistic faces.
1) Method Overview:: Stage II starts from Gaussian noise and injects identity-related gradients computed from clean intermediate-state estimates into the sampling update direction.Progressive Guidance Scheduler modulates guidance strength over time through warm-up, sustain, and gradual decay to move states toward the target face class.

2) Learning a Generic Flow Matching Prior:

The generic prior uses Flow Matching to learn a smooth, controllable vector field that transports Gaussian noise into realistic faces. It is trained on public faces with optimal-transport paths, x-prediction, velocity matching, and Logit-Normal time sampling.

Learning a Generic Flow Matching Prior: Flow Matching learns a smooth, differentiable trajectory from noise to realistic face images, avoiding instability from irregular mappings or stochastic perturbations.The resulting vector field reliably transports Gaussian noise toward realistic face samples.
Learning a Generic Flow Matching Prior: The model maps p0 = N(0, I) to the empirical public-face distribution p1 supported by Dpub, which is disjoint from the target model’s private training set.The objective is to learn a time-dependent vector field transporting samples between these distributions.
Learning a Generic Flow Matching Prior: Optimal Transport defines straight-line trajectories between noise samples x0 and face samples x1, specifying intermediate states through linear interpolation.These paths provide the geometry used to derive the target velocity field.
Learning a Generic Flow Matching Prior: Instead of predicting velocity directly, Mϕ(xt, t) predicts the clean image x̂1, from which the induced velocity is computed using v = (x1 − xt)/(1 − t).This x-prediction parameterization follows prior practice and uses the OT path geometry.
Learning a Generic Flow Matching Prior: Training minimizes velocity-matching loss between the induced field vϕ and target velocity ut while sampling t from a Logit-Normal distribution to emphasize informative temporal regions.Each iteration samples clean faces from Dpub, Gaussian noise from N(0, I), and intermediate states along the OT path.

3) Progressive Gradient-Guided Attack:

SFMI treats white-box inversion as trajectory steering: it uses the flow-matching prior to estimate clean images, backpropagates target-identity gradients to noisy states, and injects normalized guidance into sampling. A four-phase progressive schedule and predictor-corrector integration regulate steering strength and refine the trajectory.

Trajectory Steering: SFMI formulates target-identity recovery as steering an ODE trajectory so the final sample minimizes identity mismatch with the target label.The learned flow-matching prior provides the generative trajectory, while the attack modifies its numerical integration.
Target-Specific Guidance: At each step, the prior maps noisy state x_t to clean estimate x̂_1, which the target classifier evaluates using an identity loss instantiated as PLGMI’s max-margin loss.Directly classifying noisy x_t would produce uninformative gradients because the target classifier is trained on clean images.
Target-Specific Guidance: End-to-end backpropagation through the frozen classifier and prior yields g(x_t), whose normalized signal is injected into the velocity field to steer sampling toward the target identity.Normalization preserves the adversarial direction while matching the gradient scale to the flow-matching velocity.
Progressive Guidance Schedule: The progressive guidance schedule γ(t) uses warm-up, sustain, decay, and relaxation phases instead of constant strength.Linear warm-up prevents abrupt shifts, cosine decay smooths transition, and zero guidance lets the prior refine the image on the learned face manifold.
Numerical Integration: A second-order Heun-inspired predictor-corrector scheme evaluates guidance at the current and provisional states to correct curvature induced by adversarial steering.The predictor estimates the next state, and the corrector refines the update using the gradient at the predicted position.

IV. EXPERIMENTS

The experiments evaluate SFMI from four complementary perspectives, covering the experimental protocol, baseline comparisons, and component-level ablations. The protocol includes disjoint dataset construction, target face-recognition models, and cross-evaluation metrics.

Experimental Overview: SFMI is evaluated through four complementary experimental perspectives.The section introduces protocol analysis, quantitative and qualitative baseline comparisons, and component-level ablations.
Experimental Protocol: The protocol specifies disjoint dataset construction, target face-recognition models, and cross-evaluation metrics.
Baseline Comparison: SFMI is compared with representative white-box model-inversion baselines using quantitative results and qualitative visualizations.
Ablation Study: Component-level ablations are conducted to isolate the contributions of the prior.

A. Experimental Setup

The experiments enforce identity-disjoint public and private data, evaluate SFMI across six face-recognition targets, and measure attack success, realism, and perceptual consistency. The flow-matching prior uses a Transformer architecture trained for 500k steps, followed by default 50-step attacks.

Datasets: Identity-disjoint evaluation separates the adversary’s public prior data from identities used to train target models.CelebA-priv contains 1000 identities and 30,000 images, while CelebA-pub provides a non-overlapping 30,000-image auxiliary split without identity annotations.
Datasets: The study uses CelebA-pub as the default prior dataset and FFHQ-pub for cross-distribution transferability, with both containing 30,000 identity-disjoint public images.The two public datasets use the same public-data sc…
Target Models: SFMI is evaluated against six face-recognition targets spanning different architectures, objectives, and input resolutions.Face.evoLVe and IR-152 use 64 × 64 resolution, whereas CosFace and ArcFace use 112 × 112 resolution; MobileFaceNet is a lightweight architecture for mobile, IoT, and edge devices.
Evaluation Metrics: Performance is assessed with ACC, FID, and LPIPS, respectively measuring identity recovery success, global visual realism, and fine-grained perceptual consistency.ACC measures semantic identity recovery under the cross-evaluation protocol.
Implementation Details: The flow-matching prior uses a ViT/DiT-like Transformer with 16 × 16 patches and 16 attention heads, trained for 500k optimization steps.Training uses AdamW, a per-batch learning rate of 2 × 10−5, linear warmup followed by cosine decay, and logit(t) ∼N(−0.8, 0.82) sampling.

B. Comparison with State-of-the-Art Methods

SFMI is evaluated against five representative white-box model inversion baselines across six target face recognition models. It achieves the strongest identity recovery and perceptual consistency overall, while visual comparisons show clearer facial structure and fewer reconstruction artifacts.

B. Comparison with State-of-the-Art Methods: SFMI is compared with GMI, PPA, PLGMI, IFGMI, and FGMIA on six target models: Face.evoLVe, IR-152, CosFace, ArcFace, MobileFaceNet, and ViT.The evaluation follows prior settings for standard white-box model inversion assessment.
B. Comparison with State-of-the-Art Methods: SFMI uses 50-step second-order Heun integration, corresponding to 100 model evaluations per attack.The full process is repeated three times, with eight images sampled per target class to compute reported means and standard deviations.
B. Comparison with State-of-the-Art Methods: SFMI achieves the highest ACC on every target model and the lowest LPIPS across all targets.These results indicate strong identity recovery and superior perceptual consistency under the evaluated metrics.
B. Comparison with State-of-the-Art Methods: SFMI obtains the best FID on Face.evoLVe and remains highly competitive on the other target models.The comparison uses ACC, FID, and LPIPS as complementary evaluation metrics.
B. Comparison with State-of-the-Art Methods: Visual comparisons show that SFMI preserves facial geometry and identity-related local traits more clearly than baseline reconstructions.SFMI also reduces blurry textures and unstable high-frequency noise, while several prior methods are misled during optimization.

C. Ablation Studies

Ablations show that SFMI depends on its Flow Matching prior, progressive guidance, margin-based loss, and resolution-matched setup, while remaining effective under distribution shifts and output perturbation. BiDO reduces attack accuracy but also lowers target-model recognition accuracy.

Prior and sampler: Replacing the Flow Matching prior with a DDPM prior and DDIM sampling substantially degrades semantic recovery because the guided trajectory is less robust.SFMI directly predicts the original image, whereas DDIM estimates denoised images through a DDPM noise-prediction model; Flow Matching provides a smoother steering backbone.
Auxiliary prior distribution: Using an identity-disjoint FFHQ prior instead of CelebA causes only a minor performance degradation while preserving overall attack effectiveness.Both auxiliary subsets contain 30,000 images, controlling data scale while changing the public-prior distribution.
Guidance schedule: Constant guidance catastrophically collapses optimization by pushing noisy initial states off the natural face manifold, producing sharply lower ACC and severe visual artifacts.The progressive schedule preserves generation structure while gradually injecting identity-specific gradients across integration steps.
Optimization loss: Replacing the margin-based MM loss with cross-entropy causes a slight but consistent performance drop, supporting MM for identity-level supervision.The observation is consistent with findings reported in PLGMI.
Input resolution: At 224 resolution, SFMI metrics moderately degrade but the attack remains effective, showing that it is not restricted to low-resolution targets.Higher-resolution operation requires a resolution-matched Flow Matching prior and the same detected, aligned, and cropped preprocessing pipeline.
Defenses: BiDO lowers attack ACC by 6.37% on ArcFace and 7.63% on MobileFaceNet, while reducing target-model recognition accuracy by 6.86% and 4.45%, respectively.The defense is evaluated by training defended target models on the same private-data split before applying the unchanged SFMI protocol.

D. Detailed Parameter Analysis of Progressive Guidance

The analysis shows that progressive guidance is essential for effective and stable trajectory steering, with both spatial normalization and temporal scheduling contributing to performance. Standard PGS balances target-identity steering with stable evolution along the learned face manifold, while excessive late-stage forcing harms validation transfer.

Guidance mechanism analysis: Seven guidance variants show that spatial direction alignment and temporal warm-up, sustain, and annealing phases are functionally necessary.The analysis systematically evaluates these guidance components to explain trajectory-steering dynamics.
Guidance mechanism analysis: A constant coefficient of 1.0 and an unnormalized linear schedule from 0.0 to 1.0 are too weak to redirect generation effectively.Their target-model ACC remains relatively low or fluctuates unstably, indicating limited optimization effectiveness.
Schedule and loss interaction: Excessive late-stage forcing degrades validation-model performance by accumulating refinement artifacts and widening the target–validation ACC gap.The validation-model curve drops during later steps, suggesting overfitting to short-term target gradients.
Standard PGS configuration: The standard PGS uses M = 0.3, t0 = 0.1, t1 = 0.3, t2 = 0.7, and Vmax = 1.0.It progressively introduces identity guidance, attenuates it before final refinement, and normalizes the identity gradient relative to the current FM velocity.

V. ETHICAL CONSIDERATIONS

The work audits privacy leakage in face recognition systems through white-box model inversion attacks, using public datasets for academic evaluation while avoiding redistribution of original data.

The study evaluates privacy leakage in face recognition systems under a white-box, strong-access threat model.It frames model inversion attacks as an auditing method.
Experiments use publicly available CelebA and FFHQ datasets solely for academic evaluation under their dataset terms.The passage states that the original data are not redistributed.
The paper includes qualitative reconstructions to demonstrate potential privacy leakage risks.

VI. CONCLUSION

SFMI reformulates white-box model inversion as deterministic trajectory steering using Flow Matching and progressive gradient guidance. The method offers competitive performance for privacy-leakage auditing but requires substantial access, tuning, and suitable auxiliary data.

Method: SFMI reformulates white-box model inversion attacks as a deterministic trajectory-steering problem using Flow Matching’s continuous ODE formulation.This formulation targets instability in highly non-convex GAN latent spaces and stochastic disruption in standard diffusion models.
Method: Progressive gradient guidance dynamically modulates adversarial force across integration steps to align guidance temporally and spatially with the generative flow.The conclusion reports strong and competitive performance relative to state-of-the-art methods.
Limitations: SFMI has many hyperparameters, requires full target-model access including gradients, and depends on relevant public auxiliary face data.Tuning can be difficult, and performance may degrade under larger domain shifts between public and private data.
Future Work: Future work will address weaker-access and black-box inversion, robust attacks, stronger defenses, privacy-preserving recognition, and alignment with human identity judgments.The stated goals include reducing reliance on many hyperparameters and developing methods better aligned with human identity judgments.