Source-linked AI summary
GMoE-FTI: A Universal Facial Template Inversion Method Adopting Genuine Mixture-of-Experts Mechanism
Zixuan Shen, Zhihua Xia, Kaikai Gan, Peipeng Yu
TL;DR
Leaked facial templates can be inverted into recognizable faces, while existing FTI methods are trained for a single template type and may fail to preserve identity on unseen types. GMoE-FTI surveys five template distributions and uses specialized experts with matching and weighted fusion. Across heterogeneous and unseen template types, it reports an average 40% TAR improvement and average 2% reductions in FAPD and FAPC.
Problem
Existing FTI methods fit a single template type and can produce identity-inconsistent faces from unseen-extractor templates, while leaked templates can expose recognizable faces.
Source: §I · p. 1Method
GMoE-FTI surveys five template-distribution categories and uses independent specialized expert branches with a matching module to weight and fuse their outputs.
Source: §I · pp. 1–2Results
GMoE-FTI reports an average 40% TAR gain and average 2% reductions in both FAPD and FAPC against four mainstream SOTA FTI approaches.
Source: §I · p. 2Takeaways & Limitations
The authors conclude that GMoE-FTI supports identity-consistent reconstruction across heterogeneous facial templates and outperforms SOTA models on TAR, FAPD, and FAPC.
Source: §VII · p. 10
Abstract
from arXiv · showhide
Face recognition systems have been ubiquitously deployed across daily scenarios. To protect user privacy, mainstream systems store facial templates rather than raw face images for identity verification. However, facial template inversion (FTI) technologies can reconstruct faces from templates, which endangers biometric privacy. Existing FTI methods only fit single type of templates, failing to reconstruct identity-consistent faces from unseen types. To tackle this limitation, we survey mainstream template extractors which can be classified into five categories according the template distribution: unit-sphere, angle-linear, strict unit-sphere, cone-shaped, anuniform hypersphere. Inspired by this observation, this paper proposes GMoE-FTI, a universal FTI framework powered by a genuine MoE mechanism that differs from load-balanced multi-branch alternatives. It follows a three-stage training strategy: (1) independently train five mapping experts (MEs) and a matching module (MM); (2) conduct collaborative inference by weighting each ME's output with MM's output, so as to generate unified latent codes compatible with pre-trained StyleGAN3; (3) perform end-to-end joint fine-tuning over all modules to identity consistency and boost cross-expert coordination. Extensive evaluations on the LFW benchmark validate the superiority of GMoE-FTI against SOTA methods. Evaluated on seven prevailing unseen template types including AdaFace, CosFace, CurricularFace, ElasticFace, SphereFace2, SphereFace-R, and TransFace, GMoE-FTI achieves an average 40% TAR improvement, with average FAPD and FAPC each reduced by 2%, verifying its universal generalization across heterogeneous templates.
I. INTRODUCTION
Facial template inversion can reconstruct recognizable faces from leaked templates, but existing methods typically produce identity-consistent results only for the extractor seen during training. GMoE-FTI uses a five-category template taxonomy and genuine mixture-of-experts framework to target inversion across heterogeneous templates.
I. INTRODUCTION
- Existing FTI models trained on one extractor produce identity-inconsistent reconstructions from unseen-extractor templates, although the source extractor may be unknown and training a separate model for every extractor is infeasible.Facial templates can reveal recognizable faces, creating biometric privacy risks when templates leak.Source: §I · p. 1
- The authors classify mainstream templates into five distributions—unit-sphere, angle-linear, strict unit-sphere, cone-shaped, and uniform hypersphere—and train a dedicated expert for each.The survey covers mainstream extractors proposed from 2015 to the present.Source: §I · p. 1
- GMoE uses five independent expert branches specialized for distinct template distributions and a matching module to weight and fuse their outputs into a unified reconstruction.The authors distinguish this task-specific approach from multi-branch MoE designs used mainly for load balancing and channel splitting.Source: §I · p. 2
- GMoE-FTI reports an average 40% TAR gain and 2% reductions in both FAPD and FAPC against four mainstream SOTA FTI approaches.The paper presents the framework as compatible with arbitrary facial templates of varying dimensions from unknown extractors.Source: §I · p. 2
A. Facial Template Inversion
- Prior FTI methods include custom-built template-to-image networks and pre-trained-generator approaches that map templates to latent codes, while both face practical constraints when the target extractor is unknown.The related-work discussion describes methods using StyleGAN, GNeRF, StyleSwin, and Stable Diffusion backbones or representations.Source: A. Facial Template Inversion · p. 2
B. Existing Mixture-of-Experts Mechanism
Conventional MoE designs use multiple branches and gating, but the paper identifies their lack of specialized expertise and template-aware matching as limitations for universal FTI. The paper also defines the facial template extractor and inverter mappings.
B. Existing Mixture-of-Experts Mechanism
- The paper describes MoE as a gated system of multiple expert branches that dynamically activates partial subnetworks to balance computational overhead and model-fitting capacity.The cited overview presents MoE as a modular design used in language models and computer-vision generation pipelines.Source: B. Existing Mixture-of-Experts Mechanism · p. 2
- Conventional MoE branches are co-trained from random initialization, while gating uses top-k selection and load balancing without distribution-aware matching or collaborative inference.The authors identify these structural limitations as making existing MoE architectures unsuitable for universal FTI.Source: B. Existing Mixture-of-Experts Mechanism · p. 2
A. Model Definition
- In the paper’s model definition, a facial template extractor maps a face image to a d-dimensional template, and the universal inverter maps templates back to face images.The inverter is implemented as a generative network gθ with learnable parameters θ.Source: A. Model Definition · pp. 2–3
B. Adversary’ Capabilities
The assumed adversary can query black-box extractors to build paired training data, and the inverter is optimized for identity consistency on both seen and unseen templates. The paper’s extractor survey groups template spaces into five geometric distributions.
B. Adversary’ Capabilities
- The adversary is assumed to query one black-box extractor in each category with arbitrary face images and use the resulting image-template pairs to train the universal inverter.This access model is described as prevalent in prior FTI work and aligned with real-world scenarios.Source: B. Adversary’ Capabilities · p. 3
C. Optimization Objectives
- The optimization objectives require reconstructions from seen and unseen extractors to resemble the source faces and yield templates similar to their input templates.For seen templates, the stated goal includes maintaining identity consistency within the extractor category.Source: C. Optimization Objectives · p. 3
IV. AN INVESTIGATION INTO FACIAL TEMPLATE EXTRACTION
- Softmax provides the baseline for facial representation learning, while later loss designs regularize features or weights into distinct template-distribution categories.The paper describes five categories: unit-sphere, angle-linear, strict unit-sphere, cone-shaped, and uniform hypersphere.Source: §IV · p. 3
- Dual feature-and-weight normalization defines strict unit-sphere embeddings; later margin-based methods and Transformer extractors retain this rigid spherical constraint.The passage names NormFace, CosFace, ArcFace, CurricularFace, ElasticFace, SphereFace2, SphereFace-R, X2-Softmax, JAMsFace, TransFace, and LVFace in this category.Source: §IV · pp. 3–4
- Cone-shaped methods link template norms to facial quality, whereas uniform-hypersphere regularization spreads templates across the sphere but adds computation and yields negligible accuracy gains over dominant strict unit-sphere losses.The cone-shaped distribution is associated with MagFace and AdaFace; SphereFace+ is given as an example of uniform-hypersphere regularization.Source: §IV · p. 4
- Mainstream facial extractors are categorized into unit-sphere, angle-linear, strict unit-sphere, cone-shaped, and uniform hypersphere distributions, with one expert branch trained for each representative extractor.The survey covers extractors from 2015 to the present.Source: §IV · p. 4
V. METHODOLOGY
GMoE-FTI applies a genuine mixture-of-experts design to facial template inversion, covering five mainstream extractor categories.
V. METHODOLOGY
- GMoE-FTI uses a genuine MoE design to cover all five mainstream facial-template extractor categories.The framework is built on the taxonomy of template distributions.Source: §V · p. 4, Figure 2 · p. 4
A. Core Modules
GMoE-FTI maps heterogeneous facial templates into StyleGAN3-compatible latent codes through five experts, adaptive expert weights, and collaborative refinement.
A. Core Modules
- Five Mapping Experts map templates from five mainstream extractors to expert latent codes, and the Collaborative Mapping Network refines weighted codes for StyleGAN3.Each template is standardized to 1024 dimensions, padding shorter vectors with zeros.Source: A. Core Modules · p. 5
- Each expert uses two fully connected layers and two LeakyReLU activations, taking a 1024-dimensional template concatenated with 8-dimensional random noise to output a 512-dimensional code.Source: A. Core Modules · p. 5
- The Matching Module takes 1024-dimensional templates and outputs five posterior-probability weights for aggregating the expert outputs.The five weights correspond to FaceNet, SphereFace, ArcFace, MagFace, and SphereFace+.Source: A. Core Modules · p. 5
- The Collaborative Mapping Network processes weighted latent codes through two fully connected layers to produce a code compatible with StyleGAN3.The pre-trained StyleGAN3 synthesis network then reconstructs 1024 × 1024 face images.Source: A. Core Modules · p. 5
B. Loss Functions
GMoE-FTI combines four losses to regularize latent codes and supervise template identity, pixel similarity, and facial attributes.
B. Loss Functions
- The training objective combines latent-code, template, pixel, and attribute losses, with weights 1 for the first three and 10 for the attribute loss.The stated weights balance the loss magnitudes for stable optimization of the training objectives.Source: B. Loss Functions · p. 5
- The latent discriminator regularizes expert and collaborative latent codes toward the StyleGAN3 latent distribution.Its loss uses discriminator scores where real codes receive lower scores and generated codes higher scores.Source: B. Loss Functions · p. 5, Figure 3 · p. 5
- Template loss combines element-wise mean squared error and cosine similarity between original and reconstructed-image templates to preserve identity information.The templates are extracted using one of the five facial recognition extractors.Source: B. Loss Functions · p. 5
- Pixel loss penalizes per-pixel RGB discrepancies, while attribute loss minimizes L1 distance between 40-dimensional attribute vectors from original and reconstructed faces.The attribute classifier is pre-trained on CelebA.Source: B. Loss Functions · p. 5
C. Training Strategy
Training progresses from independent expert and matching-module learning to collaborative mapping and joint fine-tuning, with FFHQ for training and LFW for testing.
C. Training Strategy
- Training has three stages: Stage 1 independently trains a five-class matching-module classifier with softmax crossentropy and five experts; Stage 2 trains CMN with four losses while freezing Stage 1 modules and SNS; Stage 3 fine-tunes all but SNS with four losses.Source: C. Training Strategy · p. 6
A. Experimental Setup
- The experiments train on 70,000 FFHQ images and test on 13,233 LFW faces spanning 5,749 identities.The reported setup uses separate training and test splits.Source: A. Experimental Setup · p. 6
B. Evaluation Metrics
The evaluation uses verification and facial-attribute metrics to assess reconstructed identity matching and visual consistency.
B. Evaluation Metrics
- TAR measures reconstructed-face matching at fixed FAR thresholds of 1% and 0.1%.The evaluation distinguishes matching to the corresponding original image (Type-I) from matching other images of the same identity (Type-II).Source: B. Evaluation Metrics · p. 6
- FAPD measures pixel-wise error in aligned eyebrow, eye, nose, and mouth regions, with lower values indicating tighter attribute consistency.Facial landmarks are used to crop and align the regions.Source: B. Evaluation Metrics · p. 6
- FAPC measures semantic feature distance between aligned facial patches using intermediate VGG-16 features, with smaller values indicating greater perceived similarity.Source: B. Evaluation Metrics · p. 6
C. Comparative Experiments
GMoE-FTI is compared with four established FTI methods across templates from training and unseen extractors. The reported advantage is strongest on fully unseen templates, while qualitative comparisons describe identity consistency across backbones.
C. Comparative Experiments
- On fully unseen templates, GMoE-FTI reports a 40% average TAR gain and 2% reductions in both average FAPD and average FAPC.The comparison covers CosFace, CurricularFace, ElasticFace, SphereFace2, SphereFace-R, and TransFace; the supplied tables identify several of these template-specific comparisons.Source: C. Comparative Experiments · p. 6, Table X · p. 7, Table XI · p. 7
- On in-training ArcFace templates, GMoE-FTI performs slightly worse than the SOTA baselines while remaining favorable overall.Source: C. Comparative Experiments · p. 6, Table III · p. 6
- FaceNet, SphereFace, MagFace, and SphereFacePlus are unseen domains for the baselines but belong to GMoE-FTI’s training distribution.The experiments compare these template types with the four representative SOTA methods.Source: C. Comparative Experiments · p. 6, Table IV · p. 7, Table V · p. 7
- Every standalone mapping expert degrades severely on out-of-distribution templates, whereas the complete model’s cooperative fusion improves cross-domain inversion.The CosFace comparison also tests an input dimension unsupported by the existing SOTA methods: 1024 dimensions versus their 512-dimensional support.Source: C. Comparative Experiments · pp. 6, 8
- Qualitative comparisons show GMoE-FTI producing clear, identity-consistent images on unseen templates, while baselines show identity inconsistency and facial distortion.The reported baseline issues include mismatched features and blurriness; the comparisons span Figs. 4–8.Source: C. Comparative Experiments · p. 8
D. Ablation Experiments
Ablations on unseen AdaFace templates assess the necessity of the model’s expert structure, adaptive weighting, weighted aggregation, and joint fine-tuning.
D. Ablation Experiments
- Replacing GMoE with conventional load-balanced MoE hurts generalization on unseen templates.The paper attributes this to static expert allocation and uniform load balancing that cannot adapt to heterogeneous template types.Source: D. Ablation Experiments · p. 8
- Removing the matching module forces equal averaging of five experts’ latent features and leads to cross-branch fusion collapse, identity drift, and a TAR decline.Source: D. Ablation Experiments · pp. 8, 10
- Feeding expert codes and weights directly to the CMN without weighted aggregation damages long-range facial semantics and increases FAPD and FAPC.The reported visual consequence is severe local texture distortion.Source: D. Ablation Experiments · p. 10
- Without end-to-end joint fine-tuning, cross-module coordination collapses and reconstruction quality declines on diverse unseen template distributions.Source: D. Ablation Experiments · p. 10
VII. CONCLUSION
The conclusion presents GMoE-FTI as a universal inversion framework using genuine MoE for identity-consistent reconstruction across prevalent template types. It reports gains on unseen-template tasks and identifies broader privacy-defense applications as future work.
VII. CONCLUSION
- GMoE-FTI uses five independently pre-trained expert branches with adaptive matching and collaborative joint inference to reconstruct identity-consistent faces from five prevalent extractor types.Source: §VII · p. 10
- Quantitative and qualitative experiments are reported to show gains over SOTA models on TAR, FAPD, and FAPC for unseen-template inversion.Source: §VII · p. 10
- Future work will optimize expert number and allocation, improve inversion accuracy, and broaden the framework’s use in biometric privacy defense systems.Source: §VII · p. 10