PaperlayerView exact revision on arXiv

Source-linked AI summary

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

Guray Ozgur, Mustafa Efe Tamyapar, Naser Damer, Fadi Boutros

arXiv:2608.21486v1cs.CV

TL;DR

Deep face-recognition models achieve near-saturated accuracy but remain opaque about which semantic attributes support similarity scores. EXPL-FR aligns frozen VLM image embeddings with a frozen FR space using face images only, transfers the adapter to text, and evaluates the resulting semantic anchors for explanation and auditing. The approach selects model-specific detectable attributes for signatures and supports prompt-driven audits against verification behavior, while relying on VLM pseudo-label separability as a stated limitation.

  • Problem

    FR models can produce near-saturated accuracy while giving practitioners no semantic account of why a verification similarity score was produced.

  • Method

    EXPL-FR trains a lightweight adapter from frozen VLM image embeddings to frozen FR embeddings on face images only, then applies it to text prompts and retains detectable attributes as semantic signatures.

  • Results

    Top-100 detectable attributes form model-specific signatures that separate identities better than the full vocabulary and support explanations and audits across four FR models and two VLMs.

  • Takeaways & Limitations

    The prompt-driven audit can rank FR models by measured per-ethnicity RFW errors and rank controlled attribute changes by their true verification cost without labels.

  • Takeaways & Limitations

    Without per-image labels, detectability uses VLM pseudo-labels, which measure pseudo-label separability rather than verified concept presence.

Abstract

from arXiv · show

Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.

1 Introduction

EXPL-FR makes opaque face-recognition embeddings semantically interpretable by aligning a frozen vision-language image space with a frozen FR space, then transferring that alignment to text. It measures which concepts remain detectable and uses them for explanations and label-free auditing.

  • Motivation: EXPL-FR addresses the gap between near-saturated FR accuracy and opaque similarity scores by connecting frozen FR embeddings to an open, human-specified vocabulary.The framework is designed for deployed FR systems where verification decisions can carry legal, financial, or safety consequences.
  • Explanations and auditing: The framework supports identity-level, per-image, and genuine/imposter/morph-differential explanations without retraining or white-box access.It also benchmarks attribute-level auditing under labeled, VLM-proxy, and fully prompt-driven supervision settings against real verification behavior.
  • EXPL-FR: A lightweight image-side adapter trained on face images alone maps frozen VLM image embeddings into frozen FR space and applies unchanged to text prompts.The shared VLM image-text space turns prompts into FR-space semantic anchors without text training or access to FR internals.
  • Semantic signatures: A label-free detectability measure keeps concepts that remain separable after mapping, forming model-specific semantic signatures from the top-100 attributes.The selected coordinates are intended to represent what each FR model retains rather than concepts it has discarded through learned invariances.

2 Related Work

Prior explainable face-recognition methods mainly show where a matcher looked or how its geometry behaves, while other approaches explain decisions through auxiliary models. EXPL-FR instead names evidence using concepts aligned with the deployed FR encoder’s own coordinates.

  • Explainable FR: Most explainable FR methods localize decisions spatially or geometrically using saliency, patch decompositions, neurons, or score backpropagation.These approaches commonly require image pairs, gradients, inpainting, or architecture-specific concept sets.
  • EXPL-FR: EXPL-FR reads the deployed FR encoder’s own coordinates, so its explanations answer for the matcher itself rather than for a second descriptive model.Its transfer depends on imperfect coupling between independently trained representations and the modality gap within a VLM.

3 Methodology

EXPL-FR trains one small adapter from frozen VLM image embeddings to frozen FR embeddings, applies it to text to create FR-space prompt anchors, and selects detectable attributes for semantic signatures. It then uses these anchors and signatures for explanations and audits, while treating prompt transfer as an empirical hypothesis rather than a guarantee.

  • Image-only alignment: EXPL-FR trains only a small adapter mapping frozen VLM image embeddings onto frozen FR embeddings from face images, while both encoders remain frozen.The adapter uses paired embeddings and cosine alignment, with normalized outputs optimized toward target FR directions.
  • Vocabulary: The fixed vocabulary contains single-concept template prompts across disjoint categories, but each prompt can still encode multiple visual factors, so readings remain directional.The same vocabulary is reused across FR models, VLMs, and datasets.
  • Zero-shot text transfer: The adapter is applied unchanged to frozen text embeddings, producing one normalized FR-space anchor per prompt and a face signature from cosine projections onto those anchors.The transfer is unsupervised and measured empirically because the adapter is trained exclusively on image embeddings.
  • Signature selection: A label-free detectability procedure compares prompt separability in VLM and mapped FR spaces, retaining attributes whose FR-space detectability survives the mapping.Its pseudo-labels measure separability relative to the VLM’s own image-text coupling rather than verified concept presence.
  • Signatures and audits: The semantic signature uses the top-m detectable prompts, with m=100, while discounted concepts remain available for auditing rather than explanation.Explanations read only retained coordinates; audits measure every candidate concept because invariances are themselves the audit subject.
  • Auditing FR models: Prompt-driven auditing constructs FR-space axes from mapped prompts and compares projections across concepts, while alternative settings construct axes from labels or VLM pseudo-labels.All settings use the same projection statistics and differ only in how each attribute axis is built.

4 Results

EXPL-FR validates cross-modal alignment across multiple FR targets and VLM encoders, then uses label-free detectability to select readable semantic signatures. The resulting analyses cross-check attribute agreement, identify retained and discounted concepts, and connect unsupervised audits to verification behavior.

  • Experimental setup: Table 1 evaluates alignment across CLIP and SigLIP encoders and four FR targets, with identical adapter architecture and training.The four FR self-verification upper bounds span only 0.36 points, supporting comparison across targets.
  • Vocabulary projection: 94.56% mean verification accuracy for aligned cross-encoder pairs places the adapter within 2.88 points of the 97.44% FR upper bound.The adapter raises CLIP self-verification from 82.33% to 92.58% and SigLIP from 72.77% to 84.78%.
  • Signature selection: 0.79-0.98 is the detectability range in adapter-mapped FR space, compared with 0.976 mean AUC in VLM space, enabling selection of the top-100 signature attributes.Eyewear and hair-related concepts survive most, while distance, scene context, camera angle, face coverings, and lighting are discounted most.
  • Signature selection: 0.63-0.97 is the per-category correlation between mapped and true-FR detectability rankings, while top-100 overlap reaches Jaccard 0.56-0.79.The mapped measure is conservative: AUCF averages 0.894 versus AUC∗F 0.740, with the ordering preserved across all four targets.
  • Labeled cross-checks: 0.89 AUC is achieved by blond-hair agreement on CelebA, while the primary model averages 0.64 across 40 labeled attributes.The FR/VLM comparison preserves the detectability structure; eyeglasses, for example, decline from 0.97 to 0.50.
  • Unsupervised auditing: 0.92 mean Kendall τ links prompt-driven audit rankings to real per-ethnicity RFW-error rankings across four FR models.The ranking by mean dependence is identical to ranking by mean real error under all three audit settings, while controlled-generator EER is at the floor for most variables.

5 Conclusion

EXPL-FR maps a frozen VLM into a frozen FR model’s embedding space, enabling semantic signatures and audits without labels. Its readable vocabulary is selective, and prompt-driven audits track model behavior while remaining bounded by VLM limitations.

  • EXPL-FR aligns a frozen VLM image encoder with a frozen FR space and transfers the adapter unchanged to text prompts.The resulting anchors support explanations for identities, images, and genuine/imposter/morph pairs.
  • The top-100 detectable attributes form each model’s semantic signature and separate identities better than the full vocabulary.Detectability is measured without labels, reflecting which concepts remain represented after FR invariances discard other factors.
  • Prompt-driven auditing ranks four FR models by per-ethnicity RFW errors with τ = 0.92 and remains informative across supervision settings.CelebA reports τ = 0.45 against the proxy’s 0.63 when supervision is dropped.
  • Limitations: The vocabulary projection remains 25.78 points below the FR upper bound, limiting how completely the semantic signature captures FR behavior.

A Supplementary Figures

The supplementary figures extend EXPL-FR’s validation across identities, images, controlled changes, benchmarks, supervision settings, and FR targets. They show recurring semantic signatures, selective attribution, and measurable agreement between label-free audits and supervised or real verification behavior.

  • Controlled variation: 82% of age, 75% of expression, and 64% of hair-color traversals are attributed to their matching prompt groups, while photometric changes remain diffuse.Chance is 12.5%; orientation remains near chance because CLIP cannot rank pose on aligned crops.
  • Identity-wise semantic signatures: Identity-wise signatures aggregate 50 images per identity and reveal distinct, persistent attribute patterns across capture conditions.The examples use five CASIA-WebFace identities and a 100-prompt signature.
  • Per-image semantic signatures: Per-image and differential signatures compare reference, genuine, imposter, and morph images through prompt-level cosine activations and signed differences.
  • Standard-benchmark selection: Age audits rank models consistently with cross-age benchmark penalties, whereas pose audits are inconclusive across the two pose benchmarks.The reported age rank correlations are τ = +0.67 for AgeDB-30 and +0.33 for CALFW.
  • Model diagnosis: Controlled-variation diagnosis remains informative across supervision settings, with matched-axis sensitivity correlations of 0.69-0.95 against real EER.The three settings are labeled, VLM-proxy, and prompt-driven auditing.
  • Label-free auditing: Label-free dependence rankings correlate with supervised audits across CelebA, with Spearman ρ values of 0.64, 0.65, and 0.70 for the remaining targets.The same label-dominant attributes—bald, blond hair, and male—remain dominant across targets.

B Per-Attribute Values and Between-Model Contrasts

Across FR targets and VLM variants, the supplementary analyses quantify between-model sensitivity, extend the evaluation structure, and compare filtered signatures for identity separability. These results clarify both model-specific behavior and the scope of the signature-selection procedure.

  • Between-model contrasts: ViTs move more along hair color than ResNets, with sensitivity 0.085-0.087 versus 0.067-0.074, while ViT-B is most age-invariant at 0.222.The other models’ age sensitivities range from 0.248-0.262.
  • All four targets: The remaining FR targets repeat the primary analyses for detectability, CelebA anchor agreement, and label-free dependence ranking.
  • VLM generality: SigLIP-B16 is evaluated as an alternative VLM, with 768-dimensional embeddings PCA-reduced to 512 dimensions before cosine comparison.
  • Filtered signatures: Table 4 compares top-m, random, bottom-m, and full prompt subsets for identity separability using genuine/imposter AUC on the WebFace4M holdout.The supplied passage identifies the comparison structure but does not provide the continuation of the numerical results.

C Limitations

EXPL-FR’s explanations are bounded by what its vocabulary and VLM can represent. Pseudo-labeling, adapter-mapped measurements, validation resources, and alignment-data mismatch introduce additional limits on how results should be interpreted.

  • The 978 named directions cannot express all identity information, especially fine distributed geometry, and correlated prompts make readings directional rather than disentangled.These vocabulary limits bound the resolution of prompt-based explanations without necessarily limiting the alignment itself.
  • VLM blind spots can silently produce inconclusive audits for pose, illumination, negated phrasings, and extreme low resolution.The FR-versus-VLM comparison is the built-in diagnostic; flagged concepts must not be read as findings.
  • Detectability uses VLM pseudo-labels and adapter-mapped embeddings, so its values show self-consistency and inherit the VLM’s competence limits rather than isolating FR geometry.Rescoring the same detector on true FR embeddings for 27,648 holdout images widens the gap roughly threefold.
  • The ordering of detectability remains preserved, with per-target ρ = 0.63-0.97 and within-model top-100 Jaccard overlap of 0.56-0.79.Gender changes standing between the two measures on three of four targets, without an account for the discrepancy.
  • Validation resources constrain conclusions: GAN-Control EERs saturate, age traversals partly leave identity, RFW overlaps MS1MV2 training data, and rankings use only n = 4 models or groups.These constraints affect the ground truth and make some model-ranking statistics noisy.
  • All adapters are aligned on WebFace4M, so MS1MV2’s 69.57% versus 75.18% signature faithfulness may partly reflect distribution mismatch.The comparison uses the same architecture on MS1MV2 and WebFace4M.

D Explaining Data Under a Fixed FR Model

Under a fixed AdaFace ViT-B / WebFace4M model, EXPL-FR produces identity-level and differential explanations from semantic activation profiles. These profiles identify attributes consistently associated with identities and categories separating morph contributors.

  • The analyses use the fixed row-7 configuration: AdaFace ViT-B / WebFace4M with CLIP anchors.
  • Identity-wise: Identity profiles aggregated over 50 images are markedly identity-specific, with each identity concentrating activations in a different category subset.For example, ID 003879 is dominated by eyewear prompts, and concentration persists across all 50 capture conditions.
  • Differential: Differential morph explanations compare reference-genuine, reference-imposter, and morph pairs to identify semantic categories separating source identities.Reference-genuine differences are near-uniformly small, while reference-imposter differences are large and concentrated in specific categories.

E Selection on the Standard Benchmarks

Prompt audits are compared with cross-age and cross-pose penalties on standard benchmarks using independent unlabeled images. Age rankings align with benchmark penalties, whereas pose rankings are inconclusive when the VLM cannot rank pose.

  • Benchmark setup: AgeDB-30 and CALFW vary age, while CFP-FP and CPLFW vary pose; audits compare independent pooled-RFW rankings with benchmark penalties.The benchmark penalty is each model’s variation error minus its LFW error.
  • Age: τ = +0.67 on AgeDB-30 and +0.33 on CALFW: age audits rank models as their cross-age penalties do without benchmark images.
  • Pose: τ = −1.0 on CPLFW and 0.00 on CFP-FP: pose audits are inconclusive despite systematic-looking rankings.The contradiction arises because the VLM cannot comprehend pose on aligned crops, so the two-space diagnostic must flag the audit before interpretation.

F Attribution and Sensitivity Under Controlled Variation

Controlled GAN-Control traversals test whether prompt differences attribute verification changes to the varied factor and rank its sensitivity. Attribution succeeds most for encoded attributes, while photometric factors and pose remain near chance for different reasons.

  • Evaluation: The controlled evaluation uses 500 held-out GAN-Control identities with anchors built from the other 500.
  • Attribution: 35% of traversals match the varied attribute, versus 12.5% chance when eight attributes compete.Encoded attributes are attributed reliably: age 82%, expression 75%, and hair color 64%.
  • Attribution: Photometric attributes land near chance because FR largely erases them, while orientation stays near chance because CLIP cannot rank pose despite FR pose separability at AUC 0.97.Genuine photometric traversal pairs retain cosine 0.98-0.99; a labeled FR probe separates |yaw| ≥30° at AUC 0.97.
  • Sensitivity: Mean matched-group |∆| ranks sensitivity as age (0.222-0.262) ≫ hair color (0.067-0.087) ≈ expression (0.071-0.084) > orientation ≈ illumination (0.040-0.043) > image quality (≈0.019) > brightness (0.012-0.014) > hue (0.007).
  • Vocabulary: The candidate vocabulary contains 978 prompts across 22 categories, with one example attribute value embedded in a shared template.

G The Semantic Vocabulary and Written Attribute Prompts

The audit uses a fixed vocabulary of 978 single-attribute prompts spanning 22 disjoint categories, supplemented by separately written prompts for specific evaluation datasets.

  • The main vocabulary contains K=978 prompts organized into M=22 disjoint categories.Each prompt uses the template “A photo of . . . ” and specifies one attribute value.
  • Additional written prompts are created independently for CelebA, GAN-Control, and other audit settings rather than drawn from the fixed vocabulary.The supplied text specifies 40 CelebA prompts, 41 prompts across eight GAN-Control groups, and three phrasings per ...
Loading 2608.21486v1…