PaperlayerView exact revision on arXiv

Source-linked AI summary

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

Guray Ozgur, Mustafa Efe Tamyapar, Naser Damer, Fadi Boutros

arXiv:2608.21486v1cs.CV

TL;DR

Face-recognition systems provide similarity scores without revealing which semantic attributes informed a decision. EXPL-FR maps VLM prompts into a frozen FR space and selects a detectable signature that separates identities better than the full vocabulary.

  • Problem

    FR systems can achieve near-saturated accuracy yet expose only similarity scores, leaving practitioners unable to ask why a verification decision was made.

    Source: §1 · p. 1

  • Method

    EXPL-FR aligns frozen VLM image features to a black-box FR space with a face-image-trained adapter, transfers it to text prompts, and selects detectable attributes as anchors.

    Source: §1 · pp. 2–3

  • Results

    The label-free top-100 signature separates identities better than the full vocabulary.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    EXPL-FR treats the concepts that remain detectable after alignment as the semantic structure over which explanations should be read.

    Source: §1 · p. 2

    Without per-image labels, detectability measures VLM pseudo-label separability rather than verified concept presence.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.

1 Introduction

EXPL-FR bridges a frozen face-recognition embedding space to an open, human-specified vocabulary using an image-only adapter, enabling semantic explanations and attribute-level audits without modifying the FR model. Its label-free detectability measure identifies which attributes remain informative after alignment.

1 Introduction

  • Despite near-saturated accuracy, deployed FR systems provide similarity scores without explaining why, while prior XFR work uses spatial or geometric explanations or small prechosen attribute sets.Existing VLM approaches route classifier decisions through concepts or adapt VLMs for performance; a post-hoc bridge into a frozen FR space is missing.Source: §1 · pp. 1–2
  • An image-only adapter aligns frozen VLM image embeddings to a frozen FR space, then transfers to the text encoder to create FR-space semantic anchors without text training.The bridge is post-hoc and requires neither FR-model internals nor retraining.Source: §1 · pp. 2–3
  • The top 100 label-free detectable attributes define a semantic signature that separates identities better than the full vocabulary.Detectability compares each concept’s separability in FR space against VLM space; the face-verification protocol validates text-side transfer across several FR architectures and VLM backbones.Source: §1 · pp. 2–3
  • EXPL-FR benchmarks attribute-level auditing with human labels, VLM pseudo-labels, and fully prompt-driven audits against real verification behavior on RFW, GAN-Control, and CelebA.The benchmark quantifies the trade-off between supervision cost and audit quality.Source: §1 · p. 3
  • The framework supports identity-level, per-image, and genuine/imposter/morph-differential explanations without retraining or white-box access.Identity-level signatures can capture identity-specific attributes, while differential signatures distinguish genuine, imposter, and morph comparisons.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Work

Prior explainable face-recognition methods often localize evidence or depend on annotated attributes, while related vision-language approaches explain other model spaces or require a decision layer. EXPL-FR instead uses textual concepts in the deployed FR encoder’s own coordinates and supports instance-level explanations.

Explainable FR

  • Most explainable FR methods localize evidence spatially or by frequency, rather than expressing a match as similarity to a freely chosen textual concept,,,.Other methods mark pixels arguing for or against a match, estimate trustworthiness, or explain in the frequency domain, without naming attributes.Source: §2 · p. 3
  • Leroy et al. analyze how attributes shape FR embedding geometry but require image labels, GAN-Control, and a vocabulary limited to annotations; EXPL-FR uses prompts and adds instance-level explanations.The paper also compares analyses using labels, pseudo-labels, and prompts alone.Source: §2 · p. 3

VLMs: concepts, tasks, decisions

  • Concept-based methods need a decision layer, while LaVMD aligns through a classifier head and diagnoses subgroup errors rather than explaining a deployed FR encoder.An FR encoder provides extracted features, not the decision layer or shared classification head these approaches use.Source: §2 · p. 3
  • CLIP adapters and face-centric instances adapt within CLIP’s space, while LLM-based methods explain decisions in a separate VLM’s space.EXPL-FR reads the deployed FR encoder’s own coordinates, so its explanations address the matcher itself.Source: §2 · pp. 3–4

Cross-model alignment

  • EXPL-FR’s transfer relies on imperfect coupling between representation convergence across independently trained models and the modality gap within a single VLM.The paper identifies both effects as dependencies of cross-model alignment.Source: §2 · p. 4

3 Methodology

EXPL-FR trains only a lightweight adapter to map frozen VLM image embeddings into the frozen FR space, then reuses it to create prompt-based explanations and audits. Its pipeline selects detectable semantic directions, explains faces and identities, and measures how models structure attributes.

3.1 Setup: Two Frozen Encoders and a Semantic Vocabulary

  • The vocabulary uses one-concept prompts across disjoint categories, reused unchanged across models and datasets to give signatures a common basis; readings remain directional, not disentangled.A prompt can still carry multiple visual factors, such as a hairstyle prompt carrying length, volume, and texture.Source: §3.1 · p. 4

3.2 Image-Only Alignment: Can Two Frozen Spaces Be Bridged?

  • A 4-layer MLP adapter maps frozen VLM image embeddings to frozen FR embeddings, trained on identity-labeled face images with cosine alignment and no attribute labels or text.The loss aligns normalized output directions without penalizing magnitude; the best checkpoint is selected by held-out 1:1 verification accuracy.Source: §3.2 · pp. 4–5

3.3 Zero-Shot Transfer to Text: Anchors and Signatures

  • The image-trained adapter maps text prompts into FR-space anchors without text supervision; the method retains the top 100 prompts by label-free detectability as the semantic signature.Detectability compares pseudo-label separability in adapter-mapped embeddings with that in VLM embeddings; the VLM score is structurally near-saturated, and the resulting discount is a prompt ranking, not an absolute measure.Source: §3.3 · pp. 5–6

3.4 Using Signatures: Explaining Data, Auditing Models

  • Label-free explanations rank retained signature coordinates per image, aggregate them across an identity, or compare image pairs to expose attributes associated with separation or rejection.The method focuses explanations on coordinates retained by the FR model, since discounted coordinates cannot explain its decisions.Source: §3.4 · p. 6

3.5 Auditing FR Models: Supervision Levels

  • Audits measure every candidate concept using FR-space axes built from human labels, VLM-ranked image groups, or prompt sets alone, then validate statistics against verification behavior.The prompt axis uses the first principal direction of mapped ordered prompts (or the normalized two-prompt difference); categorical axes run from the FR population mean toward the mapped prompt-set mean. Audits cover dependence, profile, sensitivity, and attribution; axes and analysis data are disjoint.Source: §3.5 · pp. 6–7
  • The faithfulness protocol compares eight representations, from the FR and raw VLM baselines through aligned embeddings and vocabulary signatures, to test alignment and text-anchor transfer.Using the full candidate vocabulary before selection, comparing unaligned and aligned anchors isolates cross-modal transfer; one representation replaces the FR inference pass with adapter-mapped VLM embeddings.Source: §3.5 · p. 8

4 Results

Results show that an image-trained adapter transfers VLM prompts into an FR space where they preserve identity information, while label-free detectability selects a 100-attribute semantic signature. Prompt-driven audits recover model rankings by ethnicity error and controlled-attribute verification cost without labels, with limitations when the VLM cannot comprehend a concept.

4.1 Experimental Setup

  • The experiments used 978 prompts in 22 categories, four FR targets, two VLM encoders, and an identity-disjoint holdout for adapter training and detectability selection.The primary target was AdaFace ViT-B/WebFace4M; the same held-out identities served as the unlabeled pool for detectability, preventing selection on adapter-training identities.Source: §4.1 · p. 8

4.2 Vocabulary Projection as a Faithful Proxy for the FR

  • Mapping the 978 prompts into FR space raised CLIP vocabulary-projection faithfulness from 51.98% to 71.66%, isolating the contribution of cross-modal transfer.The aligned cross-encoder pairing reached 94.56% mean verification accuracy, within 2.88 points of the FR upper bound; TinyFace remained an exception because of its extremely low-resolution images.Source: Table 1 · p. 9, §4.2 · p. 10

4.3 What Does the FR Model Keep? Selecting the Signature

  • Selecting the top 100 detectable attributes improved identity separability over the full vocabulary, while true-FR rescoring confirmed the keep/discount ordering conservatively.On the identity holdout, detectability ranged from 0.79–0.98 in mapped FR space; mean true-FR AUC was 0.740 versus 0.894 for the mapped measure, but category ordering remained correlated (ρ = 0.63–0.97).Source: §4.1 · p. 9, §4.3 · pp. 11–12
  • CelebA cross-checks showed that named directions can miss densely encoded facial geometry, while hair color and eyewear can genuinely affect identity embeddings.Recoloring hair moved embeddings by 0.07–0.09; GAN-Control sensitivity ranked age highest, and photometric factors were largely erased, with genuine traversal-pair cosine 0.98–0.99.Source: §4.3 · pp. 12–13

4.4 Can You Audit an FR Model Without Supervision?

  • The prompt-driven audit ranked four FR models by RFW verification error and controlled variations by verification cost, including without labels.Across settings, mean dependence and mean real-error rankings were identical; for controlled variation, the prompt-only audit reached ρ = 0.95 on the primary target. Pose and illumination remain VLM blind spots, so their audit results can be inconclusive.Source: Figure 4 · p. 13, §4.4 · p. 14
  • Prompt-based dependence profiles reproduced labeled CelebA structure imperfectly, with stronger agreement for the VLM-proxy audit than the prompt-only audit across attributes.Across four models, mean per-attribute ranking agreement with the supervised audit was τ = 0.63 for the proxy and τ = 0.45 for the prompt-only setting.Source: §4.4 · p. 15

5 Conclusion

EXPL-FR aligns a frozen VLM with a frozen face-recognition model’s embedding space using face images alone, then applies the adapter unchanged to text prompts. A label-free measure selects detectable attributes as semantic signatures for explaining and auditing face-recognition models.

5 Conclusion

  • τ = 0.92 on RFW shows attribute selection survives the prompts-only audit; on CelebA, τ = 0.45 against the proxy’s 0.63 quantifies the cost of removing supervision.Diagnosis remains informative across the three benchmarked supervision settings.Source: §5 · p. 15
  • The label-free measure selects each model’s identity-discriminating attributes, and the top 100 form a semantic signature that separates identities better than the full vocabulary.The adapter aligns the VLM image encoder with the FR space and transfers unchanged to its text encoder, allowing arbitrary vocabularies to supply candidate anchors; faithfulness is measured across four FR targets and two VLMs.Source: §5 · p. 15
  • Signatures support explanations for images, identities, and genuine, imposter, or morph pairs, as well as audits under three supervision settings.The prompt-only audit extends an audit by a written sentence.Source: §5 · p. 15
  • Readings are directional: projection remains 25.78 points below the FR upper bound, prompts are not disentangled, and VLM ranking failures can make pose, illumination, and extreme low-resolution audits inconclusive.The comparison flags these failures imperfectly, and detectability is scored against the VLM’s pseudo-labels and read through the adapter.Source: §5 · p. 15

A Supplementary Figures

The supplement extends EXPL-FR’s evidence across identity-, image-, and differential-level signatures and across all four FR targets, with audits on controlled and standard benchmarks.

Supplementary figures

  • On controlled GAN-Control traversals, age, expression, and hair color are reliably attributed, while photometric changes are diffuse and orientation remains near chance.For the primary model, matching-group attribution is 82% for age, 75% for expression, and 64% for hair color; the other targets show top-1 accuracy 0.36/0.37/0.36 and the same keep/discount split. Orientation limitations reflect CLIP’s failure to rank pose on aligned crops.Source: Figure 6 · p. 23, Figure 16 · p. 30
  • Five sampled CASIA-WebFace identities show distinct semantic signatures that persist across 50 capture conditions, surfacing concepts along which the model separates identities.The signatures use the top 100 prompts by FR-space detectability for AdaFace ViT-B / WebFace4M; one identity, for example, is dominated by eyewear.Source: Figure 7 · p. 24
  • On DCMorph, Ref-Genuine signature differences are small, Ref-Imposter differences name identity-separating attributes, and Ref-Morph differences are smaller and closer to Ref-Genuine.Per-image signatures project each reference, genuine, imposter, and reference×imposter morph onto the same 100-prompt signature; the morph’s residual divergence reflects attributes inherited from the imposter.Source: Figure 8 · p. 25, Figure 9 · p. 26
  • On standard benchmarks, the prompts-only age audit ranks models in line with cross-age penalties (τ = +0.67 / +0.33), whereas pose is inconclusive (0.00 / −1.0).The two pose benchmarks disagree; because CLIP cannot comprehend pose on tightly aligned crops, the pose audit supports no conclusion.Source: Figure 10 · p. 27
  • At CelebA scale, proxy and prompt-driven audits achieve mean Kendall τ = 0.63 and 0.45, respectively, against the supervised ranking, with 90% and 78% consistency.Across GAN-Control targets, Spearman ρ with real EER is 0.81-0.88 for VLM-proxy and 0.69-0.95 for the prompt-driven audit.Source: Figure 11 · p. 27, Figure 12 · p. 28
  • Across four FR targets, the kept top-489 concepts overlap at Jaccard 0.60-0.78, while every model keeps eyewear and hair attributes and discounts distance, context, camera angle, coverings, and lighting.The adapter-mapped versus true detectability gap appears on all targets, making the reported discount conservative; age and ethnicity fall below the median cut only for ViT-B.Source: A Supplementary Figures · p. 29, Figure 13 · p. 29
  • Across the remaining FR targets, CelebA anchor-attribute agreement reproduces the primary model’s pattern: identity-stable attributes score highest, transient accessories are discounted, and low/low attributes remain unrescued.Mean per-attribute AUCs are 0.65/0.65/0.61. Table 2 notes that positively phrased No_Beard with flipped sign recovers to 0.74 / 0.93, unlike naive negation.Source: Figure 14 · p. 29, Table 2 · p. 31
  • For the remaining targets, label-free dependence ranking correlates with ground-truth-label ranking at Spearman ρ = 0.64/0.65/0.70, recovering bald, blond hair, and male as dominant attributes.All p < 10−4, n = 40.Source: Figure 15 · p. 30

B Per-Attribute Values and Between-Model Contrasts

Across four FR targets, top-100 filtered signatures improve identity separability over the full vocabulary; the section also reports cross-model sensitivity differences and broader generality analyses.

B Per-Attribute Values and Between-Model Contrasts

  • ViTs move more along hair color than ResNets (0.085-0.087 vs. 0.067-0.074), while ViT-B is most age-invariant (0.222 vs. 0.248-0.262).These are matched-group |∆| sensitivity contrasts under GAN-Control.Source: B Per-Attribute Values and… · p. 31
  • Generality checks cover SigLIP-B16 and three further FR targets, while remaining-target analyses repeat per-category detectability, CelebA anchor agreement, and label-free dependence ranking.The SigLIP evaluation uses the same faithfulness protocol; the further FR targets are scored by FR upper bound, alignment fidelity, and vocabulary-projection faithfulness.Source: B Per-Attribute Values and… · p. 31, Table 3 · p. 32
  • From m=100, top-m signatures beat random subsets and the full vocabulary on all four targets; from m=50, they do so on three.At m=25, top-m is inconsistent and falls below random on three targets, where near-duplicate prompts dominate. The top-489 kept halves overlap at Jaccard 0.60-0.78 across six model pairs; the top-100 sets overlap at 0.48-0.77.Source: Table 4 · p. 33

C Limitations

EXPL-FR’s explanations are bounded by what its prompt vocabulary can express, what the VLM can measure, and the limits of its validation resources. These constraints make some audits inconclusive and several measurements conservative or noisy.

Identity information outside the vocabulary

  • The 978 named directions leave the FR-space vocabulary projection 25.78 points below the FR upper bound, limiting how finely prompt-based explanations resolve identity.Some identity cues, such as fine distributed facial geometry, may not be nameable, while correlated prompts make attribute readings directional rather than disentangled.Source: C Limitations · p. 34

The VLM bounds what can be measured, and its blind spots fail silently

  • VLM blind spots can make audits of pose, illumination, negated phrasings, and extreme low resolution inconclusive rather than reveal FR invariances.The built-in FR-versus-VLM comparison is the diagnostic; concepts it flags must not be read as findings.Source: C Limitations · p. 34

Detectability rests on pseudo-labels, and is read through the adapter

  • Detectability values are upper bounds, not absolute measures of FR geometry, because pseudo-labels come from the VLM and AUCF is read through adapter-mapped embeddings.Rescoring the same detector on true FR embeddings for 27,648 holdout images widens the gap roughly threefold; its direction is consistent across all four targets, making keep/discount claims conservative.Source: C Limitations · p. 34

The ordering is preserved

  • The ordering is preserved across measures (ρ = 0.63-0.97 per target; within-model top-100 Jaccard overlap 0.56-0.79), but gender changes standing on three of four targets without explanation.AUCF is retained because it ranks whether anchors can reach concepts, not only whether FR space separates them.Source: C Limitations · p. 35

Limits of the ground truth, not the audit

  • Validation resources constrain conclusions: GAN-Control EER saturation prevents per-attribute model ranking, its age traversals confound attribute sensitivity with identity drift, and RFW rankings are noisy.GAN-Control EER is ≤0.014% for most variables; age endpoints have cosine 0.55-0.59 versus ≈0.79 overall. RFW’s MS-Celeb-1M origin may favor MS1MV2, and ranking statistics use n = 4 models or groups.Source: C Limitations · p. 35

Alignment-data mismatch

  • Alignment-data mismatch may partly explain MS1MV2’s lower signature faithfulness: 69.57% versus 75.18% for the same architecture aligned on WebFace4M.All adapters are aligned on WebFace4M, so the difference may reflect training-to-alignment distribution mismatch rather than target geometry alone.Source: C Limitations · p. 35

D Explaining Data Under a Fixed FR Model

Under a fixed AdaFace ViT-B/WebFace4M model with CLIP anchors (row-7 configuration), the section uses the selected signature to explain identities and morphing-related verification differences.

Identity-wise

  • Across 50 images for each of five CASIA-WebFace identities, strongest activation categories differ by identity and remain concentrated across all 50 capture conditions.These profiles identify attribute directions that the model consistently encodes for a person, rather than properties of one photograph.Source: D Explaining Data Under a Fixed FR Model · p. 36

Differential

  • In DCMorph quadruplets, reference–morph differences are smaller than reference–imposter differences and closer to genuine, while residual category-specific divergence exposes semantics inherited from the imposter.Reference–genuine differences are near-uniformly small; the morph retains substantial reference identity semantics, a property that makes morphing attacks succeed.Source: D Explaining Data Under a Fixed FR Model · p. 36

E Selection on the Standard Benchmarks

On standard benchmarks, the label-free age audit tracks cross-age penalties, while pose rankings are inconclusive because the VLM does not reliably represent pose on tightly aligned face crops. The audit compares variation-specific benchmark penalties with estimates made from an independent unlabeled pool.

E Selection on the Standard Benchmarks

  • The audits use graduated age prompts and three frontal-to-profile pose prompts on pooled RFW, comparing each model’s estimate with benchmark error minus its LFW error.CelebA agrees in sign with the pooled-RFW audit.Source: E Selection on the Standard Benchmarks · p. 36
  • τ = +0.67 on AgeDB-30 and +0.33 on CALFW for age, but −1.0 on CPLFW and 0.00 on CFP-FP for pose; pose rankings are inconclusive.The ResNet-100/MS1MV2 model with the lowest pose-audit value pays the largest CPLFW penalty, while CFP-FP shows no agreement.Source: E Selection on the Standard Benchmarks · p. 36
  • CLIP cannot comprehend pose on tightly aligned 112×112 crops, limiting the adapter; within-identity pose projections correlate only weakly with |yaw| (mean ρ = +0.11 to +0.34), versus labeled FR-probe AUC 0.97.Residual correlation can produce systematic-looking but unsupported rankings; disagreement between pose benchmarks signals an audit outside the VLM’s competence, and the FR-vs-VLM comparison flags such blind spots before interpreting results.Source: E Selection on the Standard Benchmarks · pp. 36–37

F Attribution and Sensitivity Under Controlled Variation

Under controlled GAN-Control traversals, prompt-group attribution identifies the varied attribute above chance overall but varies sharply by attribute; sensitivity ranks attributes consistently across four FR models.

Attribution

  • Evaluation uses 500 held-out GAN-Control identities, with anchors built from the other 500; per-target confusions are shown in Fig. 16.The attribution results are shown in Fig. 6.Source: F Attribution and Sensitivity Under… · p. 37
  • The varied attribute is top-attributed in 35% of traversals (35–37% across targets; chance 12.5% with eight competing attributes), with age 82%, expression 75%, and hair color 64%.The aggregate masks a sharp split between reliably attributed encoded attributes and attributes that remain near chance.Source: F Attribution and Sensitivity Under… · p. 37
  • Photometric attributes and orientation remain near chance, but photometric factors are erased by FR invariance whereas pose anchors fail despite FR pose AUC 0.97.Genuine photometric traversal pairs retain cosine 0.98–0.99; CLIP cannot rank pose on tightly aligned 112×112 crops, making pose anchors the weak link.Source: F Attribution and Sensitivity Under… · p. 37

Sensitivity

  • Across all four models, matched-group |∆| ranks age (0.222-0.262) ≫ hair color (0.067-0.087) ≈ expression (0.071-0.084) > orientation ≈ illumination (0.040-0.043) > image quality (≈0.019) > brightness (0.012-0.014) > hue (0.007).Source: F Attribution and Sensitivity Under… · p. 37

G The Semantic Vocabulary and Written Attribute Prompts

The section defines a 978-prompt vocabulary across 22 disjoint categories and describes separate written prompts used for attribute audits, including graduated sets for controlled variations.

G The Semantic Vocabulary and Written Attribute Prompts

  • The candidate vocabulary contains 978 prompts in 22 disjoint categories, each using “A photo of . . . ” with one attribute value.The complete prompt lists ship with the code.Source: G The Semantic Vocabulary and Written… · p. 38
  • Audits use written prompts outside that vocabulary: 40 CelebA attributes, 41 GAN-Control prompts, and three phrasings per RFW ethnicity.Negated CelebA concepts are phrased positively with the sign flipped; an RFW example is “A photo of a person of European descent.”Source: G The Semantic Vocabulary and Written… · p. 38
  • Setting (3) uses graduated variation prompts: six age stages, pose phrasings from frontal to profile, and per-variable GAN-Control level sets.The pose ordering is frontal → three-quarter → profile; the age range runs from baby to elderly.Source: G The Semantic Vocabulary and Written… · p. 38

Related papers

Loading 2608.21486v1…