Source-linked AI summary
Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models
Fizza Rubab, Yiying Tong, Arun Ross
TL;DR
Face-recognition embeddings are effective for matching but opaque to semantic and generative interfaces. The paper aligns them with foundation-model spaces using one linear map estimated from paired embeddings, enabling text retrieval, appearance rendering, and name matching. These capabilities transfer to different degrees: coarse semantics approach native performance, appearance stops short of fine identity, and identity supports ranking names without calibrated open-set rejection.
Problem
Face-recognition embeddings support biometric matching but lack direct semantic, visual, and gallery-free identity interfaces.
Method
The paper fits one linear transformation from paired face and foundation embeddings and applies existing foundation-model heads without labels or model retraining.
Results
Aligned embeddings support text search, realistic appearance rendering, and name matching, while unaligned and randomly transformed embeddings perform at chance across tasks.
Takeaways & Limitations
Alignment exposes semantic, visual, and identity-related information already present in face embeddings, with transfer strength varying across capabilities.
Takeaways & Limitations
Decoded faces recover appearance and soft-biometric attributes but not fine identity, because the diffusion decoders are appearance-oriented.
Abstract
from arXiv · showhide
Modern face recognition (FR) owes much of its success to deep neural networks that learn to extract compact identity embeddings from face images. These models are typically trained for identity discrimination, producing embeddings that are highly effective for biometric matching but largely opaque to semantic interpretation. In contrast, foundation models, pretrained on broad visual or vision--language tasks, provide rich interfaces for describing, retrieving, generating, and organizing visual content. This contrast raises a natural question: what capabilities become available when face embeddings from domain-specific FR models are made interoperable with foundation models? Building on recent work on embedding compatibility across models, we use simple pre-computed linear transformations, estimated from paired embeddings alone, to connect existing FR models with off-the-shelf foundation models. Once aligned with a foundation model, a face embedding can be 'unmasked' in multiple ways, without training or modifying either model: it can be read in natural language, enabling free-form text queries over a gallery of FR embeddings; rendered into a face image that recovers a person's appearance, using an unmodified diffusion decoder; and converted to a name, enabling identification even in the absence of an enrolled face gallery. In effect, one linear transformation turns an identity embedding into a rich embedding for web-scale foundation models. This interoperability exposes face embeddings as semantically and visually rich biometric representations, with direct implications for interpretability, retrieval, reconstruction, and template security.
1 Introduction
The paper proposes aligning face-recognition embeddings with foundation-model spaces so they can expose semantic, visual, and identity-related capabilities without modifying the underlying models.
- Motivation: Domain-specific face embeddings support biometric matching but cannot directly be queried in language, decoded into images, or linked to names without an enrolled gallery.These models are trained with face images and identity labels alone.
- Motivation: Foundation models provide language retrieval, semantic description, embedding-to-image decoding, and name associations learned from broad web-scale data.The cited capabilities come from image–text models, diffusion models, and web-scale pretraining.
- Approach: A single linear transformation estimated from paired embeddings aligns a face-recognition model with an off-the-shelf foundation model without labels, fine-tuning, or retraining downstream components.The transformation is estimated once from paired embeddings and then used directly by the foundation model.
- Capabilities: Aligned embeddings can be searched with free-form text, rendered as realistic face images, and matched to names of well-known individuals.The reported capabilities span retrieval, appearance reconstruction, and naming.
- Evaluation: Comparisons with native, unaligned, and randomly transformed embeddings test whether the reported capabilities arise from the learned alignment.The baselines are intended to separate alignment effects from incidental factors.
2 Related Work
Prior work studies face representations, soft-biometric information, retrieval, inversion, and open-set recognition, but this paper uses one generic linear alignment for multiple capabilities.
- Face and Foundation Representations: Face-specific models optimize identity discrimination, whereas foundation models learn general-purpose representations for diverse downstream tasks.Contrastive image–text models align visual and textual features, while diffusion decoders map image representations back into images.
- Prior Capabilities: Earlier approaches recover soft-biometric information, retrieve faces from language, or reconstruct faces using task- or model-specific training.Examples include supervised classifiers, paired face-caption models, reconstruction networks, diffusion-based inverters, and identity-conditioned generators.
- Positioning: The paper differs by aligning FR embeddings into generic vision–language spaces with one closed-form linear transformation supporting retrieval, reconstruction, and naming.The approach uses no face-specific training and reuses the same alignment across capabilities.
3 Method
The method fits a centered least-squares linear map from paired face and foundation embeddings, then applies foundation-model heads to the transformed representation while controlling for alignment-specific effects.
- Linear Alignment: Paired source and target embeddings are used to estimate a linear transformation by least squares without labels beyond the pairing.The source is a face model and the target is a foundation model evaluated on the same images.
- Linear Alignment: Centering by the training-set means and restoring the target mean places each transformed embedding in the foundation model’s coordinate frame.Downstream heads then operate on the transformed embedding as if it were in the target space.
- Deployment: The fitted map is computed once per face–foundation pair, takes only a few seconds on a CPU, and can feed any foundation head.A head is any downstream module that accepts the foundation embedding.
- Baselines: Native, aligned, unaligned, and random methods provide ceiling and control comparisons for every evaluated task.Unaligned uses the raw source embedding, while Random uses a matrix with the same shape as the learned transformation.
- Head Compatibility: Diffusion decoders require target embeddings from their specific conditioning encoders, so each head is aligned to its own expected input space.Kandinsky uses CLIP ViT-bigG, while Stable unCLIP uses CLIP ViT-H.
4 Experimental Setup
The experiments combine multiple face-recognition source models with general-purpose text and diffusion foundation models, using public checkpoints without fine-tuning.
- Models: The study evaluates ArcFace, AdaFace, AdaFace-ViT, and KPRPE sources spanning CNN and ViT architectures.The face models include IR-101 CNNs and ViT-B-based models.
- Foundation Targets: Text-based retrieval and naming use CLIP, MetaCLIP, and SigLIP, while generation uses Kandinsky 2.2 and Stable unCLIP.The diffusion decoders are conditioned on CLIP ViT-bigG and CLIP ViT-H embeddings, respectively.
- Foundation Targets: None of the five foundation targets uses a face-specific objective, so recovered capabilities derive from generic web-scale pretraining.DINOv2 is additionally used as an independent appearance-similarity reference space.
5 Searching Embeddings with Language
The aligned embeddings make an FR gallery searchable with natural-language queries, with retrieval performance close to the foundation model’s native ceiling across demographic and attribute prompts. The improvement depends on learned alignment rather than unaligned or random transformations.
- Method: Natural-language queries are encoded by the foundation model and ranked against aligned FR embeddings using cosine similarity.This converts a face-only gallery search into text-to-face retrieval.
- Results: Aligned retrieval closely tracks native performance and can exceed it on UTKFace, including 0.809 versus 0.795 for MetaCLIP and 0.796 versus 0.762 for SigLIP.These comparisons are reported for aligned AdaFace-ViT on two of three UTKFace targets.
- Results: Unaligned and randomly transformed embeddings remain at chance, attributing the retrieval gains to the learned alignment across all three image–text models.The evaluated models differ in pretraining corpus and contrastive objective.
- Results: Free-form prompts retrieve faces matching descriptions that the source FR model was never explicitly trained to represent.Figure 2 shows top-five retrievals with one image per identity from held-out identities.
6 Decoding Embeddings into Faces
A linear map into a diffusion decoder’s conditioning space turns FR embeddings into realistic face images that preserve appearance and soft-biometric attributes. The reconstructions do not preserve fine identity, reflecting a limitation of the decoder rather than the alignment.
- Method: The method transforms each FR embedding into the image-embedding space expected by an unmodified diffusion decoder.Evaluation uses 1,500 held-out CFP test faces and measures attributes, appearance, realism, and identity retention.
- Appearance and realism: Aligned outputs achieve DINOv2 resemblance of approximately 0.53–0.59 and FID of 64–81, substantially better than random and unaligned baselines.Native embeddings remain stronger, with DINOv2 resemblance near 0.68–0.69 and the best FID.
- Identity retention: Fine identity does not survive decoding: aligned reconstructions score 0.04–0.16, near the decoder-limited ceiling rather than genuine-match scores.Even native CLIP-conditioned decoding reaches only 0.10–0.17, while genuine matches score 0.49±0.17 under ArcFace and 0.51±0.17 under AdaFace.
- Interpretation: The decoder limitation means the capability should be treated as appearance recovery, not identity reconstruction.A face-native Arc2Face reference recovers more identity but still falls well short of a genuine match.
7 Naming a Face from Its Embedding
Aligned face embeddings support zero-shot naming against candidate name text embeddings, including without an enrolled face gallery. Naming remains above chance, tracks native performance as vocabularies expand, depends on web exposure, and transfers across datasets with ViT sources.
- Zero-shot naming: Aligned embeddings enable naming well-known individuals by matching them against foundation-model text embeddings, without an enrolled labeled face gallery.On CFP’s 500-name vocabulary, every aligned source performs far above chance while unaligned and random baselines remain at chance.
- Closed-set naming: AdaFace-ViT reaches 63.0% top-1 naming accuracy through CLIP, while KPRPE reaches 61.0% and 92.0% top-10.Across three foundation targets, aligned ViT embeddings reach up to 73% of the native ceiling, with a ViT source best for every target.
- Backbone effects: Matching architecture matters: with objective and training data fixed, AdaFace-ViT more than doubles naming accuracy relative to the aligned IR-101 CNN.Through CLIP, the CNN reaches 24.7% top-1 versus 63.0% for AdaFace-ViT.
- Vocabulary size: As the vocabulary grows from 500 to 4,000 names, aligned KPRPE→CLIP top-1 accuracy falls from 61.0% to 49.5%, paralleling the native ceiling.Aligned ArcFace declines from 16.7% to 10.8%, while chance drops from 0.2% to 0.025%; unaligned and random methods remain at chance.
- Web exposure: Naming accuracy rises with web exposure, from 11.6% for identities with one LFW image to 95.2% for the most-photographed identities.Aligned KPRPE→CLIP follows the native ceiling across exposure bins, indicating that naming reflects representation in pretraining data.
- Cross-dataset transfer: ViT sources transfer across datasets with little naming loss: KPRPE changes from 25.4% to 25.3% top-1, while retrieval falls from 0.761 to 0.610 mAP.CNN sources degrade more, and the larger retrieval drop is associated with the wider CelebA–UTKFace domain gap.
8 Discussion
Alignment transfers different kinds of information from face embeddings with unequal strength: coarse semantics and soft-biometric attributes transfer most readily, appearance supports reconstruction without fine identity, and identity supports ranking names without calibrated open-set rejection.
- Coarse semantics and soft-biometric attributes transfer near the foundation models’ native ceiling.
- Appearance transfers sufficiently to drive a generator, but not to recover fine identity details.The passage attributes this boundary to appearance-oriented decoders rather than alignment.
- Identity transfers enough to rank the correct name highly in a large vocabulary, but not enough for calibrated open-set rejection.
- The two ViT embeddings are named correctly at 3.0 to 3.7× the rate of the two CNN embeddings across every foundation target.Within this four-model set, matched-pair comparisons point to the backbone rather than the training recipe.
- A stored biometric template can become searchable, renderable, and sometimes nameable through public models and one linear fit.The discussion calls for protection schemes that resist alignment with public foundation spaces while preserving matching accuracy.
9 Conclusion
The paper shows that a single linear alignment can unmask face embeddings for semantic search, face rendering, and name matching without labels, fine-tuning, or specialized networks. These capabilities broaden the security implications of biometric templates because publicly available foundation models can expose attributes, appearance, and identity cues.
- A single linear transformation aligns face-recognition embeddings with foundation models without labels, fine-tuning, or specialized networks.
- Aligned face embeddings support free-form text search, realistic face decoding, and matching to a person’s name.
- The findings broaden the threat model because matching-only templates may disclose semantic attributes, visual appearance, and name-level identity cues.