Source-linked AI summary
AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relations
Ying Huang, Wencan Zhang, Brian Y. Lim
TL;DR
Existing face-similarity metrics poorly proxy human perception because they lack cognitive alignment and assume a universal observer. AlignFace combines cognitively grounded attributes, interpretable modeling, and nonlinear perception mapping, achieving stronger alignment with human judgments than existing baselines across overall, attribute-level, and subpopulation evaluations.
Problem
Existing face-similarity metrics are poor proxies for human perception because they lack cognitive alignment and assume a universal observer.
Method
The paper collects FACETS human similarity judgments and uses cognitively grounded attributes, concept bottlenecks, and neural generalized additive modeling to build interpretable AlignFace.
Results
AlignFace correlates more strongly with human perception than heuristic, learned-perceptual, and cross-modal baselines overall, per attribute, and across participant groups.
Takeaways & Limitations
AlignFace provides a more faithful and interpretable proxy for human face perception when evaluating generated facial content.
Takeaways & Limitations
The study uses controlled settings, a relatively compact dataset, a specific participant demographic, and excludes trials with 30–70% rater agreement.
Abstract
from arXiv · showhide
Computer vision models for generated facial content, such as face editing and privacy protection, increasingly affect people, requiring similarity metrics that serve as faithful proxies for human perception. While perceptual evaluation has progressed from signal-based heuristics to representation-based metrics, current approaches are limited to behavioral modeling without cognitive alignment. They rely on implicit and spurious relations while assuming a universal observer, failing to account for inherent variations across diverse human populations. This leads to inaccurate evaluative models of stakeholders and misleading guidance for generative model debugging. Rather than treating perception as a black box, we leverage scientific findings from cognitive psychology of human face similarity perception: dependence on facial featural and configural attributes, nonlinear psychophysical response scaling, and own-group biases. We introduce the FACETS dataset and propose AlignFace, an interpretable, human-aligned, face similarity metric that encodes these cognitive principles through ante-hoc modeling. It employs visual-language modeling (VLM) to encode paired face images and text-based attributes, gated cross-attention (CA) to extract attribute-specific facial difference representations, concept bottleneck modeling (CBM) to constrain reasoning via interpretable face attributes, and neural generalized additive model (GAM) to model their nonlinear influence. Experiments found AlignFace significantly improves alignment with human subpopulation perceptions compared to baseline metrics, including recent domain-free learned perceptual metrics. By bridging learned representations and human cognitive processes, this work enables more transparent and aligned perceptual evaluation metrics for face images.
1 Introduction
The paper introduces AlignFace, a cognitively grounded and interpretable face-similarity metric designed to better model human perception across attributes and demographic groups. It is trained and evaluated using large-scale human similarity annotations and integrates visual-language encoding, gated cross-attention, concept bottlenecks, and nonlinear perceptual mapping.
- Motivation: Human-centric face editing and privacy applications require evaluation metrics that serve as reliable proxies for human perception.Standard metrics and models are described as poor proxies for human perception in these data-centric applications.
- Cognitive principles: Human face perception depends on featural and configural attributes, nonlinear perceptual scaling, and demographic own-group bias.The identified demographic bias concerns White and Asian groups.
- Dataset: 9.36k triplet ratings from 78 participants covered overall face similarity, while 73.32k triplet ratings from 611 participants covered 20 face attributes.The ratings were collected through two-alternative forced choice comparison tasks.
- Method: AlignFace predicts interpretable face attributes from paired images and textual prompts, then maps them to overall perception through nonlinear modeling.Its design uses contextual fusion, multi-label concept bottleneck modeling, and nonlinear attribute-overall perception mapping.
- Results: AlignFace correlated more strongly with human perception than heuristic similarity, learned perceptual, and cross-modal baselines overall, per attribute, and per group.The evaluation compared overall and attribute-specific similarity for all humans and specific ethnic groups, including module ablations.
2 Related Work
Prior face representation and perceptual similarity models are largely data-driven and can miss fine-grained human perception or rely on spurious correlations. Cognitive face-perception principles and ante-hoc interpretability motivate AlignFace’s human-aligned, interpretable approach.
- Face Representation Models: Face representation models support identity, expression, landmark, and attribute tasks but remain primarily data-centric despite using human-annotated data.Common backbones include margin-based ArcFace and CosFace, alongside triplet-loss-based FaceNet.
- Perceptual Similarity Metrics: Conventional PSNR, SSIM, CLIP, and DINO metrics provide poor proxies for human judgment because they miss semantics or fine-grained perceptual nuances.LPIPS and DreamSim improve alignment, but their black-box nature risks dependence on spurious features and correlations.
- Human Face Perception: Human face perception combines featural and configural attributes additively, with nonlinear psychophysical effects and observer-dependent own-group bias.Own-group bias is characterized by superior recognition of one’s own demographic group.
- Explainable Vision Models: Ante-hoc interpretability encodes cognitive principles or domain knowledge, unlike potentially spurious saliency-map explanations, while concept-based methods expose human-understandable concepts.Explainable-AI work has focused mainly on discrimination, with comparatively few methods addressing similarity tasks, including facial similarity.
3 Face Attributes for Comparative Evaluation with Triplet Similarity (FACETS) Dataset
FACETS collects comparative human judgments of overall and 20-attribute face similarity from edited face triplets, grounded in featural and configural attributes. The dataset reveals unequal, nonlinear, and demographic-dependent attribute effects that motivate human-aligned modeling.
- Dataset construction: FACETS measures overall and per-attribute similarity across 20 cognitively grounded featural and configural face attributes.The attributes were selected as highly influential in face space and edited using diffusion-based inpainting, landmark-based warping, and GAN-based transfer.
- Human evaluation: The dataset contains 480 edited face triplets from 120 identities, with 8,880 overall and 72,450 attribute triplet ratings split 80% for training and 20% for testing.Each triplet comprises an original reference and two edited versions, and ratings were collected using comparative 2AFC judgments.
- Data quality: 87.0% agreement with human judgments was achieved after fine-tuning the pretrained VLM on 8,880 ratings.The paper characterizes this participant sample as modest but sufficient for fine-tuning and high agreement.
- Cognitive findings: Human similarity judgments assign significantly greater influence to attributes such as face aspect ratio and forehead height than to mouth size and lip color.Attribute relevance was estimated with generalized additive models using 2AFC selection as the response and relative attribute-change differences as factors.
- Cognitive findings: Perceptual sensitivity to attribute differences is strongly nonlinear, and own-group versus other-group ratings emphasize different attributes for White and Asian participants.Own-group judgments were more sensitive to face aspect ratio and forehead height, whereas other-group judgments were more sensitive to skin color and cheek shape.
4 AlignFace for Perceptual Similarity Metric
AlignFace is an interpretable face-similarity metric designed to model overall and attribute-level perception through cognitive principles. Its architecture combines semantic visual-language representations, attribute-gated reasoning, concept bottlenecks, and nonlinear additive modeling.
- Model objective: AlignFace predicts overall and attribute-level face similarity while accounting for multiple attributes, nonlinear scaling effects, and demographic-specific variance.The model is grounded in cognitive principles identified earlier in the paper.
- Semantic representation: AlignFace uses vision-language models to embed face images and open-ended attribute prompts in a shared semantic space, enabling attribute-targeted comparisons.A shared semantic representation addresses the limitations of semantic-agnostic standard visual encoders.
- Attribute-conditioned comparison: Attribute-Gated Cross-Attention conditions image-difference features on textual attributes, using a learnable gate and residual connection to retrieve attribute-relevant visual information.Image-difference features serve as queries, while textual attribute features serve as keys and values.
- Concept bottleneck: The resulting attribute distance measures are interpretable binary concepts that bottleneck subsequent reasoning, forming a concept bottleneck model.These concepts are denoted Δĉ_attr ∈ [−1, +1].
- Nonlinear aggregation: A neural Generalized Additive Model represents overall distance as additive nonlinear functions of attribute distances while enforcing that greater attribute differences cannot reduce overall face distance.Each attribute contribution is modeled by a spline function implemented with a small neural network.
- Triplet metric training: AlignFace2AFC applies AlignFace to both reference-face pairs in a triplet and learns relative overall and attribute distances from human similarity-selection labels.Training uses squared hinge loss with margin m = 0.05 to enforce relative ranking.
5 Experiments
Experiments evaluate AlignFace’s agreement with human face-similarity judgments, attribute-level alignment, reasoning faithfulness, subgroup perception, and robustness to dataset shifts. AlignFace outperforms competing metrics, while ablations show that attribute-gated cross-attention and interpretable nonlinear aggregation are important for alignment.
- Evaluation Protocol: AlignFace is evaluated against baselines using held-out triplets, behavioral agreement with human preferences, and partial dependence plots for reasoning analysis.The evaluation compares predicted metric preferences with human annotations and examines attribute-specific relationships with overall face perception.
- Reasoning and Subpopulation Alignment: PDP analysis shows high nGAM–human correlations of 0.94–0.98, while AlignFace agrees more strongly with Own-group than Other-group perception.The stronger Own-group agreement may reflect fewer Other-group faces in the imbalanced training data.
- Overall Face Perception Alignment: AlignFace outperforms all competing metrics for overall face perception, while heuristic and self-supervised models show the lowest human-rating consistency.Perceptual methods outperform those groups; DreamSim slightly exceeds LPIPS, VLM baselines perform relatively strongly, and face-specific models perform significantly worse.
- Face Attributes Alignment: AlignFace achieves M=0.79 attribute agreement with the FLIP backbone, with strongest results for Hair length, Eyebrow shape, and Forehead height.Agreement is lowest for Nose shape, Jaw width, and Chin shape, and remains below overall face agreement because of attribute diversity and data sparsity.
- Domain-Shift Robustness: AlignFace maintains consistent agreement across within-dataset and cross-dataset evaluation, indicating robustness to domain shifts between CMU Multi-PIE and CelebA.Models with CLIP and FLIP backbones were retrained separately on the two datasets and tested under both settings.
- Ablation Studies: Replacing attribute-gated cross-attention reduces overall and attribute agreement, while Linear aggregation performs worst and MLP matches nGAM without interpretability.The direct-fusion variant misses attribute-specific cues; MLP also learns spurious trends despite its nonlinear capacity.
6 Discussion
AlignFace grounds face similarity modeling in cognitive principles while establishing a foundation for human-aligned evaluation. The approach remains limited by controlled settings and a compact, demographically specific dataset but may extend to semantic-aware visual similarity beyond faces.
- Contributions: AlignFace grounds face similarity modeling in featural and configural attributes, nonlinear scaling, and own-group bias.These principles provide the foundation for human-aligned face similarity modeling.
- Limitations: The study focuses on controlled perceptual settings, a relatively compact dataset, a specific participant demographic, and excludes trials with 30–70% rater agreement.These constraints limit the scope of the current foundation for human-aligned face similarity.
- Future directions: AlignFace suggests a general approach for semantic-aware visual similarity by using prompt-based VLMs to model open-ended concepts beyond face perception.The paper gives ABCD criteria for assessing skin-lesion disease progression as an example domain concept.
7 Conclusion
AlignFace is presented as a human-aligned face similarity metric grounded in cognitive principles, aligning with both predictive behavioral responses and underlying model reasoning. The conclusion also notes that crowdsourced training may omit Point of Subjective Equality hard examples for fine-grained perceptual differences.
- Contributions: AlignFace models both face and fine-grained attribute perception using scientific cognitive principles.The metric is designed to align with human perception rather than rely solely on behavioral modeling.
- Contributions: AlignFace achieves alignment with predictive behavioral responses and underlying model reasoning.This alignment is reported across both evaluated aspects of perception.
- Contributions: AlignFace significantly outperforms existing domain-free metrics in mirroring human judgment.The conclusion attributes this finding to experimental evaluations.
- Limitations: Crowdsourced model training may omit Point of Subjective Equality hard examples for fine-grained perceptual differences.This limitation concerns the omission of difficult perceptual comparisons from training data.
A Appendix · A.1 Dataset and Participant Details · Appendix Table 3. Demographic distribution of face targets selected during dataset curation.
The appendix reports demographic gender categories for curated face targets and describes feature distributions for source and edited faces across 20 attributes. The appendix figure notes that geometry-based heuristics provide proxy evaluation metrics.
- Appendix Table 3. Demographic distribution of face targets selected during dataset curation.: Appendix Table 3 lists Female and Male as gender categories for selected face targets.
- A.1 Dataset and Participant Details: The table text presents the gender labels in repeated Female–Male pairs.
- Appendix Fig. 11: Appendix Fig. 11 summarizes feature-value distributions for source faces across 20 face attributes.
- Appendix Fig. 11: Appendix Fig. 11 also summarizes feature-value distributions for edited faces across the same 20 attributes.
- Appendix Fig. 11: The feature values are computed using geometry-based heuristic methods.
- Appendix Fig. 11: These geometry-derived feature values serve as proxy evaluation metrics.
Appendix Table 4. Demographic statistics of participants in our psychophysical similarity measurement studies. … Appendix Fig. 15. Cross-group attribute perception agreement for a) attribute categories and b) individual attributes.
The appendix reports inter-rater agreement, statistical comparisons, module validation, baseline implementation details, and supplementary analyses supporting AlignFace’s interpretability and human-alignment claims. Reported results include strong correspondence between predicted attribute distances and physical measurements, plus significant AF-FLIP differences from most baselines.
- Appendix Table 5. Inter-rater agreement (Fleiss’ 𝜅) [35] for overall face and 20 facial attributes.: Inter-rater agreement was substantial for eyebrow shape, eyebrow thickness, eye color, and hair length, while agreement was moderate for the remaining reported facial attributes.Reported Fleiss’ κ values range from 0.537 for skin color to 0.736 for hair length.
- Appendix Table 8. Justifications for AlignFace modules compared to standard alternatives.: AlignFace’s visual-language model supports open-domain, label-free attributes more scalably than hard-coded feature engineering.Attr-gated cross-attention additionally focuses reasoning on attribute-relevant information while reducing dimensionality and preserving interpretability.
- Appendix Table 10. Baseline implementation details.: Baseline evaluation used official pretrained checkpoints, standardized preprocessing, and 224 × 224 inputs by default, with 112 × 112 inputs for face-recognition models.PSNR and SSIM used scikit-learn; other models were implemented in PyTorch.
- A.5 Supplementary Results: Supplementary analyses examine partial dependence between relative attribute judgments and overall face selection or distance differences across 20 attributes.The distance-difference plots report Pearson correlations between human judgments and comparators.