PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleID: A Perception-Aware Dataset and Metric for Stylization-Agnostic Facial Identity Recognition

Kwan Yun, Changmin Lee, Ayeong Jeong, Youngseo Kim, Seungmi Lee, Junyong Noh

arXiv:2604.21689v1cs.GRcs.CVcs.HCcs.MM

TL;DR

Stylized portraits require identity metrics that remain reliable beyond natural photographs, but existing encoders can misread appearance changes or miss geometric identity drift. StyleID introduces human-judgment benchmarks and psychometric supervision to calibrate semantic encoders across styles and strengths, yielding stronger agreement with human judgments and robustness on artist-drawn portraits.

  • Problem

    Photo-trained and photo-calibrated identity encoders are brittle under stylization, revealing a lack of style-agnostic evaluation and supervision across styles and strengths.

  • Method

    StyleID combines StyleBench-H human same–different judgments with StyleBench-S 2AFC-derived recognition–strength supervision to fine-tune semantic encoders for human-aligned similarity orderings.

  • Results

    StyleID shows higher correlation with human judgments, stronger recognition and retrieval performance than prior identity encoders, and better robustness on out-of-domain artist-drawn portraits.

  • Takeaways & Limitations

    Human perception can anchor identity evaluation and modeling so identity representations remain more stable across stylization styles and strengths.

  • Takeaways & Limitations

    StyleBench-H is limited by human-annotation cost and is demographically skewed toward young, white subjects, which may produce uneven performance for underrepresented and intersectional groups.

Abstract

from arXiv · show

Creative face stylization aims to render portraits in diverse visual idioms such as cartoons, sketches, and paintings while retaining recognizable identity. However, current identity encoders, which are typically trained and calibrated on natural photographs, exhibit severe brittleness under stylization. They often mistake changes in texture or color palette for identity drift or fail to detect geometric exaggerations. This reveals the lack of a style-agnostic framework to evaluate and supervise identity consistency across varying styles and strengths. To address this gap, we introduce StyleID, a human perception-aware dataset and evaluation framework for facial identity under stylization. StyleID comprises two datasets: (i) StyleBench-H, a benchmark that captures human same-different verification judgments across diffusion- and flow-matching-based stylization at multiple style strengths, and (ii) StyleBench-S, a supervision set derived from psychometric recognition-strength curves obtained through controlled two-alternative forced-choice (2AFC) experiments. Leveraging StyleBench-S, we fine-tune existing semantic encoders to align their similarity orderings with human perception across styles and strengths. Experiments demonstrate that our calibrated models yield significantly higher correlation with human judgments and enhanced robustness for out-of-domain, artist drawn portraits. All of our datasets, code, and pretrained models are publicly available at https://kwanyun.github.io/StyleID_page/

1 Introduction

StyleID addresses the mismatch between photo-trained identity encoders and human judgments of identity under stylization. It builds human-anchored evaluation and supervision across stylization methods and strengths, then calibrates encoders to perception.

  • Photo-trained identity encoders can mistake texture or color changes for identity changes, while missing identity drift from exaggerated geometry.
  • StyleID targets the missing style-agnostic, human-calibrated protocol for measuring identity consistency across stylization methods and strengths.
  • StyleBench-H collects human same–different identity judgments for stylized source images generated by three controllable diffusion- and flow-matching-based methods at multiple strengths.
  • StyleBench-S derives recognition–strength curves from controlled 2AFC responses and uses them to generate supervision for human-aligned similarity orderings.
  • StyleID fine-tunes a semantic encoder with contrastive and angular identity losses, producing higher correlation with human judgments and better generalization to artist-drawn portraits.

2 Related Work

Related work establishes that modern identity encoders use geometric margin-based objectives but remain primarily calibrated on photographic data. Face stylization has advanced across translation, GAN, diffusion, and flow-based methods, while existing stylized-face recognition lacks human calibration and varied-strength evaluation.

  • Identity Recognition: Modern face-recognition methods shape normalized hyperspherical embeddings with angular margins and clear decision boundaries.
  • Identity Recognition: Existing identity encoders are trained and calibrated on photographic datasets, creating unstable similarity scores and poor calibration for stylized portraits.
  • Stylized Face Recognition: StylizedFace trains on generated stylized faces but uses a single stylization strength and lacks calibration with human perception.
  • Face Stylization: Face stylization has progressed from image-to-image translation and GAN priors to instruction- or exemplar-based conditioning, diffusion, flow-matching, and consistency generators.
  • Face Stylization: Identity preservation in stylization commonly uses face-ID losses or latent-space constraints, despite the photo-domain training of the underlying encoders.

3 Dataset Construction

StyleBench combines controlled stylization generation with human judgments and psychometric calibration to evaluate identity preservation across styles and strengths. Its construction yields StyleBench-H for benchmarking and StyleBench-S for scalable supervision aligned with recognition behavior.

  • Dataset Construction: StyleBench contains StyleBench-H, a human judgment benchmark, and StyleBench-S, a synthetic supervision set for model calibration.The two subsets support evaluation and large-scale training, respectively.
  • Controllable Stylization Pipeline: Three diffusion- or flow-based frameworks—IP-Adapter, InstantID, and InfiniteYou—provide explicit control over stylization strength.The strength parameters are normalized across methods and discretized into seven levels, though equal normalized values need not have equal perceptual effects.
  • StyleBench-H: Human Perception Benchmark: StyleBench-H pairs source portraits with renderings across 10 artistic styles and seven strength levels for human same-person verification.The benchmark uses filtered FFHQ portraits and explicitly tells annotators that the second image is a stylistic rendering.
  • StyleBench-S: Large-Scale Synthetic Supervision Set: StyleBench-S derives method- and style-specific recognition curves from controlled 2AFC experiments to calibrate synthetic perceptual positives.The calibration study estimates human recognition probability across 10 styles and seven strength levels.
  • StyleBench-S: Large-Scale Synthetic Supervision Set: Recognition declines sharply for IP-Adapter but more gradually for InstantID and InfiniteYou as stylization strength increases.Different artistic styles also produce markedly different recognition behavior, ruling out a single global threshold for identity preservation.
  • StyleBench-S: Large-Scale Synthetic Supervision Set: A 90% human-recognition threshold preserves identity-consistent features while permitting stylistic variation, unlike a 70% threshold.This thresholding rule is used to select perceptual positives for StyleBench-S.

4 Stylization-Agnostic Identity Recognition Model

StyleID adapts CLIP for stylized identity recognition by combining angular identity separation, supervised contrastive learning, and embedding regularization. The objective preserves pretrained CLIP representations while improving identity discrimination across stylized images.

  • Model adaptation: StyleID freezes CLIP and injects LoRA adapters to learn a lightweight, style-robust identity representation.The adaptation is supervised with StyleBench-S and is designed to avoid overfitting and catastrophic drift from CLIP’s pretrained manifold.
  • Angular identity loss: The angular identity loss enforces discriminative angular margins between identity classes.It normalizes sample embeddings and class weights before computing cosine logits for ArcFace-style classification.
  • Supervised contrastive loss: The supervised contrastive loss pulls together samples sharing an identity while pushing apart different identities at the instance level.For each anchor, positives are minibatch samples with the same identity label, complementing the class-level angular margin.
  • Embedding regularization: Embedding regularization constrains adapted representations to remain close to frozen CLIP embeddings, stabilizing training and preventing overfitting.The total objective combines angular identity, supervised contrastive, and regularization losses.

5 Experiments

Experiments evaluate StyleID against identity and semantic encoders across human-annotated and artist-drawn benchmarks, backbone choices, training objectives, and computational variants. StyleID generally provides the strongest stylized identity verification, retrieval, and cross-domain robustness, while lighter variants reduce cost.

  • Comparison Results: StyleID consistently achieved the best verification performance across nearly all StyleBench-H metrics, exceeding 0.9 TPR in Cross-ID and Cross-Style splits.ArcFace and AdaFace reached 0.76–0.78 TPR, while CLIP and SigLIP2 achieved 0.17–0.25 TPR.
  • Comparison Results: StyleID maintained robust verification on artist-drawn SKSF-A portraits, whereas most baselines degraded under the out-of-distribution abstractions.StylizedFace remained limited on SKSF-A, and CLIP was still insufficient for reliable verification.
  • Backbone Selection: StyleID achieved the highest TPR and AUROC on both datasets in backbone replacement experiments, while SigLIP2 attained the highest SKSF-A accuracy.ArcFace* and AdaFace* showed marginal gains, whereas SigLIP2* and StyleID improved substantially.
  • Ablation Study: Combining angular identity and supervised contrastive losses consistently produced the best TPR and AUROC, indicating complementary global separation and cross-style robustness.Removing either loss degraded performance; fixed-threshold accuracy could instead reflect threshold sensitivity.
  • Efficiency: StyleID_small reduces computational cost by approximately 4× relative to the default CLIP-L configuration, while StyleID variants trade performance for lower model size.Figure 8 reports decreasing TPR and AUROC as model size is reduced.
  • Retrieval Test: StyleID achieved 84.7% matching accuracy for human-face queries and 70.44% for stylized-sketch queries, outperforming the best baselines by 10–20%.The results indicate that its embedding space better preserves identity for cross-domain retrieval.

6 Applications

Applications evaluate StyleID as an identity constraint in JoJoGAN and as a model for natural-face verification. Replacing ArcFace with StyleID improves stylization outcomes in automated and user preference evaluations, while StyleID remains competitive but not superior to ArcFace on photorealistic LFW verification.

  • JoJoGAN Application: JoJoGAN with StyleID was preferred over JoJoGAN with ArcFace across identity, expression, style, and overall quality evaluations.The comparison used GPT-5.4 evaluation and a 15-participant user study.
  • JoJoGAN Application: StyleID separates target style transfer from source color distribution more effectively than ArcFace in JoJoGAN, avoiding artifacts such as residual textures and showing teeth.The figure reports artifact-free results while maintaining the original color distribution.
  • Natural-Face Transfer: StyleID remained competitive on LFW natural-face verification but did not outperform ArcFace, whose training is specialized for photorealistic faces.StyleID’s primary strength is identity consistency under stylization rather than maximum performance on standard photographic benchmarks.

7 Conclusion

The paper presents StyleBench-H, StyleBench-S, and StyleID as a perception-aligned framework for evaluating and modeling identity under stylization. It reports stronger human agreement and stylized robustness, while identifying annotation, demographic, synthetic-supervision, pose, and occlusion limitations.

  • Conclusion: StyleBench-H exposes disagreement between existing encoders and human judgments, including failures of photo-domain similarity thresholds across stylization families and intensities.The authors characterize stylized identity preservation as an unsolved evaluation problem requiring perception-grounded protocols.
  • Conclusion: StyleBench-S provides scalable synthetic supervision derived from recognition-strength trends, supporting StyleID training for stable identity representations under stylization.The framework combines human-annotated evaluation with a modeling direction for human-aligned recognition.
  • Conclusion: Across human-annotated and real-world stylized benchmarks, StyleID consistently achieved stronger agreement with human judgments than existing identity encoders.Lightweight StyleID_small and StyleID_tiny variants reduce GFLOPs by approximately 4× to 20× compared with the full model.
  • Limitations: StyleBench-H is limited by human-annotation cost and demographic skew toward young, white subjects, which may produce uneven performance for underrepresented and intersectional groups.The authors call for broader artistic, cultural, demographic, and subgroup-aware evaluation coverage.
  • Limitations: StyleBench-S uses synthetic supervision that may not fully capture the complexity of real-world artistic variation.The authors propose more diverse real stylized data and hybrid human–synthetic supervision as future directions.
  • Limitations: StyleID does not explicitly model extreme pose or occlusion, which may reduce robustness under those conditions.The stated focus is identity robustness under appearance stylization.

B.1 StyleBench-H: Human-Judged Identity Verification Benchmark

StyleBench-H measures identity preservation under controllable face stylization through human perceptual judgments. Its controlled generation and pairwise verification protocol supports analysis across styles and strengths.

  • StyleBench-H evaluates whether stylized renderings preserve the identity of source photographs.
  • Stylized counterparts are generated across artistic styles and multiple stylization methods at seven discrete strengths from 1/7 to 7/7.The levels span mild to extreme appearance changes.
  • Annotators judge whether each source photo and stylized counterpart depict the same person in a pairwise verification task.Instructions emphasize facial structure and characteristic features rather than superficial style mismatch.

B.2 StyleBench-S: Synthetic Supervision via Human-Calibrated Recognition Statistics

StyleBench-S converts human recognition statistics into scalable, perception-aware supervision for training a style-robust identity encoder. Psychometric curves and thresholded positives preserve recognition trends across styles, methods, and strengths.

  • StyleBench-S is a large-scale synthetic training set derived from human recognition statistics.It is designed to preserve StyleBench-H trends while enabling efficient training.
  • Recognition accuracy is measured across strength levels and fit as psychometric curves for each method–style combination.These curves relate stylization strength to human recognition probability.
  • Stylizations with estimated human recognition probability above a high threshold, such as 90%, are selected as perceptual positives.The selection filters samples likely to degrade identity according to human judgment.
  • Complete recognition-accuracy curves show how perceptual difficulty varies across stylization methods and styles as strength changes.These curves motivate threshold-based construction of StyleBench-S.
  • Fixed style sets, discrete strength levels, and consistent pair formatting make the datasets reproducible or extensible under the same questionnaire protocol.The datasets and pretrained StyleID model are stated to be publicly available upon acceptance.
  • StyleBench-H exhibits higher true positive rates than SKSF-A in the low–false positive rate regime, despite SKSF-A's higher AUROC.Linear and log-scale ROC plots expose overall separability and low-FPR differences.

C.2 Additional Comparison

Additional comparisons evaluate a frozen ArcFace AntelopeV2 encoder alongside StyleID on stylized benchmarks. StyleID maintains stronger performance across the reported metrics and datasets.

  • The additional comparison uses ArcFace AntelopeV2 as a frozen encoder because it is distributed in a non-trainable format.This supplements main-paper comparisons using a finetunable ArcFace checkpoint.
  • StyleID outperforms ArcFace AntelopeV2 across TPR, accuracy, and AUROC on both StyleBench-H and SKSF-A.The comparison is reported in Table 10, with best results denoted in bold.
  • ArcFace AntelopeV2 degrades under stylization despite strong natural-photograph performance, whereas StyleID remains consistently high across both datasets.
  • The results indicate that photo-domain face-recognition performance does not directly translate to robustness under appearance shifts.They underscore explicit calibration for stylized and cross-domain scenarios.

C.3 Additional Metric Results

Additional metric results examine stricter false-positive operating points and challenging unseen stylization conditions. The reported evidence emphasizes StyleID's robustness under severe appearance shifts.

  • StyleID is evaluated at stricter operating points of FPR = 10^-3 and 10^-4 on StyleBench-H.These evaluations supplement the main-paper setting of FPR = 10^-2.
  • StyleID consistently achieves the strongest performance across the reported operating points.The results are presented in Table 11.
  • The additional results further confirm StyleID's robustness under severe appearance shifts and highlight limitations of conventional identity encoders in stylized domains.
  • A pilot user study uses Flux.1 Kontext to test whether StyleID's identity-similarity predictions align with human judgments for unseen, challenging stylizations.The study generates novel styles unobserved in standard training or evaluation benchmarks and involves 15 participants.
  • Figure 15 plots full recognition accuracy against stylization strength and circles the style and strength thresholds used to generate StyleBench-S.
Loading 2604.21689v1…