PaperlayerView exact revision on arXiv

Source-linked AI summary

Arc2Face: A Foundation Model for ID-Consistent Human Faces

Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, Jiankang Deng, Bernhard Kainz, Stefanos Zafeiriou

arXiv:2403.11641v2cs.CV

TL;DR

Combining text and identity features can make controlled, identity-conditioned face generation difficult, while limited high-quality datasets constrain model development. Arc2Face fine-tunes Stable Diffusion using ArcFace identity features and restored WebFace42M images, and the authors report greater identity similarity alongside diversity and realism compared with existing methods.

  • Problem

    Combining text and identity features can make controlled generation conditioned on a specific subject’s identity difficult.

    Source: §1 · p. 3

  • Method

    Arc2Face fine-tunes pretrained Stable Diffusion for ID-to-face generation using ArcFace features alone and restored WebFace42M images.

    Source: Face Recognition Features · p. 5, Dataset · p. 8, §6 · p. 14

  • Results

    Arc2Face generates realistic and diverse images with greater facial identity similarity than existing methods.

    Source: Datasets and Metrics · p. 10, §6 · p. 14

  • Takeaways & Limitations

    The authors present Arc2Face as a foundation model for generating faces from identity embeddings and supporting follow-up research through further fine-tuning.

    Source: §5 · p. 14, §6 · p. 14

    Generation is limited by the capacity of the face-embedding encoder and the datasets used, and only one person can be generated per image.

    Source: §5 · p. 14

Abstract

from arXiv · show

This paper presents Arc2Face, an identity-conditioned face foundation model, which, given the ArcFace embedding of a person, can generate diverse photo-realistic images with an unparalleled degree of face similarity than existing models. Despite previous attempts to decode face recognition features into detailed images, we find that common high-resolution datasets (e.g. FFHQ) lack sufficient identities to reconstruct any subject. To that end, we meticulously upsample a significant portion of the WebFace42M database, the largest public dataset for face recognition (FR). Arc2Face builds upon a pretrained Stable Diffusion model, yet adapts it to the task of ID-to-face generation, conditioned solely on ID vectors. Deviating from recent works that combine ID with text embeddings for zero-shot personalization of text-to-image models, we emphasize on the compactness of FR features, which can fully capture the essence of the human face, as opposed to hand-crafted prompts. Crucially, text-augmented models struggle to decouple identity and text, usually necessitating some description of the given face to achieve satisfactory similarity. Arc2Face, however, only needs the discriminative features of ArcFace to guide the generation, offering a robust prior for a plethora of tasks where ID consistency is of paramount importance. As an example, we train a FR model on synthetic images from our model and achieve superior performance to existing synthetic datasets.

1 Introduction

Face-recognition embeddings provide a powerful representation of identity, but combining identity and text features can make controlled, identity-consistent generation difficult. The paper introduces an ID-conditioned face foundation model and a large, identity-rich training dataset to address this gap.

1 Introduction

  • Robust identity control in StyleGAN remains challenging because it requires large, high-resolution datasets with substantial intra-class variability.The paper identifies limited high-quality facial databases as a challenge for building a foundation model.Source: §1 · pp. 2–3
  • Text and identity features can conflict because text representations carry identity information and may contradict the supplied identity, limiting controlled generation.The authors argue such combined models are unsuitable for controlled generation conditioned on a specific subject’s ID features.Source: §1 · p. 3
  • Arc2Face achieves superior performance to existing approaches as an ID-conditioned face foundation model, alongside a proposed robust benchmark.The authors also introduce the model publicly.Source: §1 · p. 3
  • The authors introduce a high-resolution dataset derived from WebFace42M, adapt a facial identity encoder using ID features alone, and report that smaller single-image-per-person datasets are insufficient.The contribution list also describes the resulting model as the first ID-conditioned face foundation model.Source: §1 · p. 3

2 Related Work

Generative methods have advanced from statistical models and GANs to diffusion models with increasingly flexible controls. For subject-specific facial generation, prior approaches still face limits in identity control, motivating Arc2Face’s use of identity features without text conditioning.

2.1 Generative Models for the Human Face

  • Text and spatial controls can manipulate images, but the reviewed approaches cannot accurately generate a person’s identity under arbitrary conditions.ControlNet is described as providing accurate spatial control of text-to-image models.Source: §2.1 · p. 4

2.2 Subject-Specific Facial Generation

  • Prior subject-conditioned methods use fine-tuning, encoder-based approaches, or CLIP and identity features, but available inversion methods rely on low-resolution or limited high-quality data.DreamBooth learns a subject-specific identifier from several images, while HyperDreamBooth [65] reduces optimization time using a single image.Source: §2.2 · pp. 4–5
  • Arc2Face instead conditions generation solely on robust identity features, addressing the identity-retention deficiency associated with combining subject embeddings and text.The overview describes projecting ArcFace embeddings through a frozen pseudo-prompt for cross-attention, then training without text annotations.Source: §2.2 · p. 5, Figure 2 · p. 5

3 Method

Arc2Face adapts Stable Diffusion to generate faces from ArcFace identity features; base-model experiments show that dataset scale and identity variation matter for identity retention and diverse same-ID generations.

3.1 Preliminaries

  • ArcFace maps aligned face crops to 512-dimensional embeddings, which condition a latent-diffusion generator through cross-attention while its pretrained Stable Diffusion backbone predicts noise.The ArcFace network is used as a frozen feature extractor, and the embedding is projected onto the generator’s cross-attention layers.Source: Face Recognition Features · pp. 5–6, Latent Diffusion Models · p. 6

3.2 Base Models

  • FFHQ-trained models retain identity less well, while WebFace42M—including its 10% subset—is suitable despite low-resolution crops and provides intra-class variation for diverse same-ID generations.The complete WebFace42M contains approximately 42M images; the 10% subset contains approximately 4M images from approximately 200K identities.Source: §3.2 · pp. 6–7, Figure 3 · p. 7

3.3 Arc2Face

  • ArcFace vectors replace the <id> token in a simple prompt, and a fine-tuned encoder maps the resulting sequence into conditioning embeddings for Stable Diffusion.The ArcFace vector is zero-padded to match the CLIP token-embedding dimension.Source: ID-Conditioning · p. 7
  • The authors upsample 21M WebFace42M images from 1M identities to 448×448, then fine-tune on less-constrained FFHQ and CelebA-HQ images for 512×512 FFHQ-aligned output.The restored WebFace42M images retain a tight facial crop; the later fine-tuning uses less-constrained face images.Source: §3.3 · p. 7, Dataset · p. 8

4 Experiments

Arc2Face leads the compared methods in face similarity, diversity, and realism across synthetic and real IDs, and users strongly prefer its identity resemblance to InstantID. The experiments also test synthetic-data recognition, spatial control, ID-conditioning choices, generalization, and implementation.

4.1 ID-Consistent Generation

  • The comparison uses a simple “photo of a person” prompt, 500 synthetic and 400 real IDs, and five generated samples per ID to assess identity, diversity, and quality.Metrics include ArcFace cosine similarity, LPIPS, expression and pose distances, and FID.Source: Prompt Selection · p. 9, Datasets and Metrics · p. 9, Table 1 · p. 10
  • Arc2Face achieves the highest facial similarity, diversity, and realism across both datasets, and users strongly prefer its identity resemblance to InstantID.The similarity evaluation uses synthetic and real IDs; the user study asked 50 people to compare results for 30 randomly selected IDs.Source: Figure 4 · p. 9, Figure 5 · p. 9, Datasets and Metrics · p. 10

4.2 Face Recognition with Synthetic Data

  • Arc2Face surpasses DCFace on all five test datasets, averaging 2.17% improvement in the 0.5M regime and 1.93% in the 1.2M regime.The authors also report especially strong performance against DigiFace and DCFace on CFP-FP, which has large pose variation.Source: §4.2 · pp. 10–11, Table 2 · p. 11

4.3 Pose/Expression Control

  • ControlNet uses rendered face normals under a reference pose and expression to guide synthesis of the source identity.The normals are derived using EMOCA v2 reconstructions and the reference image’s extracted pose and expression.Source: §4.3 · p. 11, Figure 7 · p. 12

4.4 Ablation Studies

  • Replacing the fine-tuned CLIP encoder with a four-layer MLP yields lower face similarity; the authors argue the encoder is a more natural fit for Stable Diffusion’s CLIP conditioning.The comparison keeps the training setting the same.Source: §4.4 · p. 12, Figure 8 · p. 13
  • For ID interpolations between pairs of subjects, Arc2Face generates plausible faces along the trajectory connecting their ArcFace vectors.Source: Figure 10 · p. 13
  • 0.37 average cosine similarity to the closest training image, versus 0.74 to unseen input features, supports the authors’ finding that Arc2Face does not memorize training images.The analysis covers outputs for 500 previously unseen Synth-500 identities.Source: §4.4 · p. 13, Figure 11 · p. 13

4.5 Implementation Details

  • Training starts from Stable Diffusion v1.5, fine-tunes its UNet and CLIP encoder, and proceeds from 21M restored WebFace42M images to higher-resolution FFHQ and CelebA-HQ images.The stages use 448×448 resolution for five epochs, followed by 512×512 for 15 epochs.Source: §4.5 · p. 14

5 Limitations and Impact

Arc2Face’s scope is limited by its face-embedding encoder and datasets, and it generates only one person per image. The authors also identify misuse and demographic effectiveness as concerns in face-generation applications.

5 Limitations and Impact

  • The method is limited by the capacity of its face-embedding encoder and the datasets used, and it generates only one person per image.The authors describe the encoder and datasets as state-of-the-art and say the foundation model can be further fine-tuned to other datasets and modalities.Source: §5 · p. 14
  • The authors restrict training data to facial regions because face-generation technology can be misused to create unoriginal facial images.They call for support for fake-content detection and adherence to ethical responsibilities.Source: §5 · p. 14
  • The authors identify equitable demographic effectiveness as an ongoing concern and note that ID embeddings are not perfect.They contrast their general approach with prior art based on biased celebrity datasets.Source: §5 · p. 14

6 Conclusion

Arc2Face fine-tunes pretrained Stable Diffusion on restored WebFace42M images and conditions generation solely on face-recognition identity embeddings. The authors report realistic, diverse generations with greater identity similarity than existing methods.

6 Conclusion

  • Arc2Face generates realistic images with greater identity similarity than existing methods while preserving output diversity.The conclusion describes the model as an ArcFace-to-Image model that deliberately disregards text information.Source: §6 · p. 14
  • The model fine-tunes pretrained Stable Diffusion on carefully restored WebFace42M images, with million-scale face-recognition datasets required for effective identity-conditioned training.Source: §6 · p. 14
  • Arc2Face uses identity embeddings alone as its conditioning signal, rather than text information.Source: §6 · p. 14

A Effect of Training Dataset Size

The appendix contrasts training on FFHQ with WebFace42M, reporting substantial identity distortion in FFHQ-trained samples. It states that large datasets with multiple images per identity are essential for this task.

A Effect of Training Dataset Size

B Using Multiple Images per ID

Averaging ArcFace embeddings from multiple images of one person can slightly increase generated identity resemblance, and increasing the number of averaged vectors improves sample fidelity. The figure set also describes an ID-similarity comparison across 400 input identities.

B Using Multiple Images per ID

  • Averaging multiple identity embeddings can slightly improve resemblance, with increased vector counts associated with improved sample fidelity.The paper suggests averaging helps filter identity-irrelevant expression or appearance information encoded in embeddings.Source: B Using Multiple Images per ID · p. 23, Figure 13 · p. 23
  • Arc2Face provides state-of-the-art identity retention using only the ArcFace embedding from a single input image.Source: B Using Multiple Images per ID · p. 23
  • Figure 14 compares ID similarity for generated images from different models across 400 input identities, with five generations per identity.Source: Figure 14 · p. 23

C Retrieving the Closest Training Samples

The authors report that Arc2Face avoids reproducing its training identities, unlike image memorization observed in text-to-image models. They examine generated samples against their closest training images by identity similarity.

C Retrieving the Closest Training Samples

D Additional Qualitative Results

The appendix extends comparisons with related methods and presents additional samples, including outputs guided by 3DMMs through ControlNet.

D Additional Qualitative Results

  • Using AdaFace for ID-embedding extraction, similarity distributions closely resemble the main-paper results and confirm Arc2Face's superior ID retention.The comparison follows the main paper's evaluation protocol for its real-image dataset.Source: D Additional Qualitative Results · p. 23
  • Figures 17–18 show multiple samples conditioned on input IDs, while Figures 19–20 show additional results conditioned on 3DMMs using ControlNet.Source: D Additional Qualitative Results · p. 23, Figure 17 · p. 26, Figure 18 · p. 27
  • Figure 15 pairs generated faces with their closest training samples by ArcFace similarity and crops generated faces to the training face template.The similarity is displayed at the bottom left.Source: Figure 15 · p. 24
  • Text-based methods are compared using the abstract prompt “photo of a person” to focus on their ID-conditioning ability.Source: Figure 16 · p. 25

Related papers

Loading 2403.11641v2…