PaperlayerView exact revision on arXiv

Source-linked AI summary

Identity-Conditioned Latent Consistency Distillation for Face Synthesis

Tiago Kienen Chaves, Bernardo Biesseck, David Menotti

arXiv:2608.31053v1cs.CV

TL;DR

Large-scale identity-conditioned face synthesis is costly with iterative diffusion sampling, motivating a faster alternative for synthetic face datasets. The paper distills Arc2Face into a latent Consistency Model using ArcFace identity embeddings, achieving 4.36× faster inference while remaining competitive in FID. The main scope boundary is that the current evaluation emphasizes FID and runtime, and the model is designed for ArcFace embeddings rather than direct text conditioning.

  • Problem

    Iterative diffusion sampling makes large-scale synthetic face dataset generation computationally expensive despite its need for many identities and varied conditions.

  • Method

    The paper adapts latent consistency distillation from Arc2Face’s text-to-image pipeline to embedding-to-face synthesis using ArcFace identity conditioning.

  • Results

    The student runs in 0.4819 seconds per image versus 2.1020 seconds for Arc2Face, a 4.36× speed-up, while remaining competitive across evaluation protocols.

  • Takeaways & Limitations

    Task-specific latent consistency distillation can accelerate Arc2Face while preserving competitive face-image quality for large-scale synthetic generation.

  • Takeaways & Limitations

    Evaluation is centered on FID and runtime, and the embedding-to-face model is limited to ArcFace embeddings without retraining for text embeddings.

Abstract

from arXiv · show

Diffusion models have achieved strong results in high-fidelity image synthesis, but their iterative sampling process makes large-scale generation computationally expensive. This limitation is especially relevant when generating synthetic face datasets for face recognition, where a large number of subjects with many samples in different poses, expressions, ages, etc., are required. In this work, we show that identity-conditioned face synthesis can be performed at a substantially lower computational cost by a latent Consistency Model with few iterations, without compromising image quality. For training, we distill knowledge from the foundation Diffusion Model Arc2Face (teacher) by adapting its original text-to-image pipeline to an embedding-to-face setting, replacing textual prompts with ArcFace identity embeddings. Our distilled model (student) generates identity-conditioned face images with an average inference time of 0.4819 seconds per image, compared with 2.102 seconds for Arc2Face, resulting in a 4.36$\times$ speed-up. Quantitative results, based on FID scores, show that the distilled model remains competitive with Arc2Face across all evaluation protocols. On 100k generated images, it achieves near-parity on CelebA (13.921 vs. 12.928) and outperforms the teacher on WebFace42M (9.317 vs. 9.802). Further evaluations on Synth-500 and AgeDB show a moderate performance gap for the former but comparable results for the latter. These results indicate that Arc2Face can be accelerated through task-specific latent consistency distillation while preserving high image quality for large-scale synthetic face generation. Our proposal is publicly available at https://github.com/UFPR-IPASP-PR/FaceRec-IdentityConsistency.

I. INTRODUCTION

Diffusion-based face synthesis supports large, varied datasets but remains computationally expensive because generation is iterative. The paper applies consistency distillation to Arc2Face, targeting faster identity-conditioned generation while retaining competitive quality.

  • Motivation: Large-scale synthetic face datasets require many images across identities and visual conditions, making diffusion inference cost a practical limitation.Applications include identity-balanced training sets, controlled benchmarks, and identity variations across poses, expressions, and illumination.
  • Background: Consistency Models can generate high-quality samples in one or a few steps, including through distillation from pretrained diffusion models.Latent Consistency Models extend this approach to latent diffusion systems.
  • Gap: Existing face-related acceleration work does not directly address large-scale identity-consistent face generation across varied poses, expressions, and illumination.Prior efforts primarily focus on restoration or prompt matching.
  • Contribution: The proposed model distills Arc2Face into a consistency model for identity-conditioned face synthesis with substantially reduced inference time and competitive FID scores.The method adapts the diffusion teacher’s setting to a few-step consistency model.

II. RELATED WORK AND BACKGROUND

The background frames diffusion and latent diffusion as high-quality but iterative generators, then introduces consistency models as few-step alternatives. Arc2Face provides the identity-conditioned face-generation teacher targeted by this work.

  • Diffusion Models: Diffusion models generate images by reversing a gradual noising process through repeated neural denoiser evaluations.Latent diffusion reduces computation by performing denoising in an autoencoder latent space, but sampling remains iterative.
  • Arc2Face: Arc2Face generates high-resolution face images from face embeddings using an identity-conditioning pathway within a latent diffusion architecture.Its embeddings come from an ArcFace-trained ResNet100 model based on WebFace42M.
  • Consistency Models: Consistency models map noisy samples at different noise levels to a consistent point on the data manifold.They may be trained directly or distilled from pretrained diffusion models.
  • Latent Consistency Models: Latent Consistency Models apply consistency distillation in latent space to enable high-quality generation with very few inference steps.Their objective combines skipping-step training with guided distillation.

D. LCM-LoRA and Adapter-Based Acceleration

The paper contrasts generic adapter acceleration with task-specific embedding-to-face distillation. Its pipeline uses Arc2Face’s identity-conditioned latent trajectories and teacher predictions to train a specialized student.

  • LCM-LoRA and Adapter-Based Acceleration: LCM-LoRA accelerates Stable-Diffusion-like models by training a lightweight low-rank adapter instead of the full model.This reduces trainable parameters and storage overhead.
  • LCM-LoRA and Adapter-Based Acceleration: Adapter-based acceleration may be suboptimal when conditioning and evaluation differ from generic text-to-image generation.Identity-conditioned face synthesis prioritizes identity preservation rather than only prompt-image alignment or visual appeal.
  • Related Face Applications: Prior consistency-based face work includes controllable text-to-face generation and blind face restoration, rather than ArcFace-embedding-conditioned distillation.These objectives differ from fast identity-conditioned synthetic data generation.
  • Task-Specific Distillation: The proposed setting replaces text conditioning with Arc2Face identity conditioning while operating on VAE latents and teacher noise predictions.The teacher predicts noise from a noised latent, conditioning vector, and timestep; guided predictions use a guidance parameter and an unconditional embedding sequence.

A. Consistency Mapping and the Consistency Objective

The student consistency network maps noisy latent states toward their clean latent origin, using boundary-conditioned outputs and explicit guidance-scale conditioning. Training matches teacher trajectories across subsampled diffusion paths.

  • Consistency Mapping: The student consistency network maps a point along an ODE trajectory directly to its origin latent z0.Its inputs are the noisy latent, guidance scale, conditioning vector, and timestep.
  • Consistency Mapping: Analytical boundary functions enforce fθ(z0, w, c, 0) = z0 while a trainable UNet supplies the learned mapping.The guidance scale w is injected through a sinusoidal Fourier embedding into network blocks.
  • Consistency Objective: Trajectory matching samples adjacent indices from a subsampled diffusion-time path to train the consistency objective.The implementation uses k = 1 between adjacent reduced-sequence indices.
  • Consistency Objective: Each reduced-sequence transition can represent a larger jump in the original diffusion timeline, reducing the number of transitions used during training.With N = 50 and T = 1000, each transition skips approximately T/N original timesteps.
  • Consistency Objective: The teacher’s classifier-free-guided output is used in the single-step DDIM estimation for trajectory matching.This estimation provides the teacher-side target for the sampled transition.

B. Distillation Loss with Exponential Moving Average (EMA) Targets

The distillation trains an online student to match a target consistency mapping, while an EMA target network stabilizes optimization. The procedure uses teacher-guided noisy-to-resolved latent pairs and a distance-based loss.

  • Teacher guidance: The teacher provides unconditional and identity-conditioned noise predictions that are combined with a sampled guidance weight.The guidance weight is sampled uniformly between predefined boundaries.
  • Consistency targets: A DDIM solver resolves the noisier latent to the earlier timestep, producing the target input for consistency training.The student receives the noisy latent, guidance embedding, identity condition, and timestep.
  • EMA target network: The method maintains online student parameters θ and EMA target parameters θ−, initialized identically before training.The target network is updated with momentum after each optimization step.
  • Distillation loss: The online student minimizes the distance between its noisy-step mapping and the target network’s resolved-step mapping.The paper considers either an L2 or Huber loss for this distance.
  • EMA target network: After each update, the target parameters follow θ− ← µθ− + (1 − µ)θ until convergence.This momentum update implements the EMA stabilization mechanism.

IV. EXPERIMENTS

The experiments section covers the study’s setup, datasets, training and testing procedures, results, and qualitative analysis.

  • Experimental scope: The study defines an experimental setup for evaluating the proposed model.
  • Datasets: The experiments include training and testing datasets.
  • Analysis: The section reports obtained results and qualitative analysis.

A. Training Data and Cache Precomputation

The experiments use preprocessed WebFace4M data, cached identity embeddings and VAE latents, and compare generated distributions with reference datasets using FID. The evaluation includes large-scale and Arc2Face-protocol settings.

  • Training data: 150k aligned WebFace4M face samples were used because of time constraints.WebFace4M contains approximately 4 million images across 200k identities; images were resized to 512×512 pixels and normalized to [−1, 1].
  • Cache precomputation: The cache stores ArcFace identity embeddings e with shape [N, 512] and VAE latents z0 with shape [N, 4, 64, 64].Precomputation avoids repeatedly loading images, extracting embeddings, and encoding images during training.
  • Evaluation setup: Both models receive R100/ArcFace identity embeddings and generate 512×512 synthetic faces for evaluation.The comparison focuses on computational cost, FID-based distributional quality, and qualitative outputs.
  • FID metric: FID measures the distance between real and synthetic image distributions using 2048-dimensional Inception-V3 pool3 features.Lower FID indicates a generated distribution closer to the reference distribution in feature space.
  • Evaluation protocols: The large-scale setting compares 100k generated images with CelebA and a disjoint 100k-image subset of WebFace42M.A second setting follows the Arc2Face protocol using Synth-500 and AgeDB, with five synthetic images per synthetic identity.

B. Quantitative Results

The distilled model remains close to Arc2Face on most FID evaluations, outperforming it against WebFace42M while showing a larger gap on Synth-500 and a smaller gap on AgeDB.

  • 100k-image evaluation: 13.921 versus 12.928 on CelebA gives the consistency model a slightly higher FID than Arc2Face.The absolute difference is 0.993, approximately 7.68%.
  • 100k-image evaluation: 9.317 versus 9.802 on WebFace42M gives the consistency model a lower FID than Arc2Face.The reported improvement is approximately 4.95%, and the reference domain is closer to face-recognition data.
  • Overall comparison: The FID results indicate no large distributional degradation relative to the diffusion teacher across the evaluated settings.The model remains close to Arc2Face in most settings and outperforms it against WebFace42M.
  • Arc2Face protocol: The consistency model has a larger FID gap than Arc2Face on Synth-500.The result indicates that the student does not fully reproduce the teacher’s distribution in this setting.
  • Arc2Face protocol: 6.9720 versus 6.628 on AgeDB shows a smaller difference from Arc2Face.The result suggests comparable performance on this real-face evaluation.

C. Computational Cost Comparison

The distilled model substantially reduces per-image generation time, making large-scale synthetic face generation more feasible under the same evaluation setup.

  • 0.4819 seconds per image versus 2.102 seconds for Arc2Face yields a 4.36× speed-up.This corresponds to a 77.1% reduction in per-image generation time.
  • Approximately 67 GPU-hours versus 292 GPU-hours are required to generate 500k images at the distilled model’s and teacher’s respective speeds.The estimate assumes the same single-image throughput.

D. Qualitative Results

The distilled consistency model produces recognizable and diverse identity-conditioned faces, while its task-specific design targets identity preservation rather than generic prompt-image alignment.

  • Qualitative Results: The student generates recognizable, diverse faces from ArcFace embeddings without collapsing to a single appearance pattern.Figures 2 and 3 provide the cited visual inspection.
  • Qualitative Results: Figure 3 compares generated faces from Synth-500 and AgeDB using Arc2Face, LCM-LoRA, and the proposed Consistency Model.
  • Analysis: Task-specific distillation from Arc2Face lets the student learn the relationship between ArcFace identity embeddings and generated faces.This contrasts with LCM-LoRA’s primarily universal acceleration design for Stable-Diffusion-like models.
  • Analysis: The full distilled model complements LCM-LoRA, which remains attractive because it uses fewer trainable parameters and is easier to train.

V. CONCLUSION

The distilled latent-consistency model accelerates Arc2Face substantially while retaining competitive FID performance. Evaluation remains centered on runtime and FID, with broader identity-specific and downstream assessments identified for future work.

  • 0.4819 seconds versus 2.1020 seconds per image yields a 4.36× speed-up over the Arc2Face diffusion teacher.The student was trained through latent-consistency distillation in an embedding-to-face setting using ArcFace identity embeddings.
  • FID scores remain competitive with Arc2Face across CelebA, WebFace42M, Synth-500, and AgeDB evaluation protocols.On 100k generated images, the student scores 13.9210 versus 12.9280 on CelebA and 9.3170 versus 9.8020 on WebFace42M; under the Arc2Face protocol, it scores 8.7039 on Synth-500 and 6.9720 on AgeDB.
  • The current evaluation is limited because it centers on FID and runtime rather than identity-specific or downstream face-recognition metrics.Proposed future evaluations include embedding similarity, detection rate, diversity, separability, and recognition performance using generated data.
Loading 2608.31053v1…