PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Late, Guiding Early: Timestep-Decoupled Semantic Guidance for Fair Face Generation

Subir Kumar Parida, Rajbabu Velmurugan, Ketan Kotwal, R. S. Sengar, Swati Hiremath

arXiv:2608.25862v1cs.CV

TL;DR

Demographic imbalance in synthetic face generation can propagate to downstream face recognition, while existing fairness methods may require retraining, architectural changes, or repeated guidance. SBP learns semantic boundaries from late-stage latents and applies them once at the initial noisy latent, improving fairness while preserving perceptual quality without retraining or external balanced data.

  • Problem

    Diffusion-generated synthetic faces can inherit demographic imbalance that propagates disparities to downstream face recognition systems.

  • Method

    SBP learns demographic semantic boundaries from late-stage latent representations and applies a single guidance intervention at the initial noisy latent during inference.

  • Results

    SBP consistently improves demographic fairness while preserving perceptual image quality across evaluated gender, race, and age settings.

  • Takeaways & Limitations

    SBP offers a practical lightweight approach for balanced synthetic facial data without modifying or retraining existing latent diffusion models.

  • Takeaways & Limitations

    Supporting additional demographic attributes requires multiple SBPs, and boundary learning depends on external attribute classifiers whose label errors or bias can affect boundaries.

Abstract

from arXiv · show

Demographic imbalance in synthetic face generation can propagate to downstream face recognition systems, making fairness an important consideration when diffusion models are used for data generation. Existing fairness-aware generation approaches often require model retraining, architectural modifications, or repeated guidance throughout the reverse diffusion process. In this work, we introduce Semantic Boundary Predictor (SBP), an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising. Our approach is motivated by the observation that latent representations at different diffusion timesteps play distinct semantic roles: late-stage latents provide stronger demographic separability, whereas early-stage latents offer greater flexibility for semantic intervention. SBP leverages this timestep decoupling by learning linear semantic boundaries from late-stage latent representations while applying them only once at the initial noisy latent, allowing the remainder of the reverse denoising process to proceed unchanged. The method requires neither retraining nor fine-tuning of the underlying Latent Diffusion Model and operates without external balanced datasets. Experiments on CelebA-HQ demonstrate substantial improvements in demographic fairness, reducing fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race, while maintaining perceptual image quality across demographic groups. Owing to its one-shot inference strategy and model-agnostic design, SBP introduces only a small computational overhead and can be readily integrated with existing pre-trained latent diffusion models.

1 Introduction

Synthetic face data can inherit demographic imbalance that harms fairness in downstream recognition systems. SBP addresses this with one-shot, timestep-decoupled inference-time guidance that requires neither model retraining nor external balanced data.

  • Motivation: Synthetic face datasets are scalable and privacy-preserving, but diffusion-generated faces can inherit demographic biases from training data.These datasets are used for downstream tasks including face recognition, making demographic balance relevant to responsible synthetic-data pipelines.
  • Limitations of prior approaches: Many existing debiasing methods require diffusion-model retraining, external balanced datasets, or architectural modifications.Inference-time guidance avoids retraining but commonly relies on repeated corrections throughout reverse denoising.
  • Proposed framework: SBP learns linear demographic boundaries from late-stage latent representations and applies guidance once during early denoising, leaving the remaining trajectory unchanged.The design exploits stronger demographic separability at later stages and greater intervention flexibility at earlier stages.
  • Practical design: SBP operates solely on samples generated by the biased diffusion model and therefore does not require external datasets, retraining, or fine-tuning.The framework is presented as model-agnostic and inference-time.
  • Results: Experiments show improved demographic fairness while maintaining perceptually high-quality facial generation across over-represented and underrepresented groups.The stated evaluation emphasizes both fairness and image quality rather than fairness alone.

2 Related Work

Prior work balances demographic generation through retraining-based methods or training-free inference-time guidance. SBP differs by removing repeated per-step corrections and using timestep-decoupled semantic guidance for lower-cost fair generation.

  • Training-based approaches: Training-based approaches retrain diffusion models or add trainable components to learn semantic latent representations for controlled generation.Their additional training requirements limit compatibility and increase computational demands.
  • Training-free approaches: Training-free methods perform post hoc fairness interventions during inference without altering pre-trained diffusion models.Examples include latent-distribution modeling, image-space fairness gradients, and H-space attribute guidance.
  • Training-free approaches: Unbiased-Diff repeatedly reconstructs clean images and computes fairness gradients at every denoising step, incurring substantial computational overhead.Its repeated image-space corrections distinguish it from one-shot guidance strategies.
  • Timestep analysis: Gender latent distributions overlap substantially early in denoising but separate progressively at later stages, yielding more discriminative decision boundaries.This timestep-dependent separation motivates learning semantic boundaries from late-stage representations.
  • SBP distinction: SBP removes repeated per-step corrections, thereby achieving fair demographic data generation with lower computational cost.Its distinction from related inference-time distribution guidance lies in where and when guidance is applied.

3 Proposed Approach

SBP decouples semantic-boundary learning from demographic guidance across diffusion timesteps: boundaries are learned from discriminative late-stage latents, then applied once during early denoising. The framework constructs attribute-specific latent datasets from frozen-LDM generations and uses lightweight linear predictors for inference-time guidance.

  • Why Timestep-Decoupled Guidance: Early denoising latents are more controllable for demographic manipulation, whereas later latents become more semantically discriminative for boundary estimation.The method is motivated by timestep-dependent semantic roles and the trade-off between intervention flexibility and separability.
  • Learning Semantic Boundaries: The pipeline generates synthetic faces with a frozen LDM, predicts gender, race, and age labels, pairs labels with latent representations, and builds attribute-specific training datasets.An independent predictor is trained for each demographic attribute.
  • Why Timestep-Decoupled Guidance: SBP transfers semantic boundaries learned from late-stage latent representations to early denoising without retraining or fine-tuning the pre-trained diffusion model.This separates reliable boundary learning from the timestep at which guidance is applied.
  • Evaluation Questions: The framework evaluates fairness improvement, multi-attribute generalization, perceptual quality preservation, and the computational cost of one-shot guidance.These four research questions define the intended evaluation scope.
  • Learning Semantic Boundaries: Each SBP is a lightweight linear classifier whose normal vector represents the direction of maximum semantic discrimination and separates demographic classes in latent space.The decision function uses a normal vector b and bias term c.
  • Timestep-Decoupled Guidance: During inference, SBP applies one latent-space perturbation to the initial noisy batch, after which standard reverse diffusion proceeds unchanged and the final latents are decoded.The guidance strength δ controls perturbation magnitude, while its sign selects the target demographic class; δ = 0 recovers the original LDM sampling process.

4 Experiments

Experiments show that SBP learns demographic boundaries more reliably from late-stage latents and applies them to improve fairness across gender, race, age, and joint attributes while preserving perceptual quality.

  • Timestep-Dependent Semantic Structure: Latent representations become increasingly linearly separable during denoising, with demographic overlap decreasing and decision boundaries becoming more discriminative at later stages.These observations support learning semantic boundaries from late-stage representations.
  • Gender Generation: 98% lower fairness disparity on CelebA-HQ gender generation accompanies an FID improvement from 34.66 to 31.54.SBP reduces fairness disparity from 0.051 to 0.001 and achieves the same FID pair relative to the corresponding comparison.
  • Race Generation: 95% lower fairness disparity is achieved for binary race, while four-class race yields a 15% reduction and preserves image quality.The four-class setting is more difficult because demographic overlap increases with additional groups.
  • Cross-Attribute Results: SBP improves demographic balance across gender, binary race, four-class race, and age while preserving perceptual quality.For four-class race, SBP remains better than the original LDM and Balancing Act despite increased semantic overlap.
  • Multi-Attribute Generation: Joint gender–race generation increases SBP’s FD from 0.001 for gender and 0.009 for binary race to 0.055, reflecting the difficulty of satisfying multiple constraints simultaneously.Despite the increase, SBP maintains a more balanced distribution than the original sampling process and existing methods.

5 Limitations

SBP has several scope and dependency limitations. Extending it to more attributes requires additional predictors, and its learned boundaries depend on external classifier labels; intermediate timesteps remain unexamined.

  • Supporting additional demographic attributes requires training and maintaining separate SBPs for each attribute.
  • Boundary learning depends on external attribute classifiers, so label errors or demographic bias can affect the learned semantic boundaries.
  • The experiments learn boundaries at the final denoising timestep and apply them at the initial noisy latent, leaving intermediate timesteps unexamined.
  • Future work will test whether intermediate-timestep boundaries improve transferability or the fairness–image-quality balance, and whether guidance extends to subtler semantics.

6 Conclusion

The paper concludes that SBP uses timestep-decoupled, one-shot guidance to improve fairness in latent diffusion face generation without altering the remaining denoising process. Experiments across demographic attributes report improved fairness while preserving perceptual quality.

  • SBP learns semantic boundaries from late-stage latents and applies one intervention to the initial noisy latent before unchanged reverse denoising.
  • Experiments on gender, race, and age show consistent fairness improvements while preserving perceptual image quality.
  • Diffusion models generate images by progressively adding and removing Gaussian noise, while latent diffusion performs this process in a compressed latent space.
  • The latent diffusion formulation encodes images into latents, denoises in latent space, and decodes the final latent representation into an image.

B Intuition for Timestep-Specific Semantic Roles

The paper frames timestep selection as a trade-off between demographic separability and intervention compatibility. Late stages retain discriminative semantics but resist controlled changes, whereas earlier stages permit broader transformations.

  • Reverse denoising preferentially reconstructs perceptual information such as texture and fine detail, which may not align with demographic semantics.
  • Late-stage latents can retain demographic information, but nonlinear entanglement with perceptual detail may reduce their suitability for direct intervention.
  • The analysis treats timestep selection as a trade-off between semantic accessibility and interference with perceptual structure.
  • Semantic boundaries learned from late-stage latents can guide earlier latents because latent modifications propagate through subsequent denoising.
  • The discussion does not claim global linearity of demographic attributes or perfect semantic disentanglement.
  • Figure 7 contrasts fine-grained attribute changes at t = 0 with coarser, visually consistent transformations at t = T.

C Timestep-Dependent Semantic Structure in LDMs

Experiments compare semantic boundary learning and intervention at the two extreme denoising stages. The results motivate learning boundaries late, intervening early, and relying on preserved coarse semantic alignment across timesteps.

  • Semantic Boundary Estimation at Late Denoising Stages: At t = 0, latent codes encode well-formed facial semantics, enabling accurate linear decision boundaries for demographic features.
  • Semantic Boundary Estimation at Late Denoising Stages: Manipulating t = 0 latents produces limited global changes because late denoising focuses on refining complex details, especially around lips and eyes.
  • Early-Stage Latent Intervention: At t = T, latent representations have lower SBP accuracy but perturbations yield coherent, more noticeable attribute changes through coarse facial geometry.
  • The method therefore separates semantic-boundary acquisition from semantic intervention across diffusion timesteps.
  • SBP assumes coarse semantic alignment persists across timesteps, allowing initialization changes to influence subsequent denoising.

D.1 The Impact of Sample Size Generation on Bias

Fairness evaluation remains stable across generated-sample sizes, supporting the use of 10,000 samples in subsequent experiments.

  • Measured gender fairness disparity remains largely unchanged when evaluated with 2,000, 5,000, or 10,000 generated images.This indicates that the evaluation is stable across the tested sample sizes.
  • The experiments use 10,000 generated samples based on the observed stability across sample sizes.
  • The sample-size analysis tests whether measured demographic bias could be an artifact of evaluation sample size.

D.2 Qualitative Analysis of Semantic Guidance

Traversing the learned gender semantic boundary enables smooth, controllable transitions between male and female faces while preserving image quality. A single inference-time guidance step performs this demographic manipulation.

  • The guidance strength δ controls movement along the learned gender semantic boundary.
  • Adding or subtracting the learned latent vector progressively transforms male faces into female faces or produces the reverse transition.The vector is scaled by δ to control the direction and extent of the transformation.
  • The smooth progression indicates that the learned boundary captures a meaningful semantic direction in latent space.
  • The learned boundary enables controllable demographic manipulation through a single inference-time guidance step while preserving image quality.

E Qualitative Results

Qualitative results show that SBP rebalances gender and racial demographic representation in CelebA-HQ samples while preserving perceptual image quality. The baseline LDM exhibits substantial female and White-face overrepresentation, which SBP reduces by increasing underrepresented groups.

  • Gender: SBP substantially rebalances gender representation by increasing underrepresented groups while preserving perceptual image quality.
  • Race: SBP substantially rebalances racial representation by increasing underrepresented groups while preserving perceptual image quality.
  • Gender: The baseline LDM generates approximately 66% female and 34% male faces, demonstrating disproportionate female representation.
  • Gender: SBP produces approximately 51% female and 49% male faces, with higher perceptual quality reflected by the FID score.
  • Race: The baseline LDM generates 91% White, 4% Black, 3% Asian, and 2% Indian faces, reflecting racial underrepresentation in the original dataset.
  • Race: SBP changes the generated racial distribution to 58% White, 16% Black, 20% Asian, and 6% Indian faces while maintaining high perceptual quality.
Loading 2608.25862v1…