Debias Anything: Fairness with Diversity without Supervision in Diffusion Models

• Existing methods face restrictions including attribute supervision, generator finetuning, limited attribute adaptability, or reliance on text-to-image models, while diversity-restoring methods do not target specific attribute fairness.Source: §1 · p. 2
• Debias Anything uses short text descriptions of attribute values to define fairness directions, and a model-specific adapter maps frozen-generator activations to image-text embeddings for guidance.Source: §1 · p. 2
• A second term computed from the same adapter pushes images toward rare regions emptied by fairness guidance, helping preserve diversity.Source: §1 · p. 2
• On CelebA-HQ, Debias Anything reaches the best or near-best fairness score on every considered attribute with improved image quality; on Stable Diffusion 1.5, it substantially improves fairness across all considered attributes with image quality among the best.Source: §1 · pp. 2–3
• The method compares favorably with published baselines on unconditional face models and text-to-image generation, and can target attributes for which no label exists.Source: §1 · pp. 2–3

• Prior methods variously require attribute supervision, restrict target proportions or text-conditioned models, or retrain the base model; Debias Anything avoids these assumptions.Source: §2 · p. 3
• Balancing Act uses labeled U-Net activations, while Score Guidance uses attribute exemplars or a less effective unsupervised text version; both score in pixel space and assign values greedily early in sampling.Source: §2 · p. 3
• Diversity methods recover low-density regions or repel samples but do not constrain sensitive attributes; Debias Anything transposes SGMS rarity scoring into encoder space and combines it with fairness guidance.Source: §2 · p. 3
• Other attribute-inference approaches use noisy-image classifiers, off-the-shelf models on denoised images, or task-labeled feature heads, whereas Debias Anything applies an encoder to clean, non-blurry images without supervision.Source: §2 · p. 3
• Unsupervised self-debiasing discovers clusters per prompt and sets targets over those clusters, while Debias Anything names attributes with sentences and can set any proportion using an adapter trained once per generator.Source: §2 · p. 3

• The paper focuses on diffusion models in the EDM parameterization, which can also express DDPM and VP/VE SDE models; the method can be adapted to standard flow matching.Source: §3 · p. 3
• A noisy image is formed by adding noise to a data image, and the pretrained denoiser approximates the posterior mean of the clean image given that noisy state.Source: §3 · p. 3
• Figure 2 depicts fixed prompt directions and a per-step adapter mapping each image’s inner denoiser activations to representations used to compute fairness and diversity terms.Source: Figure 2 · p. 4
• Tweedie’s identity relates the denoiser to the score of the noisy-image distribution, and sampling proceeds from the maximum noise level down to zero.Source: §3 · p. 4
• Guidance chooses a scalar function and weight to steer sampling, changing the denoiser; classifier guidance uses a noisy-image class predictor, while classifier-free guidance derives it from a conditional model.Source: §3 · p. 4
• A frozen image-text encoder maps images and text into a shared unit-sphere embedding space where matching pairs are close; the paper uses SigLIP 2.Source: §3 · p. 4

4.1 A CLEAN SIMULATION-FREE SEMANTIC EMBEDDING OF THE NOISY IMAGE

• The adapter maps denoiser activations to an embedding of the clean image from noisy inputs, avoiding simulation and averaging in embedding space.Only the adapter is trained; the denoiser and encoder remain frozen, and no attribute labels are used.Source: §4.1 · p. 5

4.2 FAIRNESS GUIDANCE FROM TEXT PROMPTS

• Text prompts define attribute directions, and a batch-wise assignment guides each image toward its assigned value while meeting target proportions as quotas allow.The assignment maximizes the batch’s total logit and is solved with the Hungarian algorithm.Source: §4.2 · pp. 5–6

4.3 THE DIVERSITY TERM AND COMBINED GUIDANCE

• The diversity score measures disagreement between two estimates of an image’s embedding, capturing semantic variation rather than pixel changes such as pose, background, or lighting.It is added to the fairness term in the combined guidance.Source: §4.3 · pp. 6–7
• The method applies to unconditional and text-conditional models and composes with other guidance; with classifier-free guidance, the adapter reads only the prompt-conditioned branch.Source: §4.3 · p. 7

5 EXPERIMENTS

• The evaluation covers CelebA, CelebA-HQ, and Stable Diffusion 1.5, with fairness measured by discrepancy and quality assessed using FID and other metrics.Source: §5 · pp. 7–8

5.1 COMPARISON WITH BASELINES

• On CelebA-HQ, fairness-setting discrepancy is lowest for race and eyeglasses and second-best for gender; at matched discrepancy, the diversity term lowers FID on gender and race.The method also targets unlabeled red hair more closely to the target proportion than a classifier trained on labeled images, with lower or comparable FID at comparable discrepancy.Source: Table 2 · p. 8, §5.1 · p. 8
• 0.56–0.75 to 0.20: on Stable Diffusion, discrepancy falls across all three attributes, with FID first on race and second on gender and age.Source: §5.1 · p. 8, Table 3 · p. 9

5.2 ABLATIONS ON CELEBA

• At equal eyeglasses weights, the adapter-read score lowers discrepancy from 0.573 to 0.32 and MIND from 20.51 to 8.9; encoder readout never passes 0.489 and raises MIND.Source: §5.2 · p. 9
• Across gender and eyeglasses ablations, diversity terms improve on fairness alone; ours lowers discrepancy most, while eyeglasses Vendi rises and FID increases by up to 1.2.The method is also resilient to changes in attribute-reference wording.Source: §5.2 · p. 9

6 CONCLUSION

• A generic adapter connects a frozen generator to an image-text embedding space, where batch-level fairness and diversity terms guide attribute proportions described in words.The method targets unconditional and text-to-image diffusion without labels or generator retraining.Source: §6 · p. 9
• Although it uses little attribute information, Debias Anything is better than or comparable to methods that rely on attribute supervision.Source: §6 · p. 9
• Evidence covers human faces, uniform target proportions, and single seeds on CelebA-HQ and Stable Diffusion.The method also depends on the image-text encoder and its biases, and steering one attribute can move others.Source: §6 · p. 9
• The authors propose auditing model bias with the adapter, extending the recipe to video and audio diffusion, and correcting classifier errors inside the guidance loop as future directions.Source: §6 · p. 9
• The method enforces an operator-chosen group distribution, but does not determine which distribution is fair.Setting a proportion to one half can be changed to zero; other methods that control an attribute offer the same lever.Source: ETHICS STATEMENT · p. 10

D ABLATIONS ON CELEBA

• The CelebA ablations test score location, fairness and diversity weights, alternative diversity terms, and diversity terms without fairness; the supplied passages list no outcomes.Source: D Ablations on CelebA · p. 16

A.1 PROOFS OF THEORETICAL RESULTS

• The fairness objective is convex and piecewise linear, and where its maximizing assignment is unique its derivatives yield the guidance expression.Source: §A.1 · pp. 16–17

A.2 BINARY ATTRIBUTES

• For binary attributes, the method ranks a batch along the text-defined direction and assigns the positive pole to the B+ highest-scoring images, meeting the quota exactly.The score’s sign acts as a zero-shot classifier; each image is pushed toward its assigned pole.Source: §A.2 · p. 17

A.3 ALTERNATIVE DIVERSITY TERM

• SigLIPMS compares two estimates of an image: it is small on model modes and large in the tail, but inherits the encoder’s behavior on blurred images.The original diversity term instead inherits adapter errors; the paper compares the terms per attribute.Source: §A.3 · p. 17

A.4 DENSITY BASELINE

• The density baseline measures rarity against 10,000 real-image embeddings using mean distance to κ nearest neighbors, unlike terms that measure posterior spread.Increasing its weight pushes toward low-density regions.Source: §A.4 · p. 17

B.1 GENERATORS

• Experiments use CelebA EDM, CelebA-HQ P2, and Stable Diffusion 1.5, with dataset-specific samplers, step counts, guidance windows, and batch sizes.The Stable Diffusion runs use Euler sampling, 30 steps, classifier-free guidance at scale 7.5, and batches of 100 images from one prompt.Source: §B.1 · p. 18, §B.3 · p. 18

B.2 ATTRIBUTES, PROMPTS AND SAMPLE SIZES

• The evaluated prompts cover binary gender, race, and eyeglasses on CelebA datasets, and gender, age, and race across four occupations on Stable Diffusion.Targets are uniform; CelebA uses five seeds, while CelebA-HQ and Stable Diffusion use one seed per configuration.Source: §B.2 · p. 18

B.4 BASELINES

• CelebA-HQ comparisons rerun Balancing Act and exemplar Score Guidance; Stable Diffusion comparisons include DiffLens and ICM, and CelebA compares diversity terms with SGMS.The Stable Diffusion evaluation reproduces the unguided DiffLens baseline under its setup.Source: §B.4 · pp. 18–19

B.5 METRICS

• Reference sets are balanced or otherwise selected by dataset and task, while Stable Diffusion uses FFHQ as the real reference for FID.CelebA-HQ race references are selected by CLIP similarity because that attribute has no label.Source: §B.5 · p. 19
• Quality and diversity are assessed with FID across datasets, plus CelebA measures including MIND, precision and recall, density and coverage, and Vendi score.Stable Diffusion also uses CLIP-T and CLIP-I, averaged across four occupations.Source: §B.5 · p. 19
• On CelebA and CelebA-HQ, fairness distance uses hard-predicted proportions against the target; Stable Diffusion uses mean softmax distance to the uniform target.Stable Diffusion predictions are not renormalized across its four race categories.Source: §B.5 · p. 19

B.6 PROTOCOLS

• CelebA fairness weights are selected by lowest MIND in a sweep, while CelebA-HQ and Stable Diffusion use specified fixed weights.CelebA-HQ also sets a diversity weight and re-noising schedule for its fairness setting.Source: §B.6 · p. 20

C ADAPTER ARCHITECTURE AND TRAINING

• Training noises clean images at two independent levels, feeds them through the frozen backbone, and regresses adapter outputs to precomputed clean-image embeddings without using sampler trajectories.Training prompts name occupations and templates but no sensitive attributes, so attributes reflect model bias rather than text conditioning.Source: C ADAPTER ARCHITECTURE AND TRAINING · pp. 20–21
• For unconditional EDM and P2, the adapter reads middle-block activations and maps pooled, noise-conditioned features to the 768-dimensional embedding space.The described adapters have 0.79 M and 2.42 M parameters.Source: C ADAPTER ARCHITECTURE AND TRAINING · p. 20
• For text-conditional diffusion, the adapter reads the prompt-conditioned branch and combines middle and decoder activations; reading only the middle block misses eyeglasses.The full adapter uses a six-layer transformer; its logits are read before classifier-free guidance combination and therefore do not depend on guidance scale.Source: C ADAPTER ARCHITECTURE AND TRAINING · p. 20
• Across the guidance window σ ≤ 10, the CelebA adapter matches the Tweedie-estimate encoder on gender and exceeds it on eyeglasses, without decoding cost.Figure 4 reports AUC against CelebA labels on 500 test images per class.Source: Figure 4 · p. 21

D ABLATIONS ON CELEBA

• The CelebA ablations share an EDM backbone, deterministic Heun sampling, and adapter-read h-space fairness guidance; their tested design choices include score location and guidance weights.The supplied passages identify the tested choices but do not report ablation outcomes.Source: D ABLATIONS ON CELEBA · p. 21

D.2 FAIRNESS WEIGHT

• For eyeglasses, quality is best near wfair = 200, while discrepancy continues falling; on gender, MIND rises sharply past about wfair = 5, so best quality and parity require different settings.At wfair = 600, eyeglasses steering also raises gender discrepancy from 0.096 to 0.343, a collateral shift a multi-attribute target would need to control.Source: §D.2 · pp. 22, 24

D.1 WHERE THE SCORE IS COMPUTED: h-SPACE AGAINST PIXEL SPACE

• On eyeglasses, h-space lowers discrepancy from 0.614 to 0.156, while pixel-space stalls at 0.489 and raises MIND and FID; on gender, their MIND difference is within seed variation.The pixel-space readout misses some target attributes even with more than twice the steps; the passage attributes this to evaluating an embedding of noisy, averaged images off the clean-image support. The h-space gradient is also cheaper, stopping at the mid-block.Source: §D.1 · pp. 21–22, Table 5 · p. 23, Figure 7 · p. 23

D.3 DIVERSITY WEIGHT

• In the diversity-weight sweeps, no setting dominates on eyeglasses; on gender, all four terms improve MIND and discrepancy over baseline, with a narrow useful range around each MIND optimum.On eyeglasses, SigLIPMS alone improves MIND, discrepancy, and FID together over its baseline at wdiv between 40 and 80. On gender, MIND turns sharply upward past its optimum; avgKNN reaches 8.34 at wdiv = 40 and SigLIPMS 9.39 at 160.Source: Table 6 · p. 24, §D.3 · pp. 24, 26, Table 7 · p. 25

D.4 DIVERSITY ON TOP OF THE FAIRNESS TERM AT THE SELECTED SETTING

• On gender, all four diversity terms lower MIND from 2.52 to 2.03–2.13 while improving discrepancy, FID, and Vendi; on eyeglasses they improve discrepancy and Vendi but not MIND.For eyeglasses, FID rises from 11.47 to 12.4–12.7 for the two semantic terms; diversity terms add approximately 43% generation-time overhead in the 50,000-image runs.Source: Table 8 · p. 26, §D.4 · pp. 26–27

D.5 DIVERSITY TERMS ALONE

• With precision ≥0.95, avgKNN has the lowest MIND at 62.7; SigLIPMS follows at 71.2, PerturbProj ranks third, and SGMS last at 101.0.SGMS’s larger recall gain is bought with precision, which the selection floor excludes.Source: Table 10 · p. 27, §D.5 · p. 27
• The standalone ranking is reference-favored for avgKNN and does not carry over once fairness guidance is added; the combined comparison favors other terms on key attributes.With fairness guidance, PerturbProj and SigLIPMS reach the lowest discrepancy on both attributes; avgKNN keeps only eyeglasses FID.Source: §D.5 · p. 27
• Hyperparameter behavior differs by term: avgKNN is nearly insensitive to κ, while SGMS and PerturbProj perform markedly worse at σs = 0.2 than at 0.4.SigLIPMS MIND bottoms out at wdiv = 250 and rises beyond it; SGMS’s low-MIND settings can fail the precision floor.Source: §D.5 · p. 27, Figure 8 · p. 28

Qualitative results

• Across shared-seed examples, guidance changes individual images, leaves unnamed content intact, and changes faces more cleanly than a labeled predictor.The section also reports that the wording of the attribute sentences matters little.Source: §D.6 · p. 30

CelebA-HQ against Balancing Act

• At matched discrepancy, Debias Anything produces cleaner faces than Balancing Act while both methods change the same two CelebA-HQ images into men.Balancing Act’s fifth-column face has blurred, uneven skin and a distorted mouth; Debias Anything’s is clean and retains the seed’s hair and lighting.Source: Figure 9 · p. 30, §D.6 · p. 30

Sensitivity to the sentences

• With five gender-prompt pairs at wfair = 20, the same two of six seeds change, while the other four remain untouched.The changed faces differ only in fine details across prompt pairs.Source: §D.6 · p. 30, Figure 10 · p. 31

Stable Diffusion

• For doctor and firefighter prompts, guidance reaches three women in six images for each occupation while retaining clothing, pose, background, and expression.It changes two doctor images and three firefighter images; the woman in the sixth doctor column remains unchanged.Source: §D.6 · p. 30, Figure 11 · p. 32

Residual discrepancy by occupation

• Gender discrepancy is 0.021 for doctors, 0.104 for firefighters, 0.296 for nurses, and 0.378 for receptionists.Guidance moves nurses and receptionists from all women to three-quarters women; the passage notes that the model does not draw male examples of these occupations on its own.Source: E FULL COMPARISON UNDER THE DIFFLENS… · p. 32, Table 15 · p. 33

CLIP scores

• CLIP-T stays within 0.01 of the unguided model for every method and attribute.The comparison reports Stable Diffusion 1.5 under the DiffLens protocol.Source: E FULL COMPARISON UNDER THE DIFFLENS… · p. 32, Table 16 · p. 33
• CLIP-I is 0.84 to 0.85 for Debias Anything, at the level of h-space distribution guidance.The passage places label-located methods at 0.85 to 0.92 and latent-space editors at 0.81 to 0.88.Source: E FULL COMPARISON UNDER THE DIFFLENS… · p. 32, Table 16 · p. 33

Residual discrepancy by attribute

• Age discrepancy is 0.092 for doctors and 0.077 for nurses, versus 0.276 for firefighters and 0.381 for receptionists; race discrepancy is 0.17 to 0.21 across occupations.The passage suggests FairFace’s age bins may contribute a floor, while identifying the race residual as attribute-related rather than occupation-specific.Source: E FULL COMPARISON UNDER THE DIFFLENS… · p. 33