PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Fairness Auditing Across Guidance Scales in Text-to-Image Diffusion Models via Causal Abstraction

Nabila Tasfiha Rahman, Rajatsubhra Chakraborty, Depeng Xu, Lu Zhang

arXiv:2609.09486v1cs.LGcs.CV

TL;DR

Fairness auditing of text-to-image diffusion models is computationally expensive because guidance settings require repeated full diffusion rollouts. The paper develops a query-specific causal-abstraction instrument that predicts fairness-relevant distributions under guidance interventions, and reports preserved fairness trends with reduced per-sample inference time across unseen guidance scales.

  • Problem

    Repeatedly auditing guidance settings requires complete diffusion rollouts, while lossy abstractions may fail to preserve intervention-relevant causal behavior.

  • Method

    The paper constructs a high-level causal model over abstract diffusion states, establishes identifiability and preservation conditions, and implements it with a probabilistic transformer.

  • Results

    Across unseen guidance scales, the instrument generally preserved final-attribute distributions and prompt- and scale-dependent fairness trends while significantly reducing per-sample inference time.

  • Takeaways & Limitations

    The instrument supports efficient fairness audits of guidance interventions for standard and fairness-enhanced diffusion models.

  • Takeaways & Limitations

    The learned high-level graph is not generally causally equivalent to the projected graph, and experiments indicate that the abstraction can discard causally relevant information.

Abstract

from arXiv · show

Fairness auditing of text-to-image diffusion models often requires generating large numbers of images across sampling configurations, making comprehensive evaluation computationally expensive. We propose a causal-abstraction-based audit instrument for efficiently evaluating fairness under interventions on the classifier-free guidance scale. Given a fixed prompt and a target feature function, we represent the diffusion process as a low-level structural causal model and construct a corresponding high-level model over abstract denoising states. We characterize the projected causal structure, establish identifiability of the fairness-relevant interventional query, and provide sufficient conditions under which the high-level model preserves this query. A probabilistic transformer implements the high-level model as an amortized predictor of target-feature distributions across guidance scales. Experiments evaluate distributional fidelity, fairness-query accuracy, and computational efficiency. We present two auditing demonstrations: one using standard Stable Diffusion 1.5 and another using StayFair, a fairness-enhanced Stable Diffusion model, to examine their behavior across guidance scales.

1. Introduction

The paper develops a query-specific causal-abstraction instrument to make fairness auditing of text-to-image diffusion models more efficient. It targets guidance-scale interventions while addressing whether lossy abstractions preserve audit-relevant causal queries.

  • Direct fairness audits are expensive because every prompt, guidance setting, and random seed requires a complete diffusion rollout.
  • The proposed instrument models the CFG scale as an intervention and predicts target-feature distributions and fairness gaps with a high-level causal model.A stochastic transformer captures the evolution of low-dimensional semantic attributes across abstracted diffusion states.
  • Experiments assess distribution fidelity, audit-decision fidelity, and computational savings using Stable Diffusion v1.5 and SD1.5 with StayFair.
  • The method introduces a query-specific causal-abstraction methodology for efficient fairness auditing of vision generative models.
  • The paper establishes correctness guarantees for recovering an identifiable audit query despite lossy abstraction and differences between learned and projected causal graphs.

2. Background and Related Work

The background connects demographic bias in diffusion outputs and guidance-scale sensitivity to causal abstraction as a way to preserve audit-relevant interventions efficiently. The framework compresses diffusion states into high-level variables while allowing probabilistic mechanisms when exact consistency is unavailable.

  • Fairness in Diffusion Models: Guidance-scale changes can alter demographic representation, so fairness-enhanced methods require evaluation across multiple inference settings.
  • Fairness in Diffusion Models: Direct evaluation across prompts, guidance scales, and random seeds is costly, motivating a learned causal abstraction for estimating demographic distributions and fairness decisions.
  • Principles of Causal Abstraction: A causal abstraction maps low-level diffusion states to high-level representations through a generally many-to-one mapping that discards information judged irrelevant to the target analysis.
  • Principles of Causal Abstraction: The abstraction aims to make low-level and high-level evaluations agree for specified causal queries, rather than requiring consistency for every intervention.
  • Principles of Causal Abstraction: Because lossy mappings may prevent exact consistency, high-level mechanisms can be probabilistic and fidelity can be assessed with distances such as Wasserstein distance.

3. Fairness Auditing of Text-to-Image Diffusion Models

The auditing procedure fixes prompts, intervenes on CFG scales, and evaluates target-feature distributions against a prespecified fairness reference. A validated high-level model estimates fairness gaps and classifies the audit as supported, violated, or inconclusive.

  • CFG scale is treated as an intervenable variable, with generated images produced under do(w) for fixed prompts and sampled randomness.
  • The target feature is obtained by mapping each generated image into [0,1]^d, with the paper using a single feature such as perceived masculine presentation.
  • The fairness gap compares the expected target-feature value at each guidance scale with a prespecified prompt-specific reference.
  • The audit constructs and validates a high-level causal model from submitted diffusion evidence before applying it to the fairness criterion.The evidence includes trajectories, prompts, CFG scales, seeds, sampler configurations, and model provenance.
  • The audit supports fairness when the upper interval endpoint is at most the tolerance, detects a violation when the lower endpoint exceeds it, and is otherwise inconclusive.

4. Causal-Abstraction Audit Instrument

The audit instrument abstracts diffusion trajectories into a high-level causal model whose CFG intervention and target-feature query remain identifiable despite lossy mappings. A partially projected causal graph captures induced dependencies, while sufficient distributional agreement lets the high-level model recover the low-level audit query.

  • 4. Causal-Abstraction Audit Instrument: The instrument constructs a high-level causal model from abstracted diffusion trajectories to evaluate a prespecified fairness query without repeatedly executing the low-level diffusion model.It requires preserving the relevant post-interventional target-feature distribution, identifying that query, and estimating it efficiently after abstraction.
  • 4.1. Query-Specific Causal Abstraction: The abstraction retains CFG scale and maps diffusion states, initial noise, and final images to high-level variables, including the target feature.The variable-wise mapping defines high-level variables A_T through A_0 from corresponding low-level states and preserves W as the intervention variable.
  • 4.1. Query-Specific Causal Abstraction: The high-level query represents the low-level probability that the target feature is at most a threshold under prompt, initial-noise, and do(w) conditions.The mapped query conditions on the abstract initial state and intervenes on the retained guidance scale.
  • 4.2. High-Level Causal Structure: The partially projected C-DAG adds directed and bidirected dependencies to account for information discarded by potentially AIC-violating state mappings.The construction applies projection rules iteratively, representing discarded information as latent dependence and avoiding invalid conditional-independence constraints.
  • 4.2. High-Level Causal Structure: The high-level audit query is identifiable from the observational high-level distribution, and a compatible model reproducing that distribution is consistent with the low-level query.The stated corollary applies when the high-level SCM is compatible with the partially projected graph and matches the induced observational distribution.
  • 4.3. Transformer Neural Causal Model: The transformer NCM may differ causally from the partially projected graph, yet its learned query converges to the high-level causal query when its joint distribution matches the observational target.Its complete directed ordering has no additional conditional-independence restrictions among high-level states, and the excess negative-likelihood result supports convergence.

5. Evaluation of Causal Abstraction Fidelity

The evaluation tests whether a transformer-based high-level abstraction preserves final-attribute distributions at unseen guidance scales despite lossy feature mapping. It also finds empirical evidence that the abstraction discards causally relevant information.

  • High-level model: The instrument uses a sliding-window causal transformer that receives attribute score, scheduler timestep, and CFG scale as inputs.It is trained with Gaussian negative log-likelihood and rolled out autoregressively.
  • Experimental setup: Evaluation uses three gender-neutral prompts, six training CFG scales, eight unseen CFG scales, and 200 held-out seeds.The reference scale is wref = 8, and subsequent experiments use window size L = 25.
  • Distributional fidelity: The high-level model generally preserves final-attribute distributions at CFG scales not observed during training.Fidelity is evaluated with Wasserstein distance against a bootstrap low-level reference.
  • AIC assessment: The abstraction violates the AIC because trajectories with nearly identical intermediate attributes can retain widely different earlier and final attributes.This dispersion remains substantial even as the matched attributes at t = 35 approach equality.

6. Fairness Audit Demonstrations

The demonstrations compare standard SD1.5 with SD1.5+StayFair across guidance scales using low-level and abstract-model fairness estimates. The abstract instrument also enables auditing with substantially lower per-sample runtime.

  • Models and audit setup: The demonstrations use standard Stable Diffusion v1.5 and a StayFair-equipped version with the underlying diffusion model held fixed.StayFair changes classifier-free guidance through a prompt-dependent null-embedding adjustment.
  • Decision fidelity: Figure 3 uses a horizontal line at 0.5 as the demographic-parity reference.Solid and dashed curves distinguish low-level and abstract-model estimates for each model.
  • Decision fidelity: The high-level instrument reproduces the prompt- and scale-dependent trends of low-level fairness outputs for standard SD1.5 and SD1.5+StayFair.Figure 3 compares expected masculine scores across CFG scales, with solid curves for low-level estimates and dashed curves for abstract-model estimates.
  • Computational savings: 0.259 seconds per sample yields a 19.4× inference speedup and a 94.9% runtime reduction relative to the low-level model.The low-level runtime is 5.035 seconds per sample, while the high-level runtime starts from the initial attribute and generates the complete abstract trajectory.

7. Conclusions

The paper concludes that query-specific causal abstraction can make fairness auditing of text-to-image diffusion models more efficient. Across unseen guidance scales, the instrument generally preserves distributions, reproduces fairness trends, and reduces inference time.

  • Main conclusions: Across unseen guidance scales, the instrument generally preserves final-attribute distributions and reproduces prompt- and scale-dependent fairness trends.The conclusion summarizes distributional and fairness-audit fidelity across the evaluated settings.
  • Contribution: The paper presents a query-specific causal-abstraction instrument for efficient fairness auditing of text-to-image diffusion models.Its analysis targets an identifiable audit query despite lossy abstraction and differences between learned and projected causal graphs.
  • Main conclusions: The instrument achieves a significant reduction in per-sample inference time.The conclusion identifies computational efficiency as a central empirical outcome.
  • Implications and future work: The demonstrations show that auditing guidance settings and prompt formulations is important.The paper proposes future extensions to broader prompt distributions, multiple attributes, alternative high-level SCMs, and additional generative-model families.

A. Proof of Proposition 1

The proof establishes that the abstraction is structurally constructive, identifies the guidance intervention from observational information, and connects high-level query fidelity to equality of the fairness-relevant expectation.

  • Audit-query preservation: Equality of target-feature distributions implies equality of their expectations and therefore the same fairness quantity ρ(w, c).The proof uses equality of the relevant distributions to establish equality of the audit outcome.
  • Constructive abstraction: The abstraction partitions low-level variables and values into clusters that correspond bijectively to high-level variables and values.The mapping acts separately on each cluster, making τ a constructive abstraction function.
  • Intervention identification: The intervention do(w) can be replaced by conditioning on W = w because W has no parents or incident bidirected edges.The empty set satisfies the backdoor criterion for the guidance variable.
  • Query identification: Observational τ-consistency implies consistency for the audit query when the query is identifiable from the projected graph and abstract observational distribution.The proof invokes dual abstract identification to recover Qw,c from the high-level representation.
  • Transformer intervention semantics: The transformer and high-level SCM represent conditioning on the intervenable root W equivalently to intervention do(w).Their complete-prefix factorizations parameterize the conditional distribution given w.

F. Implementation Details

The instrument uses a causal Transformer to model gender-attribute evolution across reverse diffusion and tests autoregressive generalization to unseen CFG scales. A window of L = 25 is selected as a cross-prompt compromise because fidelity stabilizes for larger windows without a monotonic window-size trend.

  • Model implementation: The causal Transformer predicts gender-attribute evolution over 50 reverse-diffusion steps using current gender score, timestep, and CFG value as inputs.Its architecture uses five Transformer blocks, 512 hidden dimensions, eight attention heads, and learned positional embeddings.
  • Evaluation protocol: Testing uses held-out seeds and unseen CFG values, autoregressively sampling residuals to construct high-level trajectories and comparing final scores with Stable Diffusion outputs.Training uses CFG values w ∈ {0, 1, 3, 5, 7, 9}, while testing includes w ∈ {2, 4, 6, 8, 10, 12, 15, 20}.
  • Window-size selection: L controls how many preceding abstract diffusion states inform each prediction, trading longer-range dependency capture against computational cost.The study compares L ∈ {1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50} at wref = 8 using Wasserstein-1 distance.
  • Window-size selection: L = 25 is selected as a cross-prompt compromise because fidelity becomes comparatively stable for L ≥25 and this window performs near-minimally for the lawyer prompt.The selection remains within the stable range observed for the librarian and scientist prompts.

H. Empirical Assessment of AIC Violation

The assessment tests whether nearly identical abstract states retain equivalent downstream effects, as required by the Abstract Invariance Condition. Matched states can diverge substantially immediately and at the final output, providing empirical evidence that the abstraction violates AIC.

  • AIC criterion: AIC requires low-level states mapped to the same high-level value to have equivalent causal effects on downstream high-level variables.The test therefore examines whether approximate equality under the abstraction predicts downstream equivalence.
  • Pair construction: The analysis forms held-out trajectory pairs at t = 35 and w = 8 whose abstract attribute values differ by less than 0.005, then compares their downstream divergences.The comparisons include both the next-step and final-outcome differences.
  • Empirical finding: Nearly identical abstract states at t = 35 can diverge substantially at both the next denoising step and the final attribute.One-step differences can approach 0.1, while final differences span nearly the entire attribute range.
  • Interpretation: Proximity in A35 does not ensure proximity in immediate or terminal downstream attributes, indicating that τ35 discards information relevant to later attribute evolution.The observed dispersion persists as the abstract-state distance approaches zero.

I. Additional Experimental Results

Additional experiments compare complete final-attribute distributions from low-level SD1.5 with the high-level abstraction across prompts and CFG scales. The abstraction generally preserves prompt-specific distribution shapes and guidance-dependent changes, while Figure 4 examines window-size fidelity and Figure 5 examines downstream divergence.

  • Distributional comparisons: The high-level model generally preserves the prompt-specific shapes and guidance-dependent changes of low-level attribute distributions.Figures 6–8 compare final perceived-masculine score distributions for lawyer, librarian, and scientist prompts across CFG scales using Wasserstein-1 distances.
  • Window-size fidelity: Figure 4 reports mean low–high Wasserstein-1 distance across transformer window sizes at wref = 8, with shaded regions showing one standard deviation.Lower Wasserstein-1 values indicate greater distributional fidelity.
  • Downstream divergence: Figure 5 examines downstream divergence for matched held-out trajectory pairs at w = 8, complementing the distributional comparisons with a test of abstract-state invariance.The paired trajectories have nearly identical abstract values but can show substantial next-step and final differences.
Loading 2609.09486v1…