PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatially Gated Demographic Steering for Debiasing Multi-Person Text-to-Image Generation

Venkatesh Thirugnana Sambandham, Torsten Schon

arXiv:2610.29651v1cs.CV

TL;DR

Text-to-image models reproduce demographic stereotypes, and existing scene-level steering cannot independently diversify people in a multi-person image. SGS extracts person masks from diffusion-transformer attention features and applies separate demographic steering within each mask. It raises intersectional entropy on FLUX.1-Dev and SD3.5-Large while recovering near-best aggregate balance and preserving semantic coherence.

  • Problem

    Existing mitigation applies one demographic correction to a whole scene, limiting independent demographic control in multi-person generation.

    Source: §1 · pp. 1–2

  • Method

    SGS extracts per-person masks from diffusion-transformer attention features and applies a distinct demographic steering vector within each mask.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    SGS raises intersectional entropy from 0.524 to 0.794 on FLUX.1-Dev and from 0.549 to 0.832 on SD3.5-Large, while recovering near-best aggregate balance and preserving semantic coherence.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Per-instance spatial steering breaks the trade-off between intra-image diversity and aggregate balance across two architecturally distinct diffusion transformers.

    Source: §1 · p. 2, §7 · p. 8

    SGS applies steering regardless of prompt intent, so it may redistribute demographics even when a prompt explicitly constrains them.

    Source: §6 · p. 8

Abstract

from arXiv · show

Text-to-Image models reproduce societal stereotypes, and existing mitigation applies one demographic correction to the whole scene. In multi-person scene generation this creates a trade-off: global steering improves aggregate balance only by homogenizing the people within an image. Steering-based methods like Embedding Arithmetic drive the intra-image diversity below the unsteered baseline. Text- and latent-projection approaches shift the aggregate skew with only marginal gains in intra-image diversity. We present Spatially Gated Steering (SGS), which steers the demographics of each person independently by extracting per-person masks from the Diffusion Transformer's (DiT) internal attention maps at a critical denoising timestep and applying a distinct demographic vector within each mask for the remaining steps. SGS needs no fine-tuning, inversion or layout specifications, and uses a lightweight segmentor over frozen DiT attention features to localize subjects at early timesteps. On skewed occupational prompts, SGS reaches high intra-image racial and gender diversity while keeping aggregate balance competitive and image quality near baseline, a result we validate on two architecturally distinct diffusion transformers, FLUX.1-Dev and SD3.5-Large. Codebase and model weights are available at https://github.com/cvims/spatially-gated-steering

1. Introduction

Text-to-image models reproduce demographic stereotypes, while scene-level mitigation applies the same correction to every person. SGS instead localizes people and steers them independently, targeting intra-image diversity alongside aggregate balance.

1. Introduction

  • Existing global mitigation can shift aggregate outputs but gives every person in a scene the same demographic vector, failing to diversify individuals independently.The paper notes that this multi-person problem is distinct from single-subject evaluation.Source: §1 · p. 1
  • SGS steers each detected person independently within a spatial mask, addressing the scene-level limitation of applying one demographic correction to everyone.The paper frames the goal as achieving high intra-image diversity with competitive aggregate balance, without fine-tuning, flow inversion, or layout input.Source: §1 · pp. 1–2
  • SGS identifies people during intermediate denoising and steers their demographics while maintaining concept coherence.The approach uses a lightweight detector to generate instance masks.Source: §1 · p. 2
  • The paper reports that SGS achieves high intra-image diversity with competitive aggregate balance, and benchmarks fairness on both FLUX.1-Dev and SD3.5-Large.The benchmark measures intra-image diversity and aggregate deviation, surfacing the homogenization failure mode of global methods.Source: §1 · p. 2

2. Related Works

Prior work measures and mitigates demographic stereotypes in text-to-image generation, but existing methods generally apply scene-level corrections rather than independently controlling people in multi-person scenes. Related work also explores spatial features and attention control, often with different goals or layout requirements.

2. Related Works

  • Existing bias mitigation includes prompt engineering, embedding manipulation, and fine-tuning, but applies one attribute correction to the whole scene rather than independently steering each person.Park et al.’s Entanglement-Free Attention [30] suppresses background edits with mask-constrained regularization but remains scene-level.Source: §2 · p. 3
  • Multi-person studies document biases missed by single-person audits, while aggregate representation metrics do not provide per-instance control.Wan and Chang report that over 74% of DALL-E 3 images with two contrastive social identities display gender-occupational biases.Source: §2 · p. 3
  • Spatial-control approaches either require user-provided bounding boxes or fine-tune models, and some use face similarity rather than steering specific demographic attributes.Bounded Attention [7] uses predefined boxes; DisCo [3] penalizes intra-image facial similarity.Source: §2 · p. 3
  • Diffusion features encode spatial structure, and prior work has used DiT attention or diffusion attention at selected timesteps for saliency and spatial masks.SGS extends this line of work to demographic fairness in multi-person text-to-image scenes.Source: §2 · p. 3

3. Preliminaries

The preliminaries describe rectified-flow denoising and the MM-DiT architecture used by the paper, then introduce Embedding Arithmetic as a global demographic-steering baseline.

3.1. Rectified Flow Denoising

  • Rectified flow transports samples from Gaussian noise toward the data distribution along approximately straight trajectories, with FLUX initializing from noise and iteratively denoising.The timestep decreases from 1 for pure noise to 0 for a clean image, and the text condition is encoded as c.Source: §3.1 · p. 3

3.2. Multimodal Diffusion Transformer (MM-DiT)

  • MM-DiT patchifies the noisy image latent into image tokens, concatenates text-conditioning tokens, and processes them through transformer layers with joint image-text self-attention and an MLP.FLUX.1-Dev uses 19 double-stream layers, while SD3.5-Large uses 38 joint transformer blocks.Source: §3.2 · pp. 3–4

3.3. Embedding Arithmetic

  • Embedding Arithmetic averages differences between demographic-target and neutral prompt embeddings across professions to form an attribute direction vector.For FLUX, the vector is constructed in pooled CLIP space and shifts the conditioning vector globally at inference.Source: §3.3 · p. 4

4. Methods

SGS uses internal attention features to localize people during denoising, then applies independently steered predictions within cached person masks. The intervention is confined to a steering window, after which standard denoising resumes.

4. Methods

  • SGS intervenes within the standard rectified-flow loop without model fine-tuning, trajectory inversion, or external layout input.It extracts masks from MM-DiT attention features at a critical denoising step and divides generation into pre-steering, steering, and post-steering phases.Source: §4 · p. 4

4.1. Instance Detection

  • Concatenated attention bins 3 and 4 provide the selected features for person-instance segmentation, and the resulting masks are cached for reuse during steering.The DETR-style segmentor predicts instance masks and discards queries below a confidence threshold; it is trained on generated multi-person images with SAM2 pseudo-labels.Source: §4.1 · pp. 4–5

4.2. Per-Instance Steering

  • At each detected person, SGS shifts the conditioning embedding using that instance’s assigned demographic attribute and runs a separate fair forward pass.The attribute vectors are precomputed for gender and racial attributes, and assignments may be overridden to reflect user-specified fairness targets.Source: §4.2 · p. 5
  • SGS spatially accumulates each fair-minus-base prediction through its corresponding cached mask, then returns to base-only denoising outside the steering window.The diversity strength λ scales the masked contribution, and the mask is broadcast over the feature dimension.Source: §4.2 · p. 5

5. Experiments

Across FLUX.1-Dev and SD3.5-Large, SGS achieves high intra-image diversity with competitive aggregate balance, unlike global methods that trade diversity for balance. Human raters strongly preferred SGS for diversity, while prompt matching was mostly tied.

5.3. Instance Segmentation

  • At mIoU 0.664, AP50 0.797, and AP50–95 0.416, DETR outperforms all baselines, peaking near t≈0.7–0.8.Bins 3+4 perform best across metrics, with mIoU 0.631, AP50 0.747, and AP50-95 0.368 averaged across timesteps.Source: Figure 3 · p. 6, §5.3 · pp. 6–7

5.1. Implementation Details

  • SGS is evaluated on FLUX.1-Dev and ported to SD3.5-Large, whose MM-DiT also provides the joint image-text self-attention maps required by the method.The reported configurations use bfloat16; the SD3.5-Large evaluation uses its native 1024×1024 resolution.Source: §5.1 · pp. 5–6

5.2. Evaluation Metrics

  • Hix measures per-image intersectional diversity, DR aggregate demographic skew, and CLIP score and Aes∆ prompt fidelity and aesthetics; race and gender categories are restricted for classifier reliability.DRrace and DRgender aggregate face-level counts across all people in each image.Source: §5.2 · p. 6

5.4. Steering Parameters Ablation

  • The selected steering settings are α=2.0, λ=4.0, and tstart=4, tend=10; earlier starts hurt background preservation and aesthetics, while later starts leave less steering time.The window balances steering accuracy against background preservation and semantic drift.Source: §5.4 · p. 7

5.5. Benchmark Comparison

  • SGS is the only method to achieve the highest Hix by a wide margin while keeping DRrace second only to Embedding Arithmetic on both architectures.Prompt fidelity remains near baseline; the pattern replicates on FLUX.1-Dev and SD3.5-Large.Source: §5.5 · p. 7, Table 1 · p. 8

5.6. Human Evaluation

  • Raters favored SGS for diversity 88–90% against every baseline, while prompt matching was mostly tied except for an 89% preference over EA.The study collected 65 completed sessions and 1,971 judgments.Source: §5.6 · p. 7, Table 2 · p. 8

5.7. Qualitative Analysis

  • Across nurses, CEOs, and farmers, base outputs are homogeneous, while SGS confines steering to person regions and maintains scene coherence without leaking into attire or background.EA’s uniform shift is clearest for nurses, where the figures converge on one demographic.Source: §5.7 · p. 8

5.8. Compute Cost

  • At K=3, SGS takes 14.6s versus 7.8s for the baseline; its overhead is 6K additional forward passes and grows with scene complexity.The reported per-image timing is measured on an A6000 GPU.Source: Table 3 · p. 8, §5.8 · p. 8

6. Limitations

The system has three stated limitations: accuracy in crowded scenes, compute overhead, and steering that does not account for explicit prompt constraints.

6. Limitations

  • For k≥4 in heavily occluded scenes, segmentor accuracy degrades and also affects steering accuracy.The per-instance forward pass also adds compute proportional to K; batching trades wall-clock time for VRAM, a trade-off not explored within the hardware budget.Source: §6 · p. 8
  • SGS steers regardless of prompt intent, so a prompt such as “four male doctors” will still have demographics redistributed.The authors suggest a lightweight prompt parser that zeroes steering for explicitly constrained slots.Source: §6 · p. 8
  • The per-instance forward pass adds compute proportional to K; batching could reduce wall-clock time but uses more VRAM.This trade-off was not explored within the available hardware budget.Source: §6 · p. 8

7. Conclusion

SGS applies demographic steering per person without fine-tuning, inversion, or layout input. Across FLUX.1-Dev and SD3.5-Large, it raises intersectional entropy while recovering near-best aggregate balance and preserving semantic coherence.

7. Conclusion

  • SGS raises intersectional entropy from 0.524 to 0.794 on FLUX.1-Dev and from 0.549 to 0.832 on SD3.5-Large.The conclusion describes it as the only evaluated method to break the trade-off between aggregate balance and intra-image diversity.Source: §7 · p. 8
  • SGS performs per-instance demographic steering for multi-person generation without fine-tuning, inversion, or layout input.The conclusion attributes its result to per-instance spatial steering.Source: §7 · p. 8
  • The reported result holds across FLUX.1-Dev and SD3.5-Large, two architecturally distinct diffusion transformers.Both architectures preserve semantic coherence while SGS recovers near-best aggregate balance.Source: §7 · p. 8

Supplementary Material

The supplement documents the segmentor’s generated-image dataset and DETR-style architecture, including its attention features, mask outputs, and training setup.

A. Segmentor - Training Dataset Details

  • The segmentor dataset contains 1,440 FLUX.1-Dev images across person counts 2–5, nine scene types, and three difficulty levels.The 720-image test set uses nine disjoint out-of-distribution scene types and a different framing protocol.Source: A. Segmentor - Training Dataset Details · p. 12
  • SAM2-generated pseudo-label masks are filtered by area, matched to person count with Hungarian matching on centroids, and deduplicated at pairwise IoU above 0.5.The dataset split is stratified by person count and difficulty.Source: A. Segmentor - Training Dataset Details · p. 12

B. Segmentor - Architecture

  • The segmentor takes min-max-normalized MM-DiT self-attention features from layer bins 3 and 4 at the detection timestep, with timestep conditioning.The bin choice follows an ablation in which performance improved with layer depth and bins 3+4 outperformed tested alternatives.Source: B. Segmentor - Architecture · p. 12
  • Eight learnable instance queries produce confidence scores, centers, and 32×32 masks; inference retains queries above confidence 0.5 and yields at most eight masks.The model uses a three-block CNN backbone and a two-layer transformer decoder, and has 1.2 million parameters.Source: B. Segmentor - Architecture · p. 12

C. Segmentor Results - Per Difficulty

Segmentor performance depends on scene difficulty and person count, while steering validation, classifier checks, a second-architecture sweep, qualitative comparisons, and hyperparameter analyses characterize downstream behavior and operating choices.

C. Segmentor Results - Per Difficulty

  • In Figure 4, mIoU, AP50, and AP50-95 decline with greater scene difficulty and person count, as occlusion, smaller regions, and spatial competition increase.AP50 remains consistently high, while AP50-95 drops more substantially, indicating less precise mask boundaries at stricter IoU thresholds.Source: C. Segmentor Results - Per Difficulty · pp. 12–13, Figure 4 · p. 13

D. Steering Performance by Person Count and Oracle-Mask Validation

  • At K=2 and K=3, DETR matches or exceeds oracle-mask race accuracy, but at K=4 oracle masks lead by 0.058; the gender-accuracy pattern is similar but smaller.The comparison uses matched pairs; background preservation is flat across mask sources at every K.Source: D. Steering Performance by Person… · p. 13, Table S3 · p. 13

E. Classifier Validation

  • FairFace and CLIP reach 95–96% race accuracy and greater than 99% gender accuracy against known-correct labels; the three classifiers also agree strongly on race and gender (Krippendorff’s alpha 0.94 for each).The known-label validation set contains 1,832 face crops from FLUX.1-Dev and SD3.5-Large.Source: E. Classifier Validation · pp. 14–15, Table S4 · p. 15

F. Segmentor Timestep Sweep – SD3.5-Large

  • On SD3.5-Large, segmentation peaks at timestep index 17 with mIoU 0.664, AP50 0.841, and AP50-95 0.413, comparable to FLUX’s reported peak.The sweep evaluates a separate segmentor on SD3.5’s 28-step schedule and a held-out validation split of 288 images.Source: F. Segmentor Timestep Sweep –… · p. 15, Figure 6 · p. 15

G. Segmentor Results - Qualitative

  • In harder qualitative scenes, DETR produces coherent per-person masks with correct instance counts, while clustering fragments or merges masks and CenterNet can overestimate counts.The comparison spans four scenes from easy to hard.Source: Figure 5 · p. 14, G. Segmentor Results - Qualitative · p. 15

H. Steering Hyperparameter Sensitivity

  • At tstart=4 and tend=10, extending the steering window from 8 to 10 improves race accuracy from 0.632 to 0.708 and gender accuracy from 0.744 to 0.757.The selected setting balances accuracy with background preservation; further extension offers diminishing accuracy gains at the cost of increasing semantic drift.Source: H. Steering Hyperparameter Sensitivity · p. 16, Table S6 · p. 16

I. Steering Hyperparameter and Window Ablation – SD3.5-Large

The SD3.5-Large ablations select steering strength α=2.0, λ=4.0 and a window starting at step 4 and ending at step 24. Across the extended FLUX benchmark, SGS increases intra-image diversity for every occupation, while gender balance varies by occupation.

I. Steering Hyperparameter and Window Ablation – SD3.5-Large

  • On SD3.5-Large, α=2.0 and λ=4.0 are the best-performing steering-strength configuration tested.Accuracy improves monotonically across the tested values of both parameters.Source: §I · p. 16, Table S7 · p. 16
  • Among tested SD3.5-Large steering windows, start=4 and end=24 achieve the highest accuracy in the 28-step schedule.Table S8 evaluates start steps 3, 4, 5, and 6 with different end steps at α=2.0 and λ=4.0.Source: §I · p. 16, Table S8 · p. 16

J. Extended Benchmark – Occupations and Prompt Templates

K. Per-Occupation Demographic Distribution

  • SGS raises Hix for all 30 occupations, from a smallest gain of 0.480→0.578 for Driver to a largest gain of 0.395→0.861 for Tailor.No occupation has lower intra-image diversity than the unsteered baseline.Source: K. Per-Occupation Demographic… · p. 17, Table S9 · p. 17
  • DRrace falls for every occupation, while DRgender moves further from balance for five of the 30 occupations.The passage attributes the noisier per-occupation gender pattern to the two-category assignment problem and n=150 samples per occupation.Source: K. Per-Occupation Demographic… · p. 17, Table S9 · p. 17

L. Extended Qualitative Analysis

Across person counts and stereotype directions, SGS independently diversifies people while preserving occupational cues; the qualitative pattern also appears on SD3.5-Large. The supplementary benchmark adaptation reimplements UNet-oriented baselines for MM-DiT architectures.

L. Extended Qualitative Analysis

  • Across k=2, 3, and 4, SGS assigns distinct demographic targets to each person and reliably separates all people as scene complexity increases.The examples span multiple racial categories and both genders within one image.Source: L. Extended Qualitative Analysis · p. 18, Figure 7 · p. 19
  • SGS preserves occupational markers across conditions, including scrubs and stethoscopes, badges and hats, aprons, and safety helmets and vests.The passage attributes this confinement of steering to the person-region instance masks.Source: L. Extended Qualitative Analysis · p. 18

M. Qualitative Comparison – SD3.5-Large

  • On SD3.5-Large, SGS diversifies each person across four prompts while the five baselines remain homogeneous or only mildly shifted.The qualitative comparison also reports preservation of occupational context.Source: M. Qualitative Comparison – SD3.5-Large · p. 18

N. Baseline Adaptation to Diffusion Transformers

  • The compared FairImagen, RespoDiff, and SDID implementations were designed for older UNet-based models and do not run on MM-DiT as released.The paper reimplemented their core mechanisms for FLUX and SD3.5-Large rather than mechanically porting UNet-specific hooks.Source: N. Baseline Adaptation to Diffusion… · p. 18
  • FairImagen and SDID were adapted to MM-DiT by applying their mechanisms to pooled conditioning and a single representative image-side block, respectively.Broadcasting a residual across every joint-transformer block broke generation; using one representative block produced coherent, steerable output.Source: N. Baseline Adaptation to Diffusion… · pp. 18–19

O. Human Evaluation Protocol

The human evaluation compares SGS with five baselines using blinded, randomized image pairs from the same scene. Raters judge perceived diversity, caption match, and overall preference.

O. Human Evaluation Protocol

  • The evaluation pairs SGS against Baseline, EA, FairImagen, RespoDiff, and SDID across 33 scenes with 2, 3, or 4 people.Each scene uses a fixed seed shared across methods.Source: O. Human Evaluation Protocol · pp. 19–20
  • Raters are blinded to method identity, see randomized left–right placement, and answer questions about diversity, caption match, and preference.The first two questions allow ties; preference is a forced binary choice.Source: O. Human Evaluation Protocol · p. 20, Figure 9 · p. 21
  • The analysis reports SGS preference percentages with 95% Wilson score intervals, exact binomial tests, and nominal Krippendorff’s α.Sessions with identical responses to every question are excluded; ties count as half a preference for percentages and are excluded from significance tests.Source: O. Human Evaluation Protocol · p. 20

Related papers

Loading 2610.29651v1…