PaperlayerView exact revision on arXiv

Source-linked AI summary

On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers

Omer Dahary, Benaya Koren, Daniel Garibi, Daniel Cohen-Or

arXiv:2603.28762v2cs.CVcs.AIcs.GRcs.LG

TL;DR

T2I models often converge on narrow, typical visual solutions, while existing diversity interventions either require costly optimization or disrupt spatially committed latents. This paper applies on-the-fly repulsion to enriched text representations inside DiT multimodal attention, redirecting generative intent before composition is fixed. The method produces richer diversity without sacrificing visual fidelity or semantic adherence, with small computational overhead and no additional memory.

  • Problem

    T2I models’ strong conditioning can narrow outputs around typical visual modes, creating a diversity–quality trade-off for creative generation.

  • Method

    The method applies on-the-fly repulsion to enriched text tokens between DiT multimodal attention blocks, steering generative intent while image structure is emerging.

  • Results

    Across DiT architectures and COCO evaluations, Contextual Space repulsion consistently produces richer diversity without mode collapse or semantic misalignment, with small overhead and no additional memory.

  • Takeaways & Limitations

    Contextual Space provides a controllable intervention level where diversity can increase without sacrificing visual quality or relying on heavy optimization.

  • Takeaways & Limitations

    The method does not directly control which attributes vary, may favor coarse semantic changes, and leaves coordination with later generation stages open.

Abstract

from arXiv · show

Modern Text-to-Image (T2I) diffusion models have achieved remarkable semantic alignment, yet they often suffer from a significant lack of variety, converging on a narrow set of visual solutions for any given prompt. This typicality bias presents a challenge for creative applications that require a wide range of generative outcomes. We identify a fundamental trade-off in current approaches to diversity: modifying model inputs requires costly optimization to incorporate feedback from the generative path. In contrast, acting on spatially-committed intermediate latents tends to disrupt the forming visual structure, leading to artifacts. In this work, we propose to apply repulsion in the Contextual Space as a novel framework for achieving rich diversity in Diffusion Transformers. By intervening in the multimodal attention channels, we apply on-the-fly repulsion during the transformer's forward pass, injecting the intervention between blocks where text conditioning is enriched with emergent image structure. This allows for redirecting the guidance trajectory after it is structurally informed but before the composition is fixed. Our results demonstrate that repulsion in the Contextual Space produces significantly richer diversity without sacrificing visual fidelity or semantic adherence. Furthermore, our method is uniquely efficient, imposing a small computational overhead while remaining effective even in modern "Turbo" and distilled models where traditional trajectory-based interventions typically fail.

1 Introduction

T2I models achieve strong semantic alignment but often converge on typical visual solutions, creating a quality–variety trade-off. The paper intervenes in DiT Contextual Space to steer diversity after structural feedback emerges, reporting richer diversity with fidelity, adherence, and efficiency preserved.

  • Motivation: Typicality bias narrows outputs despite high-fidelity synthesis, limiting generative AI’s role as a creative partner.The paper frames diversity as a central research problem rather than a secondary metric.
  • Motivation: Strong conditioning, especially CFG, sharpens generation around a single mode and suppresses nearby semantically valid alternatives.Restoring diversity therefore requires avoiding degraded image structure and semantic adherence.
  • Prior limitations: Upstream interventions lack structural feedback or require costly optimization, while downstream latent repulsion can disrupt fixed visual modes and cause artifacts.These approaches divide the diversity problem by intervention timing along the denoising trajectory.
  • Prior limitations: The core difficulty is acute in few-step Turbo models because upstream search is slow and downstream repulsion arrives after composition is decided.The generative path is determined almost instantly in these models.
  • Approach: The proposed method repels enriched text tokens between multimodal attention blocks, steering generative intent while representations remain structurally informed but composition is not fixed.This intervenes during the transformer forward pass rather than through trajectory optimization.
  • Results: Across DiT architectures and COCO evaluations, Contextual Space repulsion produces richer diversity without mode collapse or semantic misalignment, with small overhead and no additional memory.The method is reported as compatible with rapid inference in modern distilled models.

2 Related Work

Prior diversity methods intervene either before stable image structure forms or during the denoising latent trajectory. These strategies face semantic drift, computational cost, or artifacts because their steering signals are poorly aligned with emerging image structure.

  • Diffusion Transformers: DiTs process text and image tokens concurrently, unlike U-Nets that typically incorporate text through cross-attention layers.This conditioning distinction motivates locating diversity interventions within DiT multimodal processing.
  • Taxonomy: The literature is organized around intervention stage and level: upstream methods modify initial conditions, whereas downstream methods manipulate image latents during denoising.This categorization frames the diversity–alignment gap addressed by the paper.
  • Upstream Interventions: Upstream methods optimize noise or text conditioning before stable structure emerges, often risking semantic drift or requiring costly optimization.CADS injects prompt-agnostic noise, while CNO uses an early predicted image estimate to force divergence.
  • Downstream Interventions: Downstream methods apply repulsion or guidance schedules throughout denoising, manipulating image latents or CFG to force variance.Particle Guidance and later work refine repulsion objectives, while scheduling methods modulate CFG over time.
  • Shared limitation: A recurring limitation is that raw-latent or external-encoder signals lack semantic coherence during early denoising, forcing an unfavorable timing trade-off.Upstream approaches incur computational overhead, whereas downstream interventions operate after composition is largely committed.

3 Method: Repulsion in the Contextual Space

The method intervenes in DiT Contextual Space, where multimodal attention creates semantically enriched text representations that remain tied to emerging image structure. Batch-level repulsion on these representations steers diverse generative intent without operating directly on spatially committed latents.

  • Contextual Space: Contextual Space is structurally informed yet conceptually flexible, positioning it between unformed noise and a rigid image-latent manifold.This location is the paper’s central design rationale for diversity intervention.
  • Contextual Space: Multimodal attention exchanges text and image features, allowing text to guide semantics while image features provide spatial and visual feedback.The resulting enriched text tokens become tied to the specific image being formed.
  • Contextual Space: Fixed semantic token ordering makes Contextual Space representations comparable across samples even when corresponding image content shifts spatially.Each token index generally represents the same conceptual component across the batch.
  • Repulsion mechanism: The method applies particle-guidance repulsion between multimodal blocks to separate batch samples in Contextual Space before high-level planning commits to a visual mode.All samples start from the same unmodified prompt encoding, mitigating permanent semantic drift.
  • Repulsion mechanism: On-the-fly activation updates avoid backpropagation through model layers, and inner-block iterations refine contextual token positions with low computational cost.Repulsion is applied across transformer multimodal blocks, with intervention focused on early stages where semantics and composition are most strongly determined.
  • Diversity objective: The diversity objective uses the Vendi Score, which measures the effective number of distinct batch samples from eigenvalues of a similarity matrix.Contextual Space’s global semantic intent motivates batch-level similarity rather than token-wise or local diversity measures.
  • Diversity objective: Samples are flattened into contextual vectors, compared with a cosine-similarity kernel, and optimized through the negative von Neumann entropy of normalized-kernel eigenvalues.The objective encourages tokens to span a higher-dimensional manifold and counters CFG-induced semantic collapse.

4 The Contextual Space

Controlled interpolation and extrapolation reveal that VAE latent representations are tied to spatial layouts, whereas Contextual Space supports coherent semantic transitions. The latter preserves visual fidelity and meaningful extrapolation across changing attributes and compositions.

  • Experimental setup: The experiment compares VAE latent and Contextual Space representations by blending source and target internal features during denoising.Two samples from different noise seeds are generated for each of two prompts.
  • Latent Space: VAE latent interpolation between spatially unaligned images produces structural blur, incoherent overlays, and ghostly artifacts.Extrapolation can push VAE latents outside the learned data manifold, causing severe artifacts.
  • Contextual Space: Contextual Space interpolation yields smooth semantic transitions while maintaining sharp, high-fidelity structure.The model modifies appearance and composition coherently rather than blending incompatible pixel geometries.
  • Contextual Space: Contextual Space extrapolation remains semantically consistent with the steering direction instead of collapsing into noise or artifacts.Examples include progressively removing creature horns or changing other meaningful semantic attributes.

5 Experiments

Experiments across modern DiT architectures show that Contextual Space repulsion improves diversity while preserving image quality, prompt alignment, and practical efficiency. Qualitative, quantitative, user, editing, and ablation results consistently support this trade-off.

  • Experimental setup: Experiments span Flux-dev, SD3.5-Turbo, and SD3.5-Large, covering guidance-distilled, few-step, and standard non-distilled DiT regimes.The evaluation uses COCO prompts and compares against upstream methods such as CADS and SGI and downstream methods such as PG and SPARKE.
  • Qualitative results: Contextual Space repulsion produces diverse compositions and camera angles while alleviating repetitive typicality biases in Flux-dev outputs.The method uses fixed seeds and distinct initial noises for qualitative comparisons against the base model.
  • Baseline comparisons: Compared with baselines, the method avoids downstream artifacts and upstream semantic drift while retaining richer diversity and prompt adherence.PG and SPARKE can distort image structure, CADS can weaken prompt alignment, and SGI can remain repetitive when the base model lacks diversity.
  • Qualitative results: The method transfers to Flux-Kontext without model or intervention modifications, generating diverse coherent edits that preserve edit semantics and the original image.The same Contextual Space intervention is applied within the editing instruction stream.
  • Quantitative results: The evaluation measures semantic diversity with Vendi Score alongside ImageReward, VQAScore, and KID to characterize the diversity-quality trade-off.Pareto frontiers are constructed by varying each baseline’s control hyperparameters.
  • Quantitative results: On Flux-dev, the method achieves a superior Pareto frontier across semantic diversity, human preference, prompt alignment, and distributional fidelity.It exceeds SGI, identified as the strongest baseline, while requiring only 20%–30% additional runtime relative to the base model.
  • User study: User evaluations report higher preference than all competing approaches overall, with SGI closely matched and slightly behind.The study collected 450 responses from 45 participants.
  • Ablation studies: Increasing the repulsion scale generally increases diversity by changing structures, landscapes, and seasons in generated scenes.The ablation compares base generations at η=0 with progressively stronger repulsion.

6 Conclusions

The paper identifies Contextual Space as an effective intervention point for increasing diversity while preserving structure and semantic grounding. It also acknowledges limits in attribute-level control and later-stage coordination.

  • Core conclusion: Contextual Space representations encode semantic intent shaped by the emerging image without being tied to a spatial grid.This combination allows semantic separation without tearing image geometry or introducing artifacts.
  • Ablation: Contextual-space repulsion achieves a superior diversity-quality frontier compared with image-attention repulsion across human preference, prompt alignment, and distributional fidelity.The comparison is reported for Flux-dev using Vendi Score, ImageReward, VQAScore, and KID.
  • Ablation: Image-space repulsion is limited by spatial rigidity, whereas Contextual Space repulsion produces more varied compositions.The qualitative ablation directly contrasts the two repulsion spaces.
  • Core conclusion: Repulsion in Contextual Space increases diversity in a controlled way without sacrificing visual quality or requiring heavy optimization.The intervention operates where representations remain flexible but are already grounded in the forming image.
  • Limitations and future work: The method does not directly control which attributes vary and may favor coarse semantic changes over fine, user-specified ones.The paper proposes attribute-guided repulsion as a future direction and leaves coordination with later generation stages open.

A Implementation Details

Experiments use fixed hardware, model-specific denoising schedules, and shared evaluation settings across DiT architectures. The proposed intervention targets text tokens in multimodal attention blocks, with model-specific branch and block coverage.

  • Evaluation setup: Experiments were conducted on an NVIDIA A100 using groups of four images, with denoising steps set to 4 for SD3.5-Turbo, 20 for Flux-dev, and 28 for SD3.5-Large.Diversity metrics were calculated within each four-image group and averaged across groups.
  • Proposed method: The method uses M=100 gradient steps for Stable Diffusion models and M=50 for Flux-dev, applying repulsion to text tokens in multimodal attention blocks.SD3.5-Large uses both conditional and unconditional branches; Flux-dev and Flux-Kontext additionally use later single-stream blocks.
  • Baseline setup: Baselines use official implementations when available, while incompatible methods were reimplemented and hyperparameters tuned for competitive diversity levels.The appendix specifies search ranges for PG, CADS, SPARKE, and SGI.

B Additional Qualitative Results

Additional qualitative results are provided across SD3.5-Large, SD3.5-Turbo, and Flux-dev.

  • Additional qualitative results: Additional qualitative results are reported for SD3.5-Large, SD3.5-Turbo, and Flux-dev.The Flux-dev results appear in three additional figures.
  • Additional qualitative results: SD3.5-Large and SD3.5-Turbo are each covered by one additional qualitative-results figure.The figures are identified as Figures 15 and 16, respectively.
  • Additional qualitative results: Flux-dev is covered by three additional qualitative-results figures.These are identified as Figures 17, 18, and 19.

C Additional Quantitative Results

Additional evaluations show competitive quality-diversity trade-offs across models and constrained prompts, with detailed metrics, qualitative examples, and user-study results supplied in the appendix.

  • Quantitative comparisons: The method achieves competitive quality-diversity trade-offs on SD3.5-Large and SD3.5-Turbo at a fraction of SGI’s computational cost.Detailed metrics are provided for Flux-dev, SD3.5-Large, and SD3.5-Turbo.
  • Detailed prompts: On 100 highly constrained PartiPrompts prompts, the method increases diversity and human preference with negligible impact on prompt alignment.The evaluation uses the longest prompts from the Complex and Fine-Grained Detail categories with Flux-dev.
  • User study: A user-study table compares the method with five competing approaches across four evaluation metrics using 450 pairwise comparisons per metric.Results report percentages preferring the method, the competitor, or both equally.

D Additional Ablation Studies

Additional ablations show that repulsion’s behavior depends on where and when it is applied: attention-score intervention is costly and harmful, later windows favor preference over diversity, and middle transformer blocks provide the strongest balance.

  • Attention score repulsion: −0.075 Vendi and −0.178 ImageReward: attention-score repulsion reduced semantic diversity and human preference while introducing significant computational overhead.This occurred despite using FlexAttention for optimization.
  • Batch size ablation: The intervention’s positive trend across metrics as batch size increased suggests that the method scales effectively with batch size.The ablation reports average Vendi per pair to account for the raw score’s batch-size bound.
  • Timestep ablation: Repulsion applied later in the trajectory typically improved ImageReward at the expense of diversity for both SD3.5-Large and SD3.5-Turbo.Applying intervention throughout the trajectory produced the highest diversity but a more pronounced decline in fidelity and alignment.
  • Transformer block ablation: Middle transformer blocks yielded the strongest diversity among the tested block groups while preserving high preference scores on SD3.5-Large and SD3.5-Turbo.The comparison covered the first, middle, and last thirds of the architecture’s blocks.
Loading 2603.28762v2…