PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Browsing: Controllable Diversity for Image Generation

Sara Dorfman, Maya Vishnevsky, Omer Dahary, Or Patashnik, Daniel Cohen-Or

arXiv:2606.23679v1cs.CVcs.AIcs.GRcs.LG

TL;DR

Text-to-image models often produce visually similar interpretations of the same prompt, limiting meaningful exploration. This paper induces diversity through explicit semantic decisions, producing structured galleries whose outputs represent distinct, interpretable alternatives.

  • Problem

    Text-to-image models often show limited semantic variation across samples, despite users wanting multiple plausible outputs from one prompt.

  • Method

    The method uses explicit semantic constraints and structured branching to generate navigable image galleries along meaningful variation axes.

  • Results

    Experiments across state-of-the-art text-to-image models show consistent, substantial diversity improvements and interpretable semantic alternatives.

  • Takeaways & Limitations

    Generated outputs form a structured, navigable semantic space in which differences are explicit, interpretable, and persistent rather than incidental.

  • Takeaways & Limitations

    The resulting semantic space depends on generative-model fidelity and agent reasoning, while current VLMs remain limited in proposing rich and diverse alternatives.

Abstract

from arXiv · show

Modern text-to-image models excel in visual fidelity and prompt adherence. However, this strict adherence comes at the cost of diversity: generated samples tend to collapse into a single visual interpretation. Existing methods to improve diversity produce outputs driven by incidental variations rather than meaningful design choices. This motivates a new variant of the diversity task where structure is enforced on the generated samples. We introduce a method for controlled diversity that enables Semantic Browsing, where users can navigate structured image galleries and experience creative exploration through a systematic traversal of meaningful, interpretable axes of variation. Achieving this level of semantic control requires a deep understanding of the scene. We exploit the fact that recent text-to-image models are trained on elaborated captions, effectively decoupling semantic decision-making from pixel generation. This enables a paradigm shift: instead of relying on stochastic variation within the text-to-image model, we induce diversity directly at the text level. By leveraging rich textual representations, we allow a Vision Language Model (VLM) to operate on the full scene context. To overcome the generic outputs typical of standard VLMs, we employ an agentic workflow that explicitly enforces structured variation attuned to the original prompt. We demonstrate that our method produces diverse and navigable design spaces where every variation corresponds to a specific, user-understandable semantic decision.

1 Introduction

The paper introduces Semantic Browsing, a form of controlled semantic diversity that lets users systematically explore meaningful, interpretable variations rather than stochastic differences. An agentic VLM workflow expands prompts into plausible variation dimensions, enabling structured image galleries while preserving unchanged prompt aspects.

  • 1 Introduction: Standard text-to-image sampling often produces limited semantic variation, with generations converging on the same high-level interpretation despite underspecified prompts.Samples may differ only in visually insignificant details or exhibit severe biases [Cohen et al. 2025].
  • 1 Introduction: Prior diversity methods perturb conditioning, repel sampling trajectories, or select diverse subsets, but they do not explicitly control what variations users obtain.Consequently, differences across samples are driven by incidental rather than meaningfully specified design choices [Sadat et al. 2023; Um and Ye 2025; Corso et al. 2023; Dahary et al. 2026; Parmar et al. 2025].
  • 1 Introduction: Semantic Browsing frames diversity as explicit, user-understandable changes in interpretable image attributes rather than emergent stochastic variation.Examples include object attributes, configurations, lighting, weather, and backgrounds, while preserving other prompt aspects.
  • 1 Introduction: An agentic VLM workflow enriches the user prompt and identifies plausible, under-specified semantic dimensions along which generated images can vary.The dimensions represent alternative semantic interpretations or design choices compatible with the original prompt.
  • 1 Introduction: The approach requires an image generator with fine-grained prompt control that changes specified semantics precisely while preserving all unmodified image aspects.This ensures gallery differences arise solely from intended semantic variations.
  • 1 Introduction: Experiments across state-of-the-art text-to-image models show consistent, substantial diversity improvements while enabling systematic exploration of structured galleries with controlled semantic differences.Each gallery variation corresponds to an explicitly specified semantic decision rather than stochastic sampling.

2 Related Work

Prior work improves diversity through guidance, sampling, prompt-level methods, or creative exploration, but often sacrifices prompt alignment, requires additional training, or lacks global semantic structure. This work instead organizes explicit semantic alternatives with a training-free hierarchical tree and uses multiple agents to explore interpretations rather than converge on one result.

  • Diversity in Text-to-Image Generation: Existing diversity methods trade variety against fidelity, computational cost, or reliability: CFG suppresses diversity, Autoguidance needs a separately trained weak model, and lightweight alternatives remain unreliable.CFG’s diversity suppression is documented in recent stage-wise analyses [Jin et al. 2025], while Autoguidance replaces the unconditional model with a weaker variant [Karras et al. 2024].
  • Diversity in Text-to-Image Generation: Conditioning and latent-level methods improve sample variety but can degrade prompt alignment because they relax guidance or manipulate sampling below the semantic prompt level.CADS [Sadat et al. 2023] and Guidance Interval [Kynkäänniemi et al. 2024] modulate conditioning during denoising, whereas Particle Guidance [Corso et al. 2023] and MinorityPrompt [Um and Ye 2025] use latent-level mechanisms.
  • Diversity in Text-to-Image Generation: Compared with PAG’s dataset dependence and lack of global prompt relationships, this approach is training-free and uses a hierarchical tree to reason collectively about semantic diversity.PAG uses GFlowNets for diverse sampling [Yun et al. 2025], while the proposed system organizes generated images hierarchically and reasons across multiple tree nodes.
  • Creative Generation and Exploration: Unlike prior creative-generation methods that explore object sub-categories or recombine existing concepts, this method explores creative alternatives directly within the semantic space.ConceptLab [Richardson et al. 2024] and adaptive negative prompting [Golan et al. 2025] target creative sub-categories, while other methods draw inspiration by decomposing and merging visual concepts [Goldberg et al. 2026; Vinker et al. 2023].
  • Multi-Agent Systems for Controllable Generation: Where existing multi-agent systems refine prompts toward a single best result, this work uses multiple agents to explore interpretations and organizes them in a hierarchical tree.Maestro [Wan et al. 2025] and PromptSculptor [Xiang et al. 2025] enhance intent alignment and fidelity by refining outputs, whereas the proposed workflow prioritizes exploration over narrowing intent.

3 Method

The method represents prompt interpretations as a hierarchical tree of structured scene JSONs, where semantic constraints create navigable and prompt-faithful variations. A multi-agent workflow grows this tree iteratively by identifying mutable details, organizing them into semantic aspects, proposing diverse refinements, and checking plausibility.

  • Scene Tree Representation: The method expands a prompt into a fully specified scene interpretation and constructs a rooted tree whose nodes are structured JSON scene interpretations.The structured representation enables fine-grained control over objects, attributes, and global scene properties.
  • Semantic Browsing: Each edge applies an atomic constraint instantiating one semantic aspect, and rendering every node produces a tree of images for structured Semantic Browsing.Previously fixed constraints are preserved along branches, while branching explores alternative realizations of a selected aspect.
  • Tree Requirements: Children of a node must share one semantic aspect, differ meaningfully, and remain logically consistent with the prompt and preceding branch constraints.These requirements jointly enforce Semantic Structuring, Heterogeneity, and Plausibility, balancing navigable variation with contextual faithfulness.
  • Multi-Agent Workflow: The workflow expands one node at a time: Context Analyst identifies mutable details, Brainstormer groups them into high-impact aspects, Decision Maker formulates divergent constraints, and Critic checks consistency.The workflow conditions new branches on the current scene, original prompt, and ordered trajectory of previously applied constraints.

4 Experiments

Semantic Browsing substantially improves structured diversity while preserving image quality and prompt alignment, and its semantic control remains effective across rendering backbones. Experiments further show that the generated hierarchy is semantically meaningful, logically consistent, and dependent on specialized agent roles.

  • Model-Agnostic Design: The framework is model-agnostic: FIBO modules perform prompt enhancement and scene refinement, while FLUX.2 renders the final images successfully.This separates semantic control from the underlying rendering backbone.
  • Qualitative Diversity: Semantic Browsing produces granular entity changes and holistic shifts in setting and mood, whereas baseline methods converge on a single mode.The structured outputs form triplets sharing ancestor nodes, so early branching decisions yield distinct yet internally consistent variations.
  • Quantitative Comparison: 3.34 Vendi Score and 0.61 DINO Similarity establish superior diversity over all baselines, while the 6.52 Aesthetic Score remains comparable.The results confirm that Semantic Browsing expands semantic coverage without sacrificing image quality.
  • Efficiency and User Study: Despite the added agentic structure, Semantic Browsing remains competitive in computational cost with baseline methods.A 25-participant head-to-head study across 12 randomly selected prompts per baseline comparison evaluates perceptual quality and diversity against four diversity methods.
  • Hierarchical Structure: Hierarchical Consistency reaches 0.87/1.0, while semantic distance increases with tree graph distance, showing that neighboring nodes remain related and distant nodes diverge more.These results validate logical continuity and the requirement that parent-child changes modify one semantic aspect at a time.
  • Ablations: Separating specialized agent roles raises overall diversity from 0.362 to 0.389, a 7.2% relative improvement, with larger DINO distances at every edge distance.Removing the Context Analyst preserves VQAScore at 0.90 but significantly reduces plausibility, while removing the Critic lowers VQAScore from 0.90 to 0.87 and permits semantic drift.

5 Conclusions, Limitations and Future Work

The paper frames semantic diversity as explicit, interpretable decision-making rather than stochastic variation, producing structured and navigable spaces of image alternatives. Its current scope is limited by generative-model fidelity and VLM diversity, while the paradigm may extend to video, 3D, and multimodal generation.

  • Explicit semantic decisions make differences between generated images interpretable and persistent, yielding structured spaces rather than collections of incidental variations.
  • A multi-agent reasoning process explores multiple plausible interpretations of under-specified prompts while preserving inherited semantic commitments.
  • The explored semantic space depends on the generative model’s ability to faithfully realize fine-grained prompt modifications.
  • Modern VLMs maintain consistency and plausibility but remain limited in proposing rich and diverse semantic alternatives, constraining exploration breadth.
  • Structured semantic exploration could extend beyond images to video, 3D content, and multimodal generation, enabling coherent, navigable spaces of alternatives.

Appendix · A Baselines

The appendix evaluates the approach against baselines implemented with the same FIBO generation model and setting-specific hyperparameter optimization. These baselines test stochastic seeding, post-hoc selection, high-temperature sampling, and generator-level diversity interventions.

  • A Baselines: All baselines use the same FIBO generation model, with hyperparameters optimized specifically for this evaluation setting.This design supports a fair comparison across methods.
  • A Baselines: Stochastic VLM Seeding generates 27-image galleries by varying the random seed of the initial prompt-to-JSON VLM call.It relies solely on the model’s inherent stochasticity for diversity.
  • A Baselines: Post-Hoc Diversity Optimization over-generates 79 candidates and selects 27 maximizing pairwise DINO distance via QIP [Parmar et al. 2025].A strict 300-second limit applies per instance, and the 79-candidate pool matches the proposed tree-generation method’s total LLM calls.
  • A Baselines: High-Temperature VLM Seeding maximizes the initial VLM sampling temperature to test whether lower-probability token selection can close the diversity gap without structured intervention.It is compared against conventional probability-distribution sampling as a test of unstructured generation entropy.
  • A Baselines: CADS [Sadat et al. 2023] injects random noise into text embeddings, using optimized parameters 𝜏1 = 0.5, 𝜏2 = 0.9, 𝑠 = 3, and 𝜓 = 0.5.This is a generator-level method that modifies the text-to-image input space.
  • A Baselines: Guidance Interval [Kynkäänniemi et al. 2024] applies classifier-free guidance only over one-fifth of the denoising timestep range.The restricted interval accounts for FIBO’s relatively strong performance without standard CFG.
  • A Baselines: CADS, Guidance Interval, and Power-Law CFG are combined with Stochastic VLM Seeding to generate complete 27-image galleries and test additive diversity beyond random seeding.This combination isolates whether inference-level techniques provide benefits in addition to stochastic VLM variation.

B Implementation Details

The agents use Gemini 2.5 Flash with structured response templates, role-specific token limits and temperatures, and retry handling for transient API failures.

  • Implementation Details: Agents use Gemini 2.5 Flash, predefined parseable response templates, 4K–8K role-specific output limits, fixed temperatures of 0.4 or 0.7, and up to three retries with exponential backoff.These settings improve output structure and robustness to rare transient API failures.

C Efficiency

The computational cost is measured for the agentic workflow independently of image rendering, using amortized cost per result over a 27-image gallery. Semantic Browsing requires 10.2 seconds and 15.9K tokens per result, while Stochastic VLM Seeding is cheaper at 8.5 seconds and 3.3K tokens.

  • C Efficiency: Semantic Browsing requires 10.2 seconds and 15.9K tokens per generated result over a 27-image gallery, compared with 8.5 seconds and 3.3K tokens for Stochastic VLM Seeding.The evaluation measures amortized workflow cost per result and excludes image-generation rendering, whose cost depends on the underlying text-to-image backbone and is shared when methods generate the same number of images.
  • C Efficiency: The efficiency evaluation isolates agentic-workflow cost from image-generation rendering cost.Rendering cost varies with the underlying text-to-image backbone and is shared by methods generating the same number of images.

E Sensitivity to VLM Choice

Replacing Gemini 2.5 Flash with ChatGPT-5.5 while keeping all other components fixed yields closely matching results, demonstrating robustness to VLM choice.

  • E Sensitivity to VLM Choice: Vendi 3.30, Aesthetic 6.72, and VQAScore 0.94 with ChatGPT-5.5 closely match Gemini’s Vendi 3.34, Aesthetic 6.52, and VQAScore 0.90.The comparison replaces only the VLM backbone in the agentic workflow, indicating the framework is not tailored to a specific model.

F Scaling Ablation

The ablation shows that increasing tree depth or branching factor consistently raises gallery diversity, but with diminishing Vendi gains and quality trade-offs that differ by scaling dimension.

  • Scaling Ablation: Increasing either tree depth or branching factor consistently increases Vendi, although the gains become progressively smaller.The scaling analysis varies depth D and branching factor BF.
  • Depth Scaling: Deeper trees gradually decrease VQAScore while improving aesthetic quality, indicating a trade-off between strict prompt adherence and richer semantic discovery.This effect is reported when scaling depth with BF = 1.
  • Width Scaling: Wider trees mildly degrade both VQAScore and aesthetics at large branching-factor values.This effect is reported when scaling width with D = 1.
Loading 2606.23679v1…