PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffusionAgent: Navigating Expert Models for Agentic Image Generation

Jie Qin, Jie Wu, Weifeng Chen, Yueming Lyu

arXiv:2401.10061v2cs.CVcs.AI

TL;DR

Diffusion models face challenges with diverse prompts and the differing strengths of general and domain-specific models. DiffusionAgent uses an LLM-guided model tree and human feedback to route prompts to generative experts. It reports higher image reward and aesthetic score than SD1.5.

  • Problem

    Stable diffusion models can underperform in specific domains or lose general adaptability, and descriptive-caption training leaves instructions and inspirations challenging prompt types.

    Source: §I · p. 1

  • Method

    DiffusionAgent uses an LLM to parse prompts and guide a Tree-of-Thought model structure, enriched with human feedback, to select a suitable generative model.

    Source: §I · p. 2, B. Tree-of-thought of Models · p. 3

  • Results

    0.35% on image reward and 0.44% on aesthetic score: DiffusionAgent improves both over SD1.5.

    Source: C. Quantitative Results · p. 5

  • Takeaways & Limitations

    The framework is training-free and plug-and-play, and its model tree can integrate new models based on their attributes.

    Source: B. Tree-of-thought of Models · p. 3, §V · p. 7

Abstract

from arXiv · show

In the accelerating era of human-instructed visual content creation, diffusion models have demonstrated remarkable generative potential. Yet their deployment is constrained by a dual bottleneck: semantic ambiguity in diverse prompts and the narrow specialization of individual models. A single diffusion architecture struggles to maintain optimal performance across heterogeneous prompts, while conventional "parse-then-call" pipelines artificially separate semantic understanding from generative execution. To bridge this gap, we introduce DiffusionAgent, a unified, language-model-driven agent that casts the entire "prompt comprehension-expert routing-image synthesis" loop into a agentic framework. Our contributions are three-fold: (1) a tree-of-thought-powered expert navigator that performs fine-grained semantic parsing and zero-shot matching to the most suitable diffusion model via an extensible prior-knowledge tree; (2) an advantage database updated with human-in-the-loop feedback, continually aligning model-selection policy with human aesthetic and semantic preferences; and (3) a fully decoupled agent architecture that activates the optimal generative path for open-domain prompts without retraining or fine-tuning any expert. Extensive experiments show that DiffusionAgent retains high generation quality while significantly broadening prompt coverage, establishing a new performance and generality benchmark for multi-domain image synthesis. The code is available at https://github.com/DiffusionAgent/DiffusionAgent

I. INTRODUCTION

DiffusionAgent addresses the mismatch between diverse prompts and diffusion models with uneven domain strengths. It uses an LLM, a Tree-of-Thought model structure, and human feedback to select generative models.

I. INTRODUCTION

  • Stable diffusion models face a trade-off: general models can underperform in specific domains, while domain-specific models sacrifice general adaptability.Their training data also consists of descriptive captions, which can make instructions and inspirations challenging inputs.Source: §I · p. 1
  • DiffusionAgent uses an LLM to parse diverse prompts and guide a Tree-of-Thought structure of generative models toward a suitable model.The structure incorporates prior knowledge and human feedback; an Advantage Database aligns selection with human preferences.Source: §I · p. 2
  • The framework is presented as an all-in-one system compatible with various diffusion models and different prompt types.The introduction contrasts this with approaches restricted to descriptive prompts.Source: §I · p. 2
  • DiffusionAgent is described as training-free and plug-and-play, combining Tree-of-Thought reasoning with human feedback to enhance accuracy and aggregate multiple experts.The authors frame this as a flexible route for integrating generative models.Source: §I · p. 2
  • The introduction claims DiffusionAgent outperforms traditional stable diffusion models and offers an efficient pathway for community development in image generation.The supplied introduction passages do not specify the comparison’s metrics or values.Source: §I · p. 2

II. METHODOLOGY

DiffusionAgent is an all-in-one system for generating images from diverse prompts by selecting a suitable generative model. An LLM controls its workflow across four stages.

II. METHODOLOGY

  • The system aims to parse diverse prompts and identify the generative model expected to produce the most optimal results.The stated objectives include high generalization, utility, and convenience.Source: §II · p. 2
  • An LLM acts as the main controller, maintaining the workflow and coordinating with multiple domain-expert generative models.The experts come from open-source communities including Hugging Face and Civitai.Source: §II · p. 2
  • The workflow proceeds through Prompt Parse, model-tree building and searching, model selection with human feedback, and execution of generation.The four steps are presented as the system’s overall pipeline.Source: §II · p. 2

A. Prompt Parse

The Prompt Parse Agent uses an LLM to extract salient content from varied user inputs. The resulting prompt is used for generation and supports selecting a suitable model.

A. Prompt Parse

  • The Prompt Parse Agent extracts salient textual information from input prompts, which can be prompt-, instruction-, inspiration-, or hypothesis-based.The passage describes this parsing as a way to handle complex user inputs.Source: A. Prompt Parse · p. 2
  • For instruction- and inspiration-based inputs, the agent extracts the core instruction or desired subject as the generation prompt.For example, “generate an image of a dog” becomes “an image of a dog,” while “I want to see a beach” becomes “a beach.”Source: A. Prompt Parse · p. 2
  • For hypothesis-based inputs, it extracts the hypothesis condition and forthcoming action’s object; parsing core content reduces noisy text and supports model selection.The example “If you give me a toy, I will laugh very happily” becomes “a toy and a laugh face.”Source: A. Prompt Parse · p. 3

B. Tree-of-thought of Models

A two-layer model tree organizes diffusion models by subject and style, then searches for candidates aligned with a prompt. Model selection combines those candidates with database-based performance information and prioritizes stronger matches.

B. Tree-of-thought of Models

  • The model-building agent groups model tags into subject categories and style subcategories, places models at leaf nodes, and integrates new models based on their attributes.The search agent uses breadth-first search to compare categories with the prompt and find a candidate set.Source: B. Tree-of-thought of Models · p. 3

C. Model Selection

  • The selection agent addresses limited model-attribute information by using human feedback and a database to align choices with human preferences.It matches the input to the top five semantically similar prompts, retrieves five top-performing models for each, then intersects the resulting 25-model set with Tree-of-Thought candidates.Source: C. Model Selection · p. 3, Figure 3 · p. 3

D. Execution of Generation

DiffusionAgent uses prompt examples from the selected model to extend prompts before generation. The reported settings use popular community models, and the SD1.5 comparison describes finer details and greater realism for human and scene images.

D. Execution of Generation

  • The Prompt Extension Agent enriches the selected model’s input prompt with detailed descriptions and vocabulary patterned on that model’s example prompts.The extended prompt is provided to the LLM alongside the examples using in-context learning.Source: D. Execution of Generation · p. 3

A. Settings

  • Compared with SD1.5, the SD1.5-based DiffusionAgent produces more realistic human and scene images with finer visual details.The figure caption reports improved visual fidelity and realism.Source: Figure 4 · p. 4
  • Table II evaluates ImageReward and aesthetic score against SD1.5 and random expert selection.The supplied table caption specifies the measures and comparison baselines but not their results.Source: Table II · p. 4
  • The experiments selected popular models across types and styles from Civitai and Hugging Face for the SD1.5- and SDXL-based systems.Table I identifies the selected public trained models for each system.Source: A. Settings · p. 4, Table I · p. 4

B. Qualitative Results

The qualitative comparison identifies semantic omissions and weak human-related details in SD1.5 outputs. DiffusionAgent is described as capturing broader prompt context and producing more detailed human-related images.

B. Qualitative Results

  • The comparison examines four prompt types along semantic alignment and image aesthetics.The evaluation compares DiffusionAgent’s generation performance against SD1.5.Source: 1) Visualization of SD1.5 Version · p. 4
  • SD1.5 misses semantic information in prompts about a man, chef, children, and toyland, and struggles with facial and body details for girl and couple prompts.The analysis characterizes these as semantic lack and poor performance on human-related targets.Source: 1) Visualization of SD1.5 Version · p. 4
  • DiffusionAgent captures broader prompt context and produces detailed human-related images in the qualitative comparison with SD1.5.Examples include a man whistling tunes pianos, children building snowmen, and a couple under a starry sky.Source: 1) Visualization of SD1.5 Version · p. 4

2) Visualization of SDXL Version.:

The paper reports higher ImageReward and aesthetic scores than SD1.5, alongside qualitative comparisons of the SDXL-based system with baseline SDXL. Those comparisons include prompts where baseline SDXL loses semantic information and examples described as visually appealing from DiffusionAgent.

2) Visualization of SDXL Version.:

  • The ablation figure compares random selection as the baseline with ToT and ToT+HF agent variants.The caption identifies these as different agents’ performance but supplies no specific outcome.Source: Figure 6 · p. 5

C. Quantitative Results

  • Using an aesthetic predictor and human-feedback-related reward model to evaluate generated results, DiffusionAgent’s overall framework outperforms baseline SD1.5 in imagereward and aesthetic score, improving by 0.35% and 0.44%, respectively.Source: C. Quantitative Results · p. 5

D. Ablation Study

The ablation analysis reports improved realism, semantic alignment, and aesthetic appeal with Tree-of-Thought and human feedback compared with random selection. It also reports that extended prompts improve aesthetics, detail, style fidelity, and image refinement.

1) Tree-of-Thought and Human Feedback:

  • Adding Tree-of-Thought and human feedback to random selection significantly enhances image quality, realism, semantic alignment, and aesthetic appeal.The random variant produces many images lacking semantic coherence with input prompts.Source: 1) Tree-of-Thought and Human Feedback · p. 5

2) Prompt extension:

  • The prompt-extension ablation compares generation results using original and extended prompts.The figure caption describes prompt extension as providing richer prompts intended to produce higher-quality images.Source: 2) Prompt extension · p. 5, Figure 7 · p. 6
  • Extended prompts improve aesthetics and detail while helping outputs capture the desired artistic style and intricate nuances.The comparisons use original and extended prompts; extended descriptions are reported to yield greater fidelity and refinement.Source: 2) Prompt extension · p. 6

E. User Study

In the user study, participants rated DiffusionAgent outputs as higher quality than baseline outputs. The related work describes diffusion models as dominant in text-based image generation and notes methods that use reward signals to align outputs with human preferences.

E. User Study

  • Users rated DiffusionAgent outputs as higher quality than SD1.5 and SD XL outputs in a study of 100 prompts and 20 users.The study collected approximately 400 votes for each baseline model.Source: Figure 9 · p. 6, E. User Study · p. 6

A. Text-based Image Generation

  • Diffusion models emerged as a dominant text-based image-generation framework, while reward-signal methods aim to align outputs with human intent and preferences, [20]–.Source: A. Text-based Image Generation · p. 6

B. Large Language Models (LLMs) for Vision-Language Tasks

The paper draws on LLM-based tool use to guide text-to-image models. DiffusionAgent uses LLMs to parse prompts and select experts through a Tree-of-Thought framework, which the conclusion describes as training-free and plug-and-play across prompts and domains.

B. Large Language Models (LLMs) for Vision-Language Tasks

V. CONCLUSION

  • DiffusionAgent combines LLM prompt parsing with Tree-of-Thought expert selection in a training-free, plug-and-play framework for different prompts and domains.Source: §V · pp. 6–7
Loading 2401.10061v2…