PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimization

Motivation

• Vague T2I prompts require precise scene, style, and technical details, while abstract concepts and sparse inputs make user intent difficult to interpret.Existing workflows often rely on repeated refinement after inspecting generated images.Source: §1 · p. 1

Framework

• PromptSculptor decomposes prompt optimization into four specialized agents: Intent Inference, Scene and Style, Feedback and Tuning, and Self-Evaluation.The architecture targets distinct subtasks within prompt refinement.Source: §1 · pp. 1–2
• The Intent Inference Agent converts ambiguous concepts into concrete details, while the Scene and Style Agent adds scene, background, and style descriptions.The system uses Chain-of-Thought reasoning to support step-by-step intent understanding and scene enrichment.Source: §1 · p. 2

Contributions

• The paper presents the MAS approach as the first use of a multi-agent system for T2I prompt optimization, achieving state-of-the-art performance as shown in Table 1.The contribution is positioned as improving generation quality and flexibility over prior single-agent systems.Source: §1 · p. 2

Iterative refinement

• Self-evaluation and feedback tuning iteratively refine prompts until generated images better match user expectations, reducing required runs.The feedback mechanism is presented as a novel component of T2I prompt optimization.Source: §1 · p. 2

Applicability

• The framework is model-agnostic and works across various T2I models without model-specific tuning.Its LLM-agent and feedback-based design supports transfer across different T2I systems.Source: §1 · p. 2

2.1 Prompt optimization

• Prompt optimization evolved from white-box embedding or logit access toward black-box editing, candidate selection, evolutionary, reinforcement-learning, planning, and LLM-based strategies.The reviewed methods include AutoPrompt, BBTv2, GRIPS, APO, EvoPrompt, PromptAgent, APE, OPRO, and OPT2I.Source: §2.1 · pp. 2–3
• Unlike prior prompt-optimization methods, PromptSculptor uses a plug-and-play multi-agent system operating at the prompt level without fine-tuning.The approach is positioned as distinct from methods that optimize parameters, embeddings, or continuous prompt representations.Source: §2.1 · p. 3

2.2 Multi-agent system

• Multi-agent systems coordinate multiple large language models for complex tasks and have been explored in generation, labeling, bioinformatics, forecasting, autonomous driving, and privacy protection.PromptSculptor builds on this broader multi-agent trend for prompt optimization.Source: §2.2 · p. 3

2.3 Improve the Consistency of T2I Model

• T2I consistency methods use cross-attention, layout transformations, model fine-tuning, AI feedback, or image selection to improve alignment or generation outcomes.These approaches modify guidance, models, or outputs in different ways.Source: §2.3 · p. 3
• PromptSculptor instead uses a correction agent to check prompt-image alignment while remaining compatible with T2I models accessed through APIs, including Midjourney.This prompt-level operation broadens compatibility beyond models that can be fine-tuned.Source: §2.3 · p. 3

3.1 Multi-Agent System Overview

• The system combines intent inference, scene and style enrichment, self-evaluation, and feedback tuning in a staged prompt-optimization pipeline.The four agents respectively extract missing intent, build vivid scenes, check image alignment, and incorporate user feedback.Source: §3 · p. 4, §3.1 · p. 4

3.2 Intent Inference Agent

• Intent Inference Agent uses step-by-step reasoning to interpret implicit cues and metaphors, such as translating “lion” into strength, majesty, or courage.It captures explicit requests, contextual hints, and emotional undertones before prompt enrichment.Source: §3.2 · p. 4

3.3 Scene and Style Agent

• Scene and Style Agent converts inferred abstract concepts into concrete visual elements while specifying subjects, medium, environment, lighting, color, mood, and composition.The preceding intent and context analysis helps it generate more accurate settings.Source: §3.3 · p. 4

3.4 Self-Evaluation agent

• Self-Evaluation Agent computes CLIP similarity and invokes BLIP-2 captioning when the score is below a threshold to identify missing or discrepant content.It compares the generated caption with both the original and optimized prompts, then modifies the optimized prompt accordingly.Source: §3.4 · p. 5

3.5 Feedback and Tuning Agent

• Feedback and Tuning Agent applies direct user comments to iteratively refine the optimized prompt until the generated result meets the user’s requirements.This loop addresses intent gaps that may remain after automated self-evaluation.Source: §3.5 · p. 5

4.1 Experiment Settings

• The evaluation measures generated-image quality through PickScore, CLIP score, and Aesthetic score computed between original prompts and generated images.PickScore captures alignment and overall quality, CLIP measures prompt-image similarity, and Aesthetic score reflects human preference.Source: §4.1 · p. 6

4.2 Overall Results

• PromptSculptor achieves the highest PickScore, Aesthetic Score, and human expert preference score, while attaining the second-highest CLIP score.The comparison includes Original, GPT-4 Extended, MagicPrompt, PromptAgent, and ablations with or without the Self-Evaluation Agent.Source: §4.2 · p. 6, Table 1 · p. 6
• Self-Evaluation Agent ablations consistently improve evaluation performance compared with removing the agent, without requiring additional user feedback.The reported benefit is attributed to automatic alignment evaluation.Source: §4.2 · p. 6

4.3 Human Evaluation

• Twenty volunteers created 60 intentionally vague prompts and rated first-round images from 0 to 100 while recording modifications needed for satisfaction.PromptSculptor received the highest preference scores and required the fewest modifications in the human evaluation.Source: §4.3 · p. 6, Table 2 · p. 6

4.4 Visualization Results

• Appendix visualizations report better output quality and closer alignment with users’ intent than competing methods.The paper also includes prompt comparisons in the appendix.Source: §4.4 · pp. 6–7

5 Industry Impact

• A startup collaboration is developing a platform intended to help inexperienced users generate impressive figures from simple ideas.The platform is under development and is expected to become public soon.Source: §5 · p. 7

6 Conclusion

• PromptSculptor decomposes prompt refinement into specialized agents that generate detailed, context-aware prompts from simple and vague inputs.The reported experiments show improved image quality and aesthetic appeal.Source: §6 · p. 7

• The authors’ prompts more effectively interpret abstract concepts by using vivid visual metaphors, layered symbolism, and concrete scene details.The appendix summarizes this overall conclusion after comparing the six themes.Source: A Appendix · p. 10

Dreams Fuel Growth

• For “Dreams Fuel Growth,” the authors’ prompt links a child’s aspirations to a future astronaut through a glowing trail of stardust.The scene contrasts with prompts described as more abstract or simplistic and is presented as a vivid visual metaphor.Source: A Appendix · pp. 10–11

Hope in the Darkness

• For “Hope in the Darkness,” sunlight breaking through storm clouds and birds symbolize hope, freedom, and resilience more deeply than the alternatives.The prompt creates emotional uplift through contrast between darkness and light.Source: A Appendix · pp. 10–11, Figure 9 · p. 11

Knowledge is Power

• For “Knowledge is Power,” children studying in a bleak classroom use warm light around their books to symbolize hope and perseverance through education.The authors contrast this layered interpretation with other prompts’ more literal depictions of reading.Source: A Appendix · pp. 10, 12, Figure 10 · p. 12

Love Transcends Distance

• None of the “Love Transcends Distance” prompts, including the authors’ version, clearly conveys the crucial sense of distance, leaving this theme as an area for improvement.The compared prompts depict separated lovers or people but do not sufficiently emphasize distance.Source: A Appendix · pp. 10, 12, Figure 11 · p. 12

Time Heals All Wounds / World Peace

• The “Time Heals All Wounds” prompt uses rebuilding after a fire as a metaphor for recovery, while the “World Peace” prompt emphasizes human connection, nature, and optimism.The appendix describes the rebuilding metaphor as more thoughtful and layered, and the peace scene as more emotionally genuine and visually detailed.Source: A Appendix · pp. 10, 13, Figure 12 · p. 13