PaperlayerView exact revision on arXiv

Source-linked AI summary

World-To-Image: Grounding Text-to-Image Generation with Agent-Driven World Knowledge

Moo Hyun Son, Jintaek Oh, Sun Bin Mun, Jaechul Roh, Sehyun Choi

arXiv:2510.04201v1cs.CVcs.AI

TL;DR

Text-to-image models often misinterpret novel or out-of-distribution concepts, reflecting gaps in their pretrained world knowledge. World-To-Image uses an agent to coordinate prompt refinement and web-retrieved visual grounding; it outperforms existing methods, including a +8.1% accuracy-to-prompt improvement on NICE.

  • Problem

    Text-to-image models can misinterpret novel concepts outside their pretraining distribution, and prompts containing such concepts often produce degraded results.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    World-To-Image uses an orchestrator to choose prompt refinement, web-sourced visual retrieval, or both, grounding generation without modifying the base model.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    +8.1% accuracy-to-prompt improvement on NICE; W2I also consistently outperforms baselines across NICE, DiffusionDB, and Lexica.

    Source: §4.2 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    The findings suggest that retrieval and adaptive prompting can unlock gains by providing multimodal signals for novel concepts, rather than relying on model scaling alone.

    Source: §5 · p. 8, §6 · p. 9

    W2I depends on relevant, high-quality retrieved images, may degrade with sparse or noisy imagery, and adds test-time computational overhead.

    Source: §5 · p. 9

Abstract

from arXiv · show

While text-to-image (T2I) models can synthesize high-quality images, their performance degrades significantly when prompted with novel or out-of-distribution (OOD) entities due to inherent knowledge cutoffs. We introduce World-To-Image, a novel framework that bridges this gap by empowering T2I generation with agent-driven world knowledge. We design an agent that dynamically searches the web to retrieve images for concepts unknown to the base model. This information is then used to perform multimodal prompt optimization, steering powerful generative backbones toward an accurate synthesis. Critically, our evaluation goes beyond traditional metrics, utilizing modern assessments like LLMGrader and ImageReward to measure true semantic fidelity. Our experiments show that World-To-Image substantially outperforms state-of-the-art methods in both semantic alignment and visual aesthetics, achieving +8.1% improvement in accuracy-to-prompt on our curated NICE benchmark. Our framework achieves these results with high efficiency in less than three iterations, paving the way for T2I systems that can better reflect the ever-changing real world. Our demo code is available here\footnote{https://github.com/mhson-kyle/World-To-Image}.

1 INTRODUCTION

The introduction frames novel and long-tail concepts as a persistent T2I grounding problem that text-only prompt polish cannot solve. World-To-Image responds with an agent that diagnoses comprehension risk and conditionally combines prompt refinement with web-sourced textual and visual evidence; it reports stronger semantic faithfulness and prompt adherence than text-only optimizers.

1 INTRODUCTION

  • T2I models produce high-fidelity images for familiar prompts but often misinterpret novel, long-tail, or domain-specific concepts outside their pretraining distribution (Rege et al., 2025; Zhao et al., 2025).The introduction attributes this persistent failure to evolving world knowledge and static pretrained representations.Source: §1 · p. 1
  • Training or fine-tuning is costly for rapidly emerging concepts, while surface-level prompt optimizers cannot supply missing semantic grounding (Li et al., 2024; Arar et al., 2024; Khan et al., 2025).The proposed alternative extends prompt optimization beyond text by integrating image retrieval and reference-image conditioning without retraining.Source: §1 · pp. 1–2
  • W2I consistently improves semantic faithfulness and prompt adherence over strong text-only prompt optimizers while maintaining competitive aesthetic quality.The authors curate prompts with novel concepts outside the base model’s training for evaluation.Source: §1 · p. 2
  • The agent diagnoses whether failures reflect rendering limits or concept-comprehension gaps, then selects strategies that use external world knowledge.Its tools include semantic decomposition, concept substitution, and multimodal grounding with web-sourced evidence.Source: §1 · p. 2

2 RELATED WORKS

Related work covers automated and interactive prompt optimization, alongside benchmarks that probe T2I models’ grounding in world knowledge. These lines of work motivate addressing gaps in reliable semantic and factual representation.

2 RELATED WORKS

  • Prior strategies include iterative prompt optimization, model fine-tuning, and external knowledge augmentation to address fixed pretrained datasets’ limits.This section organizes related approaches around improving image quality, generative performance, or access to external information.Source: §2 · p. 2

2.1 PROMPT OPTIMIZATION IN TEXT-TO-IMAGE

  • Prompt optimization spans LLM- and reinforcement-learning-driven methods, embedding-space test-time optimization, fine-tuned language models (Wang et al., 2023a), and online prompt weighting (Mo et al., 2024).The literature also includes safety and fairness optimizers (Um & Ye, 2025) and human-in-the-loop systems such as PromptMagician (Feng et al., 2023).Source: §2.1 · p. 3

2.2 WORLD KNOWLEDGE DRIVEN TEXT-TO-IMAGE

  • WorldGenBench, WISE (Niu et al., 2025), and Commonsense-T2I (Fu et al., 2024) evaluate cultural, factual, inferential, or commonsense grounding and report persistent model difficulties.Their scopes range from broad knowledge domains to adversarial prompt pairs testing commonsense expectations.Source: §2.2 · p. 3

3 WORLD-TO-IMAGE: AGENT-DRIVEN WORLD-KNOWLEDGE T2I GENERATION

World-To-Image addresses degraded generation on prompts with concepts absent from the model’s training by iteratively coordinating prompt refinement and visual retrieval. An orchestrator evaluates generated images and chooses language-space, vision-space, or sequential joint updates until a score threshold or iteration limit is reached.

3 WORLD-TO-IMAGE: AGENT-DRIVEN WORLD-KNOWLEDGE T2I GENERATION

  • Novel-concept prompts often degrade because the model has not encountered those concepts during training, requiring more time and iterations for meaningful images.W2I is an iterative agent-based optimization framework that dynamically uses world knowledge.Source: §3 · p. 3
  • At each iteration, the Orchestrator uses the previous prompt, image, exemplars, and evaluation score to choose the Prompt Optimizer Agent or Image Retriever Agent.The framework begins with a baseline generation without exemplars.Source: §3 · p. 3
  • The Prompt Optimizer refines the prompt while keeping exemplars fixed; the Image Retriever updates exemplars while leaving the prompt unchanged.Both agents can run sequentially in one iteration, with the optimized prompt guiding retrieval of more contextually aware exemplars.Source: §3 · pp. 3–4
  • After updates, the generator produces and scores a new image; the loop stops at score threshold τ or the iteration limit, returning the best image if the limit is reached.The evaluation score combines semantic alignment, keyword coverage, and aesthetic quality.Source: §3 · p. 4
  • Required keywords include entities, attributes, relations, styles, and constraints, identified using rule-based parsing followed by an LLM pass to merge synonyms and remove redundancy.The set can also include descriptors extracted from reference exemplars.Source: §3 · p. 4
  • An LLM grades keyword presence as present, partially present, or missing; critical keywords can be weighted, while aesthetic quality measures perceptual appeal.Aesthetic scores may come from an automatic quality model or an LLM rubric and are normalized to [0, 1].Source: §3 · p. 4

4 EXPERIMENTS

Across Lexica, DiffusionDB, and NICE, W2I outperforms all baselines, with its largest reported overall and Accuracy-to-Prompt gains on NICE. It also maintains strong image-quality and human-preference performance.

4.1 EXPERIMENT SETTING

  • Evaluation spans Lexica, DiffusionDB, and the 100-prompt NICE benchmark, using LLM Grader, human-preference rewards, and HPSv2.NICE covers five categories, with prompts mixing concepts or referencing post-2024 entities and events.Source: §4.1 · pp. 5–6
  • The experiments use OmniGen2 as the generator, two default optimization iterations, and SDXL-Base, SD2.1, SD1.4, and Promptist as baselines.OmniGen2 was selected for stronger conditioning fidelity and stability in reference-conditioned settings.Source: §4.1 · pp. 5–6

4.2 RESULTS

  • +5.8% on NICE, +2.4% on DiffusionDB, and +3.4% on Lexica are W2I’s overall gains; Accuracy-to-Prompt gains are +8.1%, +3.4%, and +6.4%, respectively.The table reports relative improvements over the next-best model for Accuracy-to-Prompt and Overall within each dataset group.Source: Table 1 · p. 6, §4.2 · p. 6
  • W2I outperforms all baselines on image-quality and human-preference evaluations, balancing visual fidelity with semantic accuracy.Table 2 compares human-preference evaluation metrics across datasets and models.Source: Table 2 · p. 7, §4.2 · p. 7
  • W2I outperforms all baselines across NICE’s five subcategories, including memes, real-time events, and niche intellectual property.Figure 5 reports overall LLM Grader scores across subcategories.Source: §4.2 · p. 7, Figure 5 · p. 8
  • The full pipeline performs best on NICE; image retrieval alone can over-condition generation, while prompt optimization alone improves textual alignment.The ablation reports gains across metrics when retrieval and prompt optimization are combined.Source: §4.2 · pp. 7–8, Table 3 · p. 8
  • Performance improves through ten optimization steps, with the sharpest gains in the first two; retrieval is often used early and prompt optimization later.The authors use two iterations by default to balance performance and efficiency.Source: §4.2 · p. 8

5 DISCUSSIONS

The discussion interprets strong gains on novel concepts as evidence that pretrained models have latent capacity activated by multimodal signals, and reports synergy between text- and image-based optimization. It also identifies retrieval quality, capability boundaries, and test-time overhead as limitations.

5 DISCUSSIONS

  • Strong gains on novel concepts suggest pretrained models have latent capacity that the right multimodal signals can activate.The authors connect this interpretation to interface improvements such as retrieval and adaptive prompting.Source: §5 · p. 8
  • The ablation study reports strong synergy between text- and image-based optimization, which harnesses their complementary strengths.Source: §5 · p. 8
  • Figure 6 tracks LLM-Grader overall and dimension sub-scores across optimization steps.The dotted line is the overall score; solid lines are individual dimensions.Source: Figure 6 · p. 9
  • Preliminary results suggest W2I can combine novel concepts from text and images across iterations, potentially supporting compositional generalization; scalability remains future work.Source: §5 · p. 9
  • W2I may degrade when relevant, high-quality reference images are unavailable; it cannot add fundamentally new base-model capabilities and incurs test-time overhead.The framework offers control knobs for balancing efficiency and quality.Source: §5 · p. 9

6 CONCLUSION

World-To-Image uses an agent to select between prompt refinement and web-retrieved visual grounding, injecting world knowledge without modifying the base model. It outperforms existing methods, with a reported +8.1% accuracy-to-prompt improvement on NICE.

6 CONCLUSION

  • +8.1% improvement in accuracy-to-prompt on the NICE benchmark containing diverse novel concepts.Source: §6 · p. 9
  • World-To-Image uses an Orchestrator agent to choose between language-space prompt refinement and vision-space grounding via web retrieval, without modifying the base model.Source: §6 · p. 9
  • The findings present interface improvement as an additional path toward more capable generative models, alongside model-size scaling.The framework is proposed as a basis for research on retrieval strategies and agentic reasoning.Source: §6 · p. 9

A WORLD-TO-IMAGE ALGORITHM

The algorithm initializes a prompt and image, then iteratively orchestrates prompt optimization and image retrieval before generating and scoring candidates against a convergence threshold.

A WORLD-TO-IMAGE ALGORITHM

  • The algorithm distinguishes the initial prompt, iteration-specific prompts and images, final selected image, and retrieved reference-image exemplars.Source: A WORLD-TO-IMAGE ALGORITHM · p. 13
  • The algorithm initializes the prompt, exemplar set, and first image, then repeats its steps up to the maximum iteration budget.It returns a final selected image.Source: A WORLD-TO-IMAGE ALGORITHM · p. 13
  • At each iteration, the Orchestrator selects whether to invoke prompt optimization or image retrieval; the corresponding agents refine prompts or retrieve external exemplars.Source: A WORLD-TO-IMAGE ALGORITHM · p. 13
  • The system generates an image from the refined prompt and exemplar conditioning, then evaluates it using a score function and convergence threshold.The score function may use LLMGrader, CLIP similarity, or an aesthetic score.Source: A WORLD-TO-IMAGE ALGORITHM · p. 13

B EXTENDED VISUAL COMPARISONS

Figures 7 and 8 qualitatively compare generations for diverse prompts across seven systems, with each row representing a prompt and columns ordered by model.

B EXTENDED VISUAL COMPARISONS

  • Figure 7 compares outputs for diverse prompts, with prompts in rows and outputs from seven systems in columns.The columns run from Ours through SD1.4.Source: Figure 7 · p. 14
  • Figure 8 uses the same row-per-prompt and seven-system column layout for qualitative generation comparisons.The columns run from Ours through SD1.4.Source: Figure 8 · p. 15

C EXTENDED TABLE

The extended tables report model-score comparisons across NICE subgroups and compare World-To-Image with OmniGen2 on three datasets.

C EXTENDED TABLE

  • Tables 4–8 compare model scores separately for the Meme, Real-time News & Events, Pop Culture & IP, Artists/Celebrities/Influencers, and Niche Concepts subgroups.Their captions indicate that the best scores are highlighted, but provide no cell values.Source: Table 4 · p. 16, Table 5 · p. 16, Table 6 · p. 16
  • Table 9 compares World-To-Image and OmniGen2 on Promptist Reward and Aesthetic Score across three datasets; lower Promptist Reward and higher Aesthetic Score are better.Source: Table 9 · p. 17
  • Table 10 reports LLM-Grader and human-preference metrics for World-To-Image across NICE benchmark subgroups.The supplied caption does not give subgroup values.Source: Table 10 · p. 17

D PROMPT TEMPLATES

This appendix presents prompt templates for routing image-generation and editing tasks, refining prompts, and retrieving relevant visual references. It also specifies concise image-search queries and a structured rubric for evaluating generated images across prompt accuracy and other visual and thematic qualities.

D.1 ORCHESTRATOR AGENT

  • The orchestrator selects among text-to-image, reference-guided generation, and two editing modes using the prompt, image, scores, and optimization history, then assigns prompt-optimizer and retrieval strategies.Its input also includes visual analysis; the output specifies the task type and strategies.Source: §D.1 · p. 18, INPUTS · p. 18, STRATEGY SELECTION · pp. 18–19
  • Niche entities or rare styles trigger reference-guided generation; incremental changes to a good prior image favor editing, specific IP matches use a reference, and major structural changes favor regeneration.The orchestrator prefers image editing only when the current image is already good and the prompt alone is unlikely to improve it.Source: GUIDELINES · p. 18, TASK CLASSIFICATION RULES · p. 18, DISAMBIGUATION (TEXT-ONLY PROMPTS THAT… · p. 18

D.2 PROMPT OPTIMIZER AGENT

  • For editing, the optimizer uses score and visual-analysis feedback to improve the existing image while preserving its core subject and scene; prompt-only edits target lighting, color, style, or composition.Reference edits identify elements needing external visual guidance.Source: §D.2 · pp. 21–22
  • The prompt optimizer preserves essential subjects, actions, and style, clarifies ambiguous concepts, adds composition or lighting detail only when useful, and keeps prompts concise and non-contradictory.For niche concepts, it refers to reference images by index and appends concise negative prompts when artifacts are likely.Source: §D.2 · pp. 21–22
  • The optimizer prioritizes subject, action, composition, lighting or camera, then style, using concrete visual vocabulary and brief artifact-specific negatives when scores or history suggest distortions.It avoids contradictory specifications.Source: §D.2 · p. 22

D.3 IMAGE RETRIEVAL AGENT

  • The retrieval agent evaluates reference candidates for query match, visual quality, usefulness, and distinctiveness, rating images and selecting at least one that meets the stated score threshold.It examines content, style, and context; selection is limited by the provided maximum.Source: §D.3 · p. 24
  • When an image search fails, the retrieval agent rewrites the query by simplifying niche terms, adding visual descriptors, and substituting more popular equivalents.The revised query is intended to improve the likelihood of finding relevant reference images.Source: §D.3 · p. 25
  • Retrieval guidance recommends synonyms and visual-element descriptions, illustrating this with reformulations of “Dr Strange,” “backroom,” and “cyberpunk hacker.”Examples replace named or abstract concepts with descriptions of visible subjects, settings, or features.Source: §D.3 · p. 25
  • The retrieval agent must return only a modified search query of 2–6 words likely to retrieve relevant images.Source: §D.3 · p. 25
  • The image-analysis prompt asks the grader to identify what the prompt captures or misses, mismatches, omitted elements, and quality improvements that preserve the original intent.Source: §D.3 · p. 25

D.4 LLM GRADER

  • The grader’s output is a JSON object with individual aspect scores and explanations plus an overall score.The template requests concise justifications and specifies the score and explanation fields for each aspect.Source: §D.4 · p. 26
  • The LLM grader scores prompt accuracy, creativity, visual quality and realism, consistency and cohesion, and emotional or thematic resonance from 0 to 10, with a short justification for each.The rubric evaluates how well the image matches the prompt, its visual execution, coherence, and intended mood or tone.Source: §D.4 · p. 26
  • The overall score is specified as either a weighted or simple average of the aspect scores.Source: §D.4 · p. 26

Related papers

Loading 2510.04201v1…