Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesis

1 Introduction

• Existing unified models rely on fixed parametric knowledge, while loosely connected agent pipelines struggle to integrate evidence into visual guidance faithful to both source knowledge and user intent.Retrieved text may lack fine-grained visual attributes, and reference images may contain irrelevant elements or conflict with the requested composition.Source: §1 · pp. 1, 3
• Unify-Agent combines THINK, RESEARCH, RECAPTION, and GENERATE in one multimodal model, using recaptioning to turn retrieved evidence into structured constraints that preserve identity cues and user-requested composition.Source: §1 · p. 3
• 73.2 overall on FactIP, more than 22 points above Bagel, with Unify-Agent also outperforming FLUX.1-dev and SD-3.5-large.FactIP contains 2,462 curated prompts focused on rare identities and culturally significant long-tail concepts.Source: §1 · p. 3
• Unify-Agent’s ViT semantic tokens and VAE perceptual latents jointly support higher-fidelity interpretation of references and precise textual specifications for generation.FactIP evaluates identity consistency and factual faithfulness for rare and long-tail concepts.Source: §1 · p. 3
• Unify-Agent achieves superior performance among open-source unified models on FactIP, WiSE, KiTTEN, and T2I-FactualBench, approaching leading commercial models’ world-knowledge capabilities.Source: §1 · p. 3

2.1 Unified multimodal models

• Unified multimodal models jointly support visual understanding and image generation, but closed-book reliance on pretraining can cause hallucination or identity drift for rare, long-tail, or world-dependent entities.The cited work motivates an open-book approach that acquires external evidence before synthesis.Source: §2.1 · pp. 3–4

2.2 Agentic workflows

• Prior text-to-image agents often stitch frozen planners, search tools, and standalone generators through fragile APIs, whereas Unify-Agent integrates reasoning, evidence grounding, and generation planning in one framework.The passage attributes cascading errors and separation of multimodal reasoning from synthesis to shallow integration.Source: §2.2 · p. 4

2.3 World knowledge and factual evaluation.

• Conventional benchmarks prioritize aesthetics and prompt alignment, while factual benchmarks expose knowledge gaps but remain diagnostic; Unify-Agent pairs targeted evaluation with evidence-grounded synthesis.The passage names T2I-FactualBench (Huang et al., 2024b), KITTEN (Huang et al., 2024a), and WISE (Niu et al., 2025) as knowledge-oriented benchmarks.Source: §2.3 · p. 4

3.1 Unified Multimodal Model: Bagel

• Unify-Agent builds on Bagel (Deng et al., 2025b), whose Mixture-of-Transformers architecture uses a ViT encoder and dedicated experts to unify multimodal understanding and continuous image generation.The understanding expert uses autoregressive next-token prediction, while the generation expert uses rectified flow in a continuous VAE latent space.Source: §3.1 · pp. 4–5

3.2 Motivating Evidence for World-Grounded Synthesis

• In a training-free study of 200 FactIP examples, both text and visual injection improved over prompt-only Bagel, with visual injection yielding larger gains and joint injection slightly weaker than visual alone.The study covered scene, character, and object categories; the figure reports absolute performance improvements but the supplied passages give no numerical values.Source: Figure 2 · p. 5, §3.2 · p. 5
• Raw textual and visual evidence can introduce noise or conflicting constraints, motivating recaptioning that consolidates multimodal evidence into a structured, generation-oriented description.Raw text may interfere with instruction following, while raw visual anchors can over-constrain attribute manipulation and compositional reasoning.Source: §3.2 · p. 5, §3.3 · p. 6

3.3 Problem Formulation

• Unify-Agent formulates synthesis as a trajectory that detects knowledge gaps, acquires textual and visual evidence, recaptions it, and generates an image from the recaption and visual anchors.The final synthesis step is conditioned exclusively on the recaption and visual evidence, excluding the noisy reasoning history and textual trace.Source: §3.3 · p. 6

4.1 Training Data Construction

• Each sample pairs the user prompt with textual and visual research traces and a grounded recaption, training both evidence acquisition and conversion into a synthesis-ready specification.Unlike generic responses or captions, the structured recaption preserves identity-critical attributes alongside user-specified scene and style constraints.Source: §4.1 · p. 6

4.1.1 Task Source and Prompt Collection

• The curated pool contains 456K long-tail IP examples across 12 domains, with verified metadata and seed images supporting category-tailored prompts from direct rendering to compositional variations.Prompts preserve generation intent while varying pose, costume, scene, lighting, or aesthetic style.Source: §4.1.1 · p. 7

4.1.2 Multimodal Research Trace Construction

• Claude Opus 4.6 constructs multistep research trajectories, with textual research preceding visual search to resolve semantic ambiguity and establish high-level grounding.The textual query and evidence summary form a semantic scaffold for downstream reasoning and visual search.Source: §4.1.2 · p. 8
• Visual queries combine prompt constraints with textual grounding, and Gemini 3 Flash ranks candidates on four quality dimensions to select the top two references for recaptioning.The dimensions are identity consistency, subject salience, image clarity, and watermark cleanliness; supervising queries and selected references teaches structured visual grounding.Source: §4.1.2 · pp. 8–9

4.1.3 Evidence-Grounded Recaption Annotation

• Evidence-grounded recaptioning consolidates the instruction with textual context and identity-preserving visual cues into a generation-oriented specification that preserves identity while controlling scene and style.It serves as a structured interface for heterogeneous evidence rather than a generic caption or simple prompt paraphrase.Source: §4.1.3 · p. 9
• 143K high-quality trajectory-image pairs remain after generation-based validation, providing supervision for the complete evidence-grounded path from research through recaptioning to synthesis.Trajectories that fail identity-consistency checks are retried for up to five trials and discarded if no satisfactory result is produced.Source: §4.1.3 · pp. 9–10

4.2.1 Benchmark Construction

• FactIP is held out from supervised fine-tuning and uses a 500-sample test split selected from 2,500 manually filtered candidates to assess diverse, long-tail concepts requiring factual grounding.Samples pair an instruction with two seed images and metadata; filtering removes noisy information, unclear instructions, low-quality references, and ambiguous identities.Source: §4.2.1 · p. 10

4.2.2 Evaluation Criterion

• FactIP evaluates generated images on Clarity, Content, Aesthetics, and Relevance against the instruction and seed-image reference, with Relevance emphasizing identity cues and instruction consistency.The full scoring protocol and weighting scheme are deferred to Appendix C.Source: §4.2.2 · p. 10

5.1 Unified Fine-Tuning on Multimodal Agent Trajectories

• Unify-Agent is fine-tuned on interleaved multimodal agent trajectories packed for joint text and image-generation training.Its dual objective combines token-level autoregressive text supervision with latent flow-matching image supervision; separate masks and hybrid attention regulate information flow across trajectory segments.Source: §5.1 · pp. 10–11

5.2 Unify-Agent: Inference-Time Multimodal Agentic Pipeline

• The agent decomposes prompts into meaningful factors and identifies missing or unreliable visually consequential attributes that determine whether external knowledge should be invoked.Examples include facial structure, hairstyle, signature appearance, object-specific structure, and scene-specific factual details.Source: §5.2 · p. 12
• Unify-Agent searches sequentially, using textual evidence for semantic grounding and background knowledge to guide retrieval of identity-relevant, context-compatible reference images.The textual trace informs a more precise visual query, conditioning retrieval on both prompt intent and semantic grounding.Source: §5.2 · p. 12
• The agent converts retrieved evidence into identity-preserving and scene-compositional constraints, then integrates them into an evidence-grounded recaption for synthesis.Identity constraints capture target-specific visual attributes; scene-compositional constraints encode factors such as pose, environment, garment, mood, and composition.Source: §5.2 · p. 12
• Unify-Agent reframes world-grounded synthesis as an active sequence of Think, Research, Recaption, and Generate rather than prompt-only image generation.The pipeline identifies missing knowledge, acquires external evidence, reorganizes it into a synthesis-ready representation, and produces an image grounded in world knowledge and user intent.Source: §5.2 · p. 13

6.1 Experimental Details

• The experiments compare commercial, generation-only, and unified MLLM baselines, using ChatGPT-4o judges for WiSE, T2I-FactualBench, and Kitten and Seed2.0 for FactIP.The baselines include proprietary and open-source models across multiple paradigms.Source: §6.1 · p. 13

6.2 Main Results

• 73.2 is Unify-Agent’s highest Overall score among unified MLLMs on FactIP, with Relevance scores of 67.3, 71.8, and 78.2 across Character, Object, and Scene.It also exceeds FLUX.1-dev (28.9) and SD-3.5-large (27.5).Source: §6.2 · pp. 13–14, Table 1 · p. 14
• 0.77 is Unify-Agent’s best Overall WiSE score among unified MLLMs, exceeding BAGEL+CoT (0.70) and leading across most individual knowledge domains.It scores 0.82 in cultural, 0.72 in biological, and 0.70 in chemistry knowledge.Source: §6.2 · p. 14, Table 2 · p. 15
• 4.08 gives Unify-Agent a new KiTTEN state of the art across generation-only and unified MLLMs, exceeding Imagen-3 (3.50), with text alignment 4.22 and entity alignment 3.93.The benchmark covers eight categories and evaluates text and entity alignment.Source: §6.2 · p. 14, Table 3 · p. 15
• 77.4 in SKCI and 71.5 in MKCC are Unify-Agent’s top overall scores within unified MLLMs on T2I-FactBench; its SKCM concept score is 69.2 versus DALLE-3’s 55.5.The benchmark assesses single-concept memorization, instantiation, and multi-concept composition with interaction.Source: §6.2 · p. 15, Table 4 · p. 16

6.3.1 Ablation Study

• The full model improves FactIP Overall from 50.9 to 73.2 over vanilla Bagel, while removing text search, image search, or recaptioning lowers Overall to 65.4, 56.2, or 62.9.Removing image search lowers Relevance from 72.4 to 50.8; identity-preserving constraints support Relevance, while scene-compositional constraints support Content and Aesthetics.Source: Table 5 · p. 16, §6.3.1 · pp. 16–17

6.4.1 Visualization comparison results

• On knowledge-intensive and attribute-sensitive prompts, Unify-Agent qualitatively better preserves subject identity, attributes, and prompt details than Flux-1, Bagel-7b, Hunyuan, and Stable Diffusion.Examples span historical figures, fictional entities, fine-grained products, and stylized toys; competing methods often show identity drift, missing attributes, or generic substitutes.Source: Figure 5 · p. 17, §6.4.1 · p. 18

6.4 Finding: Why Generation Helps Understanding in a Unified Model

• Recaptioning in a unified model is a multimodal understanding task: removing ViT substantially harms performance, while removing VAE causes a more moderate decline.ViT supplies higher-level semantic visual representations, while VAE preserves low-level appearance cues; their joint availability supports better recaptions and grounded synthesis.Source: §6.4 · pp. 17–18

• Unify-Agent combines cognitive gap detection, multimodal evidence acquisition, grounded recaptioning, and final synthesis instead of treating image generation as a closed-book prompt-to-image mapping.Source: §7 · p. 18
• By resolving missing world knowledge with retrieved textual and visual evidence, Unify-Agent enables more faithful generation of rare, long-tail, and knowledge-intensive concepts.Source: §7 · p. 18
• The findings characterize world-grounded image synthesis as a distinct multimodal reasoning problem requiring tightly coupled understanding, acting, and generation.Source: §7 · p. 18
• The work aims to motivate future research on more reliable, open-world generative agents.Source: §7 · p. 18

Limitations and Future Work

• Current open-source unified multimodal models remain weaker than the strongest closed-source systems, and Bagel supports only a relatively small number of images in one context.Bagel’s limited long-context capability constrains more complex agent behaviors.Source: Limitations and Future Work · p. 18
• The current pipeline is a shallow one-pass workflow, not an iterative agent that interleaves text-image search, reflection, and replanning.These behaviors are described as crucial for harder open-world tasks such as travel planning or academic report generation.Source: Limitations and Future Work · p. 18
• Despite these limitations, results provide encouraging evidence for the feasibility of end-to-end unified agent training..Source: Limitations and Future Work · p. 18
• The work highlights the advantage of unified generation and reasoning models, in which generation and understanding can mutually reinforce each other.Source: Limitations and Future Work · p. 18
• Future work will validate the findings on stronger unified backbones and extend the framework toward more capable multimodal agents.Source: Limitations and Future Work · p. 18
• The planned agents will support longer-horizon planning, repeated search, and adaptive reasoning in more complex real-world settings.Source: Limitations and Future Work · p. 18

A Implementation Details

• SFT used 64 NVIDIA H20 GPUs for approximately 10 days and 10k gradient steps.Source: A Implementation Details · p. 24
• Training jointly updated the language model, ViT encoder, and multimodal connectors while keeping the continuous-generation VAE weights frozen.Source: A Implementation Details · p. 24
• The objective combines CE loss for autoregressive text and visual-token prediction with equally weighted MSE loss on image-reconstruction latents.Both losses use a weight of 1.0.Source: A Implementation Details · p. 24
• CE loss reweighting increases penalties on designated special tokens; optimization uses 500 linear warmup steps, a 5 × 10−5 peak learning rate, and gradient clipping at 5.0.The learning-rate scheduler is constant after warmup, and the clipping limit is a maximum L2 norm of 5.0.Source: A Implementation Details · p. 24

B Attention Masking Strategy

• Agentic dialogs and reasoning steps use causal masking to preserve the conversation’s temporal and logical flow.Source: B Attention Masking Strategy · p. 25
• Retrieved reference images, represented by composite VAE and ViT tokens, use full attention for global visual-feature interaction.Source: B Attention Masking Strategy · p. 25
• During generation, latent flow-matching tokens attend only to preceding recaption tokens and retrieved reference images, filtering out historical dialogs and reasoning traces.This restricted generation-phase mask is intended to prevent high-noise text from interfering with synthesis and keep it grounded in recaptions and visual evidence.Source: Figure 6 · p. 25, B Attention Masking Strategy · p. 25

C Evaluation Protocols

• For all existing benchmarks, we preserve their original MLLM-based evaluation settings, including designated expert evaluators and scoring schemes, to maintain consistency with prior literature and improve fairness and comparability.Source: C Evaluation Protocols · p. 25

WiSE Evaluation

• WiSE uses GPT-4o (Team, 2024b) to score Consistency, Realism, and Aesthetic Quality from 0 to 2, combining them in WiScore with weights 0.7, 0.2, and 0.1.Consistency is prioritized as the core measure of world-knowledge alignment; Realism and Aesthetic Quality remain auxiliary dimensions.Source: C Evaluation Protocols · p. 26

KiTTEN Evaluation

• KiTTEN uses GPT-4o (Team, 2024b) to independently score Text Alignment and reference-image-based Entity Alignment from 1 to 5, keeping the dimensions separate.This enables analysis of trade-offs between entity-specific fidelity and adherence to complex textual constraints.Source: C Evaluation Protocols · p. 26

T2I-FactBench Evaluation

• T2I-FactBench uses three GPT-4o VQA rounds across SKCM, SKCI, and MKCC: concept factuality, instantiation completeness, and MKCC composition factuality.Concept factuality checks shape, color, texture, and feature details; instantiation completeness requires both concept presence and successful realization of the instantiation phrase; composition factuality assesses seamless transition, visual completeness, authenticity, and prompt following.Source: C Evaluation Protocols · p. 26

FactIP Benchmark Evaluation

• FactIP uses Seed2.0 to score Clarity, Content, Aesthetics, and Relevance from 0 to 10, jointly comparing the output with the prompt and two references.The weighted score assigns Relevance a 0.75 weight and scales the final result to [0, 100], prioritizing stable IP identity while allowing variation across references.Source: C Evaluation Protocols · p. 27

D.1 Construction

• The benchmark organizes Character, Object, and Scene into 12 fine-grained subcategories, as shown in its hierarchical taxonomy.Source: Figure 7 · p. 27, §D.1 · p. 27, Table 7 · p. 28

D More Details about FactIP benchmark

• Figure 7(b) presents category-wise method comparisons as a radar chart of overall scores across all subcategories.Source: Figure 7 · p. 27

D.1 Construction

• FactIP is constructed to evaluate generative models’ ability to synthesize high-fidelity intellectual property and culturally significant concepts with factual grounding.Source: §D.1 · p. 27
• The full FactIP benchmark contains 2,462 curated prompts, according to its detailed category statistics.Source: Table 7 · p. 28, §D.1 · p. 28
• Prompts challenge models to generate recognizable entities, including global celebrities, intricate art toys, and iconic landmarks.Source: §D.1 · p. 28
• FactIP-Mini samples 500 prompts from the full dataset while preserving its original category distribution.Source: §D.1 · p. 28

D.2 Results

• On FactIP-Mini, Unify-Agent shows highly competitive, balanced performance across taxonomic branches; Figure 7(b) and Table 8 provide category-wise and per-subtask comparisons.The supplied passage gives no table values, so no numerical scores or specific winners are reported.Source: §D.2 · p. 28
• Unified MLLMs and commercial models generally outperform earlier generation-only approaches in IP fidelity and factual consistency, underscoring the need for multimodal reasoning and factual grounding.Figure 7(b) visualizes category-wise strengths and weaknesses on FactIP-Mini.Source: §D.2 · p. 28

E Prompt

• The prompt expert must produce one image-generation prompt describing who the IP is, what they are doing, and the scene.The scene should be a specific, realistic location fitting the IP’s profession, and the action should align with their public persona; impossible scenarios are disallowed unless the IP is a fictional fantasy character.Source: PROMPT-GENERATION_PROMPT · p. 33
• Prompt and tag language depend on nationality: Chinese IPs require Chinese, while non-Chinese IPs require English.The corresponding language codes are zh and en, respectively.Source: PROMPT-GENERATION_PROMPT · p. 33
• Prompts should vary their sentence structure, using scene-first, action-focused, or descriptive forms rather than repeating one template.The instructions give examples such as beginning with the scene or describing the IP amid contextual surroundings.Source: PROMPT-GENERATION_PROMPT · p. 33
• The response must contain only three XML tags: Image_Prompt, Tag_Name, and Language, with the tag name written in the selected language.No markdown, explanation, or conversational filler is allowed.Source: PROMPT-GENERATION_PROMPT · pp. 33–34

Case 1: Science - Copper Combustion

• For copper combustion, text search verified copper’s green flame, image search supplied vetted visual references, and recaptioning guided a scientifically accurate depiction.The references emphasized an emerald-green core and organic flame tendrils. Figure 8 shows the generated image.Source: F Image Generation Showcases · p. 36, Figure 8 · p. 37

Case 2: Art Toy - DUDOO Tasty Afternoon Tea Series

• For the DUDOO Tasty Afternoon Tea Series, a second text search filled gaps about Dudu’s visual identity before image references informed an IP-accurate popcorn-bucket recaption.The recaption included the toy’s matte resin texture. Figure 9 shows the generated art toy.Source: F Image Generation Showcases · p. 36, Figure 9 · p. 38

Case 3: Celebrity - William Butler Yeats

• For William Butler Yeats, biographical search and historical portraits grounded his appearance, which recaptioning placed in a rustic cottage study writing a manuscript.Verified details included pince-nez glasses, angular bone structure, and tousled hair. Figure 10 shows the generated scene.Source: F Image Generation Showcases · p. 36, Figure 10 · p. 39

Case 4: Celebrity - Grigory Perelman

• For Grigory Perelman, text and image searches grounded his reclusive context and distinctive appearance, while recaptioning integrated age-related hair changes into a sparsely furnished apartment scene.The scene also included scattered papers and soft natural light. Figure 11 shows Perelman scribbling mathematical equations.Source: F Image Generation Showcases · p. 36, Figure 11 · p. 40

Case 5: Celebrity - Bruce Beutler

• For Bruce Beutler, text search established his immunology and genetics credentials, while judged portrait references grounded his appearance in an advanced laboratory scene.The recaption incorporated his receding gray-silver hair and mustache; Figure 12 shows him reviewing data on a monitor.Source: F Image Generation Showcases · p. 36, Figure 12 · p. 41

G FactIP Benchmark Evaluation Showcases

• 9 overall score for Gregg Popovich’s image reflects facial resemblance and basketball-context consistency, including Spurs jerseys and a tactical whiteboard.The evaluator compares the synthesis with GT1 and GT2 and provides a fine-grained scoring profile and interpretable reasoning trace.Source: G FactIP Benchmark Evaluation Showcases · p. 42, Figure 13 · p. 43
• 5 overall score for the Max Weber image, despite clarity 9 and aesthetic appeal 8, as the evaluator flags identity mismatches and a historically anachronistic microphone.The trace notes a beard discrepancy and glasses absent from GT1/GT2, showing how strong surface quality can coexist with semantic and identity errors.Source: G FactIP Benchmark Evaluation Showcases · p. 42, Figure 14 · p. 44
• 9 overall score for Habatan in a Baroque style reflects character fidelity and stylistic execution, while the trace catches a shoe-color mismatch with GT1.The generated image has two blue shoes, whereas the reference specifies one red shoe and one blue shoe; the evaluator also recognizes chiaroscuro and painterly brushwork.Source: G FactIP Benchmark Evaluation Showcases · p. 42, Figure 15 · p. 45
• 2 overall score for the Scottie Pippen image reflects poor identity resemblance, visual artifacts, and illegible, incorrect Chicago Bulls branding and number 33.The evaluation penalizes failures in character identity, image integrity, and fine-grained text faithfulness.Source: G FactIP Benchmark Evaluation Showcases · p. 42, Figure 16 · p. 46