PaperlayerView exact revision on arXiv

Source-linked AI summary

Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesis

Shuang Chen, Quanxin Shou, Hangting Chen, Yucheng Zhou, Kaituo Feng, Wenbo Hu, Yi-Fan Zhang, Yunlong Lin, Wenxuan Huang, Mingyang Song, Dasen Dai, Bolin Jiang, Manyuan Zhang, Shi-Xue Zhang, Zhengkai Jiang, Lucas Wang, Zhao Zhong, Yu Cheng, Nanyun Peng

arXiv:2603.29620v2cs.CVcs.MM

TL;DR

Unified multimodal models struggle with rare, long-tail, knowledge-intensive concepts because they rely mainly on frozen parametric knowledge. Unify-Agent addresses this by coupling prompt understanding, multimodal evidence search, grounded recaptioning, and synthesis, achieving stronger factual-generation performance while approaching leading commercial models’ world knowledge. Its current scope remains limited by base-model capacity and a shallow one-pass workflow.

  • Problem

    Unified multimodal models rely primarily on frozen parametric knowledge, limiting faithful image generation for rare, long-tail, and knowledge-intensive concepts.

  • Method

    Unify-Agent uses a unified multimodal agent that detects information gaps, researches textual and visual evidence, recaptions it into structured constraints, and generates the final image.

  • Results

    Unify-Agent achieves the highest Unified MLLM Overall score of 73.2 on FactIP and sets superior records across several factual benchmarks, approaching leading commercial models’ world-knowledge capabilities.

  • Takeaways & Limitations

    World-grounded image synthesis is a multimodal reasoning problem requiring tightly coupled understanding, acting, and generation.

  • Takeaways & Limitations

    The pipeline remains limited to a shallow one-pass workflow and is constrained by open-source models’ weaker long-context capability and limited image-context capacity.

Abstract

from arXiv · show

Unified multimodal models provide a natural and promising architecture for understanding diverse and complex real-world knowledge while generating high-quality images. However, they still rely primarily on frozen parametric knowledge, which makes them struggle with real-world image generation involving long-tail and knowledge-intensive concepts. Inspired by the broad success of agents on real-world tasks, we explore agentic modeling to address this limitation. Specifically, we present Unify-Agent, a unified multimodal agent for world-grounded image synthesis, which reframes image generation as an agentic pipeline consisting of prompt understanding, multimodal evidence searching, grounded recaptioning, and final synthesis. To train our model, we construct a tailored multimodal data pipeline and curate 143K high-quality agent trajectories for world-grounded image synthesis, enabling effective supervision over the full agentic generation process. We further introduce FactIP, a benchmark covering 12 categories of culturally significant and long-tail factual concepts that explicitly requires external knowledge grounding. Extensive experiments show that our proposed Unify-Agent substantially improves over its base unified model across diverse benchmarks and real world generation tasks, while approaching the world knowledge capabilities of the strongest closed-source models. As an early exploration of agent-based modeling for world-grounded image synthesis, our work highlights the value of tightly coupling reasoning, searching, and generation for reliable open-world agentic image synthesis.

1 Introduction

Unify-Agent reframes world-grounded image synthesis as a unified agentic process that detects knowledge gaps, searches multimodal evidence, recaptions it, and generates images. It targets factual and visual fidelity for rare, long-tail concepts while reporting strong performance across factual benchmarks.

  • World-grounded image synthesis requires factual and visual fidelity for real people, cultural symbols, rare IPs, historical scenes, and scientific phenomena.
  • Existing agentic systems separate evidence acquisition, multimodal reasoning, and synthesis, making fine-grained evidence integration difficult.
  • Unify-Agent tightly integrates THINK, RESEARCH, RECAPTION, and GENERATE within one unified multimodal model.
  • Evidence-grounded recaptioning filters retrieved knowledge into identity-preserving and scene-compositional constraints for generation.
  • Unify-Agent sets superior records among open-source unified models across FactIP, WiSE, KiTTEN, and T2I-FactualBench, approaching leading commercial models’ world-knowledge capabilities.

2 Related Works

Related work shows that unified multimodal models harmonize understanding and image generation but remain closed-book, while conventional benchmarks often underemphasize factual correctness. These limitations motivate open-book, knowledge-oriented generation and evaluation.

  • Unified multimodal models jointly support visual understanding and image generation through shared backbones.
  • Current unified multimodal models rely on static parametric memory and frequently hallucinate or suffer identity drift for rare, long-tail, or world-dependent entities.
  • Recent text-to-image agent research shifts reasoning toward action-oriented planning with tools and environmental interaction.
  • Conventional text-to-image benchmarks emphasize aesthetics and generic prompt alignment, overlooking factual correctness.

3 Preliminary

The preliminary analysis builds Unify-Agent on Bagel’s unified understanding-and-generation architecture and examines how external evidence can address missing world knowledge. It finds that raw multimodal injection can be suboptimal, motivating structured evidence-grounded recaptioning and an interleaved generative formulation.

  • 3.1 Unified Multimodal Model: Bagel: Bagel uses a Mixture-of-Transformers architecture with a ViT encoder to unify multimodal understanding and continuous image generation.
  • 3.1 Unified Multimodal Model: Bagel: Bagel formulates understanding as autoregressive next-token prediction conditioned on multimodal context C.
  • 3.1 Unified Multimodal Model: Bagel: Bagel formulates image synthesis as rectified flow in the continuous VAE latent space, using a time-conditioned velocity field.
  • 3.2 Motivating Evidence for World-Grounded Synthesis: Rare IP failures primarily reflect missing world knowledge about appearance, identity-defining attributes, and their realization under composition.
  • 3.2 Motivating Evidence for World-Grounded Synthesis: Figure 2 compares text, visual, and joint text+visual injection against prompt-only Bagel across FactIP Scene, Character, and Object categories.
  • 3.2 Motivating Evidence for World-Grounded Synthesis: Naive raw evidence injection can introduce formatting noise, redundant information, conflicting constraints, or excessive visual constraints, motivating a structured generation-oriented interface.
  • 3.3 Problem Formulation: The proposed formulation models synthesis as an interleaved trajectory with cognitive gap assessment, textual evidence, visual evidence, and evidence-grounded recaption variables.

4 Data Pipeline

The data pipeline constructs supervision for the full world-grounded generation process, from long-tail task curation and multimodal research traces to evidence-grounded recaption annotation. It produces training samples for trajectory learning and supports the FactIP benchmark.

  • 4 Data Pipeline: The pipeline combines training-data construction for agent supervised fine-tuning with a curated FactIP evaluation benchmark.FactIP evaluates generation quality in clarity, content, aesthetics, and relevance.
  • 4 Data Pipeline: Each sample supervises the original prompt, textual and visual research traces, and a final evidence-grounded recaption for synthesis.Textual and visual traces supervise external evidence acquisition, while the recaption converts acquired evidence into a synthesis-ready specification.
  • 4 Data Pipeline: The corpus is built through task-source curation, prompt collection, multimodal research-trace construction, and evidence-grounded recaption annotation with generation-based verification.
  • 4.1.2 Multimodal Research Trace Construction: The multimodal research trace explicitly teaches the agent to formulate queries, gather heterogeneous evidence, and organize it for recaptioning and image synthesis.Textual research precedes visual research so semantic grounding can support more precise, context-aware visual search.
  • 4.1.3 Evidence-Grounded Recaption Annotation: Evidence-grounded recaptioning consolidates prompt constraints, textual context, and identity-preserving visual cues into a unified textual control signal for generation.The recaption preserves identity-critical attributes while maintaining scene-level and stylistic controllability.
  • 4.1.3 Evidence-Grounded Recaption Annotation: 143K high-quality trajectory-image pairs are obtained after recaptioning and verification, providing supervision over the complete evidence-grounded reasoning path.

5 Methodology

Unify-Agent trains and runs world-grounded image synthesis as a unified sequential process that detects missing visual knowledge, researches multimodal evidence, recaptions it into constraints, and generates an image. Its design keeps reasoning, evidence acquisition, and synthesis tightly connected rather than treating generation as a single-shot prompt-to-image mapping.

  • Unified Fine-Tuning on Multimodal Agent Trajectories: Unify-Agent learns from interleaved multimodal trajectories containing reasoning traces, tool actions, recaptioning outputs, and image-generation targets.Supervised fine-tuning frames world-grounded generation as a unified sequential decision process.
  • Unified Fine-Tuning on Multimodal Agent Trajectories: The training objective combines language modeling for multimodal understanding with latent-space regression for image generation.Separate supervision masks activate the two objectives within packed sequences, while hybrid attention regulates information flow across trajectory components.
  • Unify-Agent: Inference-Time Multimodal Agentic Pipeline: At inference time, the agent interprets prompts as partially specified requests and identifies missing identity-critical or visually consequential attributes before deciding whether to research them.If the missing-information set is nonempty, the model enters research; otherwise, it may proceed directly to generation.
  • Unify-Agent: Inference-Time Multimodal Agentic Pipeline: The research stage acquires textual evidence before visual evidence, using semantic grounding to refine retrieval of identity-relevant and context-compatible images.The sequential ordering supports identity disambiguation and more informed visual queries.
  • Unify-Agent: Inference-Time Multimodal Agentic Pipeline: Retrieved evidence is converted into identity-preserving and scene-compositional constraints, then integrated into an evidence-grounded recaption for final synthesis.The constraints preserve target-specific visual traits while specifying pose, environment, garment, mood, composition, and presentation; the recaption serves as the generator’s executable specification.

6 Experiments

Unify-Agent achieves strong factual and knowledge-grounded image synthesis across FactIP, WiSE, KiTTEN, and T2I-FactBench, outperforming unified and generation-only baselines. Ablations and qualitative comparisons attribute these gains to integrated search, recaptioning, multimodal evidence, and constraint-aware generation.

  • FactIP: 73.2 Overall on FactIP is Unify-Agent’s highest score among unified MLLMs, with Relevance scores of 67.3, 71.8, and 78.2 for Character, Object, and Scene.FactIP evaluates Clarity, Content, Aesthetics, and Relevance for culturally significant concepts and intellectual properties.
  • WiSE: 0.77 Overall on WiSE places Unify-Agent above BAGEL+CoT (0.70) among unified models, with strongest domain scores in cultural (0.82), biological (0.72), and chemistry (0.70) knowledge.WiSE covers cultural, time, space, biology, physics, and chemistry knowledge dimensions.
  • KiTTEN: 4.08 Overall on KiTTEN establishes Unify-Agent as state of the art, exceeding Imagen-3 (3.50) while reaching 4.22 text alignment and 3.93 entity alignment.The benchmark evaluates fine-grained text and entity alignment across eight categories.
  • T2I-FactBench: Unify-Agent leads unified models on T2I-FactBench’s SKCI (77.4) and MKCC (71.5), while its SKCM concept score of 69.2 compares favorably with DALLE-3 (55.5).The three tiers measure single-concept memorization, instantiation, and multiple-concept composition with interaction.
  • Ablation Study: The full model improves FactIP Overall from 50.9 to 73.2 over vanilla Bagel, with the largest gain in Relevance, from 44.9 to 72.4.The ablation indicates the principal advantage is factual grounding and identity fidelity rather than generic image quality alone.
  • Ablation Study: Removing search or recaptioning components reduces FactIP performance, with image-search removal lowering Overall to 56.2 and Relevance from 72.4 to 50.8.Text search provides semantic scaffolding, while image search supports fine-grained identity and appearance details.

7 Conclusion

Unify-Agent reframes world-grounded image synthesis as an inference-time process that resolves missing world knowledge through multimodal evidence and grounded recaptioning. The framework couples understanding, acting, and generation to support more faithful synthesis of rare, long-tail, and knowledge-intensive concepts.

  • Unify-Agent replaces closed-book prompt-to-image generation with cognitive gap detection, multimodal evidence acquisition, grounded recaptioning, and final synthesis.
  • Retrieved textual and visual evidence is used to resolve missing world knowledge before image synthesis.
  • The paper frames world-grounded image synthesis as a multimodal reasoning problem requiring tightly coupled understanding, acting, and generation.

Limitations and Future Work

The authors identify limitations in backbone capability, context length, and workflow depth. Current results focus on IP- and concept-centric synthesis, while broader iterative agent behaviors remain future work.

  • Current open-source unified multimodal models remain weaker than the strongest closed-source systems, limiting complex agent behaviors.Bagel has limited long-context capability and supports only a relatively small number of images in one context.
  • The demonstrated pipeline remains a shallow one-pass workflow rather than an iterative process with interleaved search, reflection, and replanning.The authors identify these longer-horizon behaviors as important for harder tasks such as travel planning or academic report generation.
  • The current results are limited to IP- and concept-centric world-grounded synthesis.

A Implementation Details

Training uses a unified multimodal model with joint optimization across language, vision, and connectors, while a hybrid attention strategy separates sequential reasoning from visual grounding during generation.

  • SFT uses an interleaved image-and-text dataset and runs for 10k gradient steps over approximately 10 days on 64 NVIDIA H20 GPUs.
  • The training objective jointly updates the language model, ViT encoder, and multimodal connectors while keeping the continuous-generation VAE frozen.
  • Textual dialogs and reasoning steps use causal attention, whereas retrieved reference-image tokens use full attention for global interaction.
  • During generation, a restricted mask filters historical reasoning noise so flow-matching tokens attend only to relevant reference images and recaptions.

C Evaluation Protocols

The evaluation protocols preserve benchmark-specific settings while combining complementary dimensions of image quality, knowledge alignment, factuality, and entity fidelity. Several metrics explicitly prioritize consistency or relevance to the intended world-grounded concept.

  • Evaluation design: Existing benchmarks retain their official MLLM-based evaluators and scoring schemes to improve consistency, fairness, and comparability.
  • WiSE Evaluation: WiSE evaluates Consistency, Realism, and Aesthetic Quality, with WiScore defined as their weighted combination.
  • WiSE Evaluation: WiSE uses weights α1 = 0.7, α2 = 0.2, and α3 = 0.1, prioritizing Consistency over Realism and Aesthetic Quality.
  • KiTTEN Evaluation: KiTTEN decouples fine-grained entity fidelity into Text Alignment and Entity Alignment.
  • T2I-FactBench Evaluation: T2I-FactBench uses three progressive GPT-4o VQA rounds covering concept factuality, instantiation completeness, and composition factuality.
  • FactIP Benchmark Evaluation: FactIP compares generated images with prompts and two reference images across Clarity, Content, Aesthetics, and Relevance.
  • FactIP Benchmark Evaluation: FactIP assigns α1 = 0.05, α2 = 0.10, α3 = 0.10, and α4 = 0.75, explicitly prioritizing Relevance.

D More Details about FactIP benchmark

FactIP is a hierarchical benchmark for evaluating factual image synthesis across three broad domains and 12 fine-grained subcategories. It contains 2,462 curated prompts and a 500-prompt distribution-preserving mini version.

  • Benchmark structure: FactIP organizes evaluation into Character, Object, and Scene domains with 12 fine-grained subcategories.The taxonomy is designed to cover diverse intellectual properties and culturally significant concepts.
  • Benchmark scale: 2,462 curated prompts challenge models to generate recognizable entities including celebrities, art toys, and iconic landmarks.
  • Benchmark structure: FactIP-Mini samples 500 prompts from the full dataset while preserving its original category distribution.

D.2 Results

FactIP results are reported through category-wise comparisons, prompt-generation rules, and controlled XML-formatted outputs. The benchmark analysis emphasizes balanced performance across taxonomic branches and realistic, profession-aligned prompts.

  • Results: Unify-Agent demonstrates competitive and balanced performance across Character, Object, and Scene branches on FactIP-Mini.The radar chart summarizes category-wise strengths and weaknesses, while Table 8 provides per-subtask scores.
  • Language rules: Non-Chinese IPs use English prompts, English tags, and the language code en, while Chinese IPs use Chinese prompts, tags, and zh.
  • Prompt generation: Prompt construction requires naming the IP, specifying a realistic scene, and describing a dynamic action aligned with the subject’s profession or public persona.The instructions also require syntactic diversity rather than repeatedly using one sentence structure.
  • Output format: Generated outputs must contain only Image_Prompt, Tag_Name, and Language XML tags without explanations or conversational filler.
  • Examples: Examples depict Robert Downey Jr. in a talk-show interview and Gordon Ramsay plating a dish in a high-end restaurant kitchen.

F Image Generation Showcases

The showcases demonstrate world-grounded image generation for scientific phenomena, art toys, and notable people. Examples combine text search, image search, and detailed scene descriptions to support factual and visually specific synthesis.

  • Science: The copper-combustion case uses text search to verify copper’s green flame reaction before targeted image search for visual dynamics.
  • Art toy: The DUDOO showcase uses iterative text search to move from general product information toward the character’s visual identity and iconic traits.
  • Celebrities: The Yeats, Perelman, and Beutler cases use biographical or professional text search followed by image references to ground identity and setting details.Examples include Yeats’s pince-nez glasses, Perelman’s scholarly appearance, and Beutler’s laboratory context.
  • Celebrities: Showcase outputs depict Yeats writing by an oil lamp, Perelman writing equations, and Beutler reviewing genetic data on a monitor.

G FactIP Benchmark Evaluation Showcases

FactIP evaluation showcases show that MLLM-based assessment distinguishes identity and contextual fidelity from surface image quality. Examples include high-scoring faithful generations and low-scoring cases with semantic, historical, or identity errors.

  • Successful evaluations: A Gregg Popovich example receives an overall score of 9 after evaluation verifies facial resemblance and basketball-related attributes.
  • Failure cases: A Max Weber image receives an overall score of 5 despite clarity 9 and aesthetic appeal 8 because of facial mismatch and a modern historical anachronism.
  • Successful evaluations: A Baroque-style Habatan generation receives an overall score of 9 for strong character fidelity and convincing stylistic execution.
  • Failure cases: A Scottie Pippen example receives an overall score of 2 because of weak identity resemblance, compromised image quality, and failures involving the requested jersey.
Loading 2603.29620v2…