PaperlayerView exact revision on arXiv

Source-linked AI summary

WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing

Hui Zhang, Zongkai Liu, Liqiang Niu, Juntao Liu, Han Li, Zhen Cao, Wenchao Chen, Chengduo Zhao, Fandong Meng

arXiv:2609.05171v1cs.CV

TL;DR

Knowledge-intensive image generation and editing remain unreliable when required facts or appearances are absent from model parameters, and retrieval alone does not ensure reliable verification or evidence integration. WeAgent-MMGenEdit addresses this with a multimodal harness, verifiable data and benchmarking pipelines, and post-training for both policy and image backend. The resulting policy improves across generation and editing and approaches a trillion-parameter agent while using about 3% of its total parameters.

  • Problem

    Image generation and editing systems become unreliable when tasks require up-to-date or specialized facts and authentic visual appearances beyond their parametric knowledge.

  • Method

    WeAgent-MMGenEdit combines a persistent multimodal harness, dedicated visual verification, rendered evidence carriers, verifiable data construction, benchmarking, and two-sided post-training.

  • Results

    WeAgent-MMGenEdit consistently improves agentic process and final image quality across generation and editing, while its policy approaches a trillion-parameter agent with about 3% of its total parameters.

  • Takeaways & Limitations

    The recipe substantially outperforms existing open-source agentic generation and editing systems on WeBench-MMGenEdit and public benchmarks when combined with image-side post-training.

Abstract

from arXiv · show

Image generation and editing models have advanced rapidly, yet remain unreliable when prompts require external world knowledge. Bounded and long-tail parametric knowledge prevents direct or reason-then-generate approaches from recovering the required facts and visual appearances. Existing agentic generation and editing methods mitigate this limitation with retrieval tools, yet remain constrained by insufficient visual verification, overloaded policy models, and weak integration of retrieved textual and visual evidence. To address these limitations, we present WeAgent-MMGenEdit, a full-stack recipe including a multimodal harness, a scalable data construction pipeline, a comprehensive benchmark, and post-training methods for the agent policy and image backend. We first introduce WeAgent-Harness, a multimodal runtime with persistent evidence management and dedicated verification and integration tools that organize retrieved multimodal evidence into a dense carrier. Upon this, we develop a scalable pipeline for prompt synthesis and agentic trajectory collection, yielding 23K supervised trajectories and 14.7K RL tasks with three-layer verifiable checklists. We further introduce WeBench-MMGenEdit, a bilingual benchmark covering both knowledge-intensive image generation and multi-image editing. Finally, a two-sided post-training recipe based on SFT and RL improves the agent policy and image backend. Together, WeAgent-MMGenEdit enables a 30B-total/3B-active policy to outperform similarly sized policy models and approach the performance of a 1T-parameter agent.

1 Introduction

WeAgent-MMGenEdit targets knowledge-intensive image generation and editing, where parametric knowledge and retrieval-based agents struggle to recover and verify current facts and authentic appearances. It combines a multimodal harness, benchmark, data pipeline, and two-sided post-training, with strong reported gains across tasks.

  • Motivation: External knowledge tasks require models to acquire textual facts and visual evidence beyond their parameters and transfer them faithfully into generated pixels.Direct generation is constrained by training cutoffs and sparse long-tail knowledge, while reason-then-generate methods remain dependent on internalized knowledge.
  • Motivation: Retrieval-based agents still need reliable multimodal verification, filtering, and integration before generation.Existing systems may select visual candidates from metadata or only a few top-ranked results, limiting evidence reliability.
  • Approach: WeAgent-Harness persistently stores retrieved evidence, uses dedicated visual verification, and compiles verified text and images into a dense carrier for generation and editing.The runtime supports multi-hop generation and multi-image editing and is shared between inference and reinforcement-learning rollout.
  • Evaluation: 300 cases form a human-audited bilingual benchmark balanced across generation/editing and English/Chinese, with separate evaluation of process, generation input, and final image.The benchmark includes 150 generation and 150 editing cases, and 150 English and 150 Chinese cases.
  • Results: WeAgent-MMGenEdit improves agentic process and final image quality across generation and editing, while its policy approaches a trillion-parameter agent using about 3% of its total parameters.The gains transfer across multiple image backends and, with image-side post-training, substantially outperform existing open-source systems on WeBench-MMGenEdit and public benchmarks.

2 WeAgent-Harness: Multimodal Agentic Harness

WeAgent-Harness is a constrained multimodal runtime organized around retrieve, verify, integrate, and deliver. Persistent evidence references, explicit visual inspection, and rendered carriers support attributable decisions and structured transfer into generation or editing.

  • Runtime design: The harness organizes episodes into retrieve, verify, integrate, and deliver stages with persistent multimodal evidence management.Its design targets reliable verification and structured transfer of large-scale textual and visual evidence.
  • Runtime design: One-action-per-turn reasoning and tool use make decisions individually attributable for process supervision and reinforcement learning.Each turn produces either exactly one tool call or the final answer, followed by the tool output as the next observation.
  • Evidence memory: Stable identifiers let the policy accumulate, revisit, compare, and reuse evidence without repeatedly loading all pixels into context.This decouples available evidence from policy context occupancy and mitigates long-horizon context overload.
  • Toolchain: Seven tools implement the workflow: four retrieve textual or visual evidence, while three perform verification, integration, and image generation.The policy interface abstracts backend-specific details while preserving the retrieve–verify–integrate–deliver workflow.
  • Toolchain: Vision_Analyze selectively inspects registered images, including user inputs, retrieved candidates, rendered artifacts, and generated outputs.A separately deployed policy VLM performs visual inspection so the main agent can maintain a large evidence pool without additional reasoning overhead.
  • Toolchain: Execute_Code renders verified evidence into carriers that explicitly bind entities, attributes, and spatial positions before final generation.The carrier encodes relations in pixels rather than requiring the image model to infer them from prose and loosely attached references.
  • Delivery: Image_Generation supports generation without references and editing with one or more references through a unified interface.The same agent trajectory can be paired with different image generators without changing agent behavior.

3 WeAgent-MMGenEdit Dataset and Benchmark

The paper builds verifiable training data and a human-audited benchmark for bilingual, knowledge-intensive generation and editing. Both construction and evaluation separately assess the agentic chain, delivered generation input, and final image.

  • Dataset construction: WeDataset-MMGenEdit uses a three-layer specification covering the agentic process, generation input, and output image for verifiable supervision.The dataset targets what evidence should be acquired, how it reaches the generator, and whether it is faithfully rendered.
  • Dataset construction: 864K aligned bilingual concepts support task construction, with approximately 98% containing at least one image and 6.3 images per concept on average.A balanced pool of 346K concepts includes 200K with complete multimodal assets.
  • Dataset construction: A Task VLM jointly generates user requests, retrieval DAGs, three-layer checklists, model inputs, final images, and grounded sources.The checklists separately evaluate agentic recognition and retrieval, generation-input sufficiency, and rendered-image requirements.
  • Evaluation and routing: Expert trajectories execute synthesized tasks in WeAgent-Harness, after which isolated VLM judges score the agentic process, generation input, and final image.Checklist items use a 0–10 scale, and valid trajectories are routed according to independent quality gates.
  • Benchmark: 300 human-audited benchmark cases are balanced across 150 generation and 150 editing tasks and 150 English and 150 Chinese tasks.The benchmark spans eight domains and uses held-out, deduplicated cases selected for verifiable requirements and traceable evidence.
  • Benchmark: 94% of benchmark cases require at least five retrieval hops, while 69.3% of editing cases contain multiple user-provided images.The benchmark emphasizes deep multimodal retrieval and multi-image editing complexity.
  • Evaluation: The final-image score measures instruction adherence, textual- and visual-knowledge accuracy, content preservation, and visual quality.Generation and editing use separate weighted averages, with editing additionally weighting content preservation.

4 Agentic Post-Training

WeAgent-MMGenEdit post-trains its agent policy with supervised trajectories followed by checklist-grounded reinforcement learning. The recipe masks environment-provided content, evaluates valid grouped trajectories with isolated judges, and asynchronously optimizes policy-generated actions while decoupling image rendering from agentic RL.

  • Post-training strategy: Agentic post-training uses SFT followed by checklist-grounded RL to teach harness-protocol compliance, evidence acquisition, verification, and integration.SFT imitates high-quality tool-use trajectories, while RL improves beyond imitation using task-level checklist signals.
  • Policy formulation: Each task is represented as a multi-turn reason–act–observe trajectory whose terminal Image_Generation call produces a final prompt and ordered reference images.The policy generates reasoning and actions while receiving tool outputs as environment observations.
  • Stage I: Supervised fine-tuning: 23K expert trajectories supervise complete interleaved reasoning–action–observation sequences, with next-token loss applied only to policy-generated tokens.Tool observations and harness-injected content remain context rather than prediction targets.
  • Stage II: Checklist-grounded RL: RL samples G = 8 trajectories for each prompt, compares valid attempts within groups, and scores process quality and generation-input sufficiency with isolated judges.The reward combines Sinput, Sagentic, Srelative, and Sprotocol; invalid trajectories are excluded from optimization and group statistics.
  • Stage II: Checklist-grounded RL: GSPO updates valid trajectories using sequence-level advantages and policy-token masking, while rollout, reward evaluation, and actor optimization run asynchronously.The backend is excluded from agentic RL by using a placeholder image observation and rewarding the actual final prompt and selected references.

5 Image Editing Post-Training

The image backend is post-trained in two stages to handle heterogeneous multi-reference inputs and improve editing quality through supervised conditioning followed by multi-objective reinforcement learning.

  • Two-stage post-training: The pipeline first applies multi-reference flow-matching SFT, then Diffusion-NFT RL, producing WeEdit-M-SFT and WeEdit-M-RL.SFT establishes stable conditioning; RL further targets instruction following, text rendering, preservation, and visual quality.
  • Data construction: Editing supervision is distilled from terminal image-generation calls whenever at least one reference image is present, including nominal generation tasks with retrieved or rendered evidence.This uses the actual multimodal condition rather than the task’s nominal category.
  • Data construction: The collected data contains 50K SFT samples and 15K RL samples, balanced across one to five reference images.
  • Multi-reference conditioning: Independent resolution buckets preserve heterogeneous references’ aspect ratios and detail requirements instead of forcing them into a shared spatial shape.A frozen VAE supplies latent tokens, while a vision-language encoder provides semantic conditioning jointly with the instruction.
  • Stage I: SFT: SFT trains only on target-image tokens while keeping reference latents clean, establishing stable conditioning for subsequent reinforcement learning.
  • Stage II: RL: Diffusion-NFT RL samples multiple candidates per condition, compares them within groups, and reinforces relative preferences while regularizing toward the SFT reference.Invalid or unsuitable groups are masked, and the objective uses clipped normalized advantages.
  • Stage II: RL: The RL reward decomposes quality into instruction accuracy, text accuracy, preservation, aesthetics, and relative quality.The weighted reward assigns 0.35 to text accuracy, 0.25 to instruction accuracy, 0.10 to preservation, and 0.15 each to aesthetics and relative quality.

6 Experiment

Experiments show that explicit retrieval, visual verification, structured integration, and two-sided post-training improve knowledge-intensive generation and editing. WeAgent-RL transfers across backends, outperforms open-source agentic baselines, and approaches a much larger agent with a small active policy.

  • Harness contribution: 51.28→65.73 and 50.13→65.50 are the generation/editing W_Avg gains from replacing reason-then-generate with WeAgent-Harness.Visual knowledge accuracy rises from 42.43→56.86 on generation and 50.33→68.78 on editing.
  • Agent post-training: 12.45 and 16.72 points are WeAgent-RL’s W_Avg improvements over the untrained policy on generation and editing with GPT-Image-2.Knowledge dimensions improve much more than visual quality: TKA rises 20.82/23.72 and VKA 11.20/14.95, while VQ changes 2.41/1.93.
  • Existing agentic systems: 33.54/38.50 versus 25.35/24.64, 29.20/23.38, and 32.63/24.93 are the WeAgent-RL and competing open-source systems’ generation/editing W_Avg scores with Qwen-Image.The advantage is especially pronounced on editing, where prior systems provide little improvement over direct Qwen-Image.
  • Public benchmarks: 36.35 overall on KnowGen and 0.39 overall success on Mind-Bench are reported as the best results among evaluated agentic baselines.KnowGen gains are broad across visual correctness, text accuracy, and aesthetics; Mind-Bench knowledge-driven tasks average 0.44, with a World Knowledge score of 0.70.
  • Process and prompt scores: 35.48→59.17 and 28.03→52.71 are WeAgent-RL’s generation/editing Agentic Chain Avg improvements over the untrained baseline.Prompt Avg also rises from 32.17→55.54 and 21.80→50.85, with gains across retrieval, integration, and prompt sufficiency.
  • Harness ablation: 47.22/42.14 rises to 51.86/45.80 when search, verification, and integration are combined in the harness.Adding either verification or integration alone produces smaller gains, indicating complementary contributions beyond retrieval alone.
  • Image-side post-training: 33.54/38.50, 38.65/46.24, and 47.39/49.98 are the Gen./Edit. W-Avg scores as image-side training progresses from Qwen-Image through SFT and RL.Holding WeAgent-RL fixed, multi-reference SFT and subsequent image RL produce successive gains.

7 Related Work

The related work situates WeAgent-MMGenEdit among image generation, editing, multimodal agents, and knowledge-seeking systems. It emphasizes that existing approaches acquire external information but remain limited in multimodal verification and evidence integration.

  • Image generation and editing: Image generation synthesizes visual content from text, while image editing modifies existing content and may incorporate multiple reference images.References can control subject, appearance, style, or layout.
  • Multimodal agents: Multimodal agents extend language-model agents with visual observations, intermediate artifacts, and tool interaction across web and computer environments.They instantiate goal-directed perception, decision-making, and action with external tools.
  • Open limitation: Existing approaches remain limited in visual evidence, constraining their adaptability to agentic image generation and editing.The paper positions multimodal verification and integration as unresolved needs.
  • Agentic image generation and editing: Knowledge-seeking image-generation methods introduce search, browsing, visual retrieval, or structured intermediate representations before synthesis.These methods augment tool orchestration with external information acquisition.

8 Conclusion

WeAgent-MMGenEdit combines evidence acquisition, multimodal verification and integration, post-training, and multi-reference image generation and editing in one framework. Experiments report consistent gains across backends, benchmarks, and process-level evaluations, including near-1T-agent performance from a much smaller policy.

  • Framework: WeAgent-MMGenEdit jointly addresses external evidence acquisition, multimodal verification and integration, agentic post-training, and multi-reference generation and editing.Its components include WeAgent-Harness, checklist-grounded training and evaluation, and image-side SFT and reinforcement learning.
  • Conclusion: A 30B-A3B policy surpasses similarly sized models and approaches a 1T-parameter agent with about 3% of its parameters.The reported gains extend across diverse image backends, public benchmarks, and WeBench-MMGenEdit.

9 Appendix

The appendix supplies reproducibility configurations and complete trajectory examples spanning languages, task types, retrieval, verification, integration, and delivery. It also documents the agent- and image-side post-training setups.

  • Training configurations: Tables 7 and 8 summarize the main configurations required to reproduce agent-side and image-side post-training.Agent SFT uses interleaved reasoning–action–observation trajectories, while image training uses multi-reference SFT and Diffusion-NFT RL.
  • Agentic trajectories: Four complete trajectories cover English generation, multilingual tasks, GPU-image comparison, and Chinese financial-report editing.The examples expose the retrieve–verify–integrate–deliver workflow across different evidence requirements.
Loading 2609.05171v1…