Source-linked AI summary
GEMS: Agent-Native Multimodal Generation with Memory and Skills
Zefeng He, Siyuan Huang, Xiaoye Qu, Yafu Li, Tong Zhu, Yu Cheng, Yang Yang
TL;DR
Multimodal generation models still struggle with complex instructions and specialized downstream tasks. GEMS addresses this gap with iterative agentic refinement, persistent compressed memory, and on-demand domain skills, achieving gains across mainstream and downstream tasks and enabling 6B Z-Image-Turbo to surpass Nano Banana 2 on GenEval2.
Problem
Multimodal generation models remain limited on intricate multi-faceted instructions and specialized downstream applications despite strong general-purpose performance.
Method
GEMS combines a structured Agent Loop, persistent trajectory-level Agent Memory with hierarchical compression, and an extensible Agent Skill repository with on-demand loading.
Results
Across five mainstream and four downstream tasks, GEMS consistently improves performance; with Z-Image-Turbo it gains 14.22 and 14.03 average points respectively and surpasses Nano Banana 2 on GenEval2.
Takeaways & Limitations
GEMS extends the capabilities of lightweight and varied generative backends through agentic reasoning, trajectory memory, and domain-specific expertise.
Takeaways & Limitations
Despite using lightweight Z-Image-Turbo, Agent Loop introduces noticeable inference latency and future work targets lower computational overhead.
Abstract
from arXiv · showhide
Recent multimodal generation models have achieved remarkable progress on general-purpose generation tasks, yet continue to struggle with complex instructions and specialized downstream tasks. Inspired by the success of advanced agent frameworks such as Claude Code, we propose \textbf{GEMS} (Agent-Native Multimodal \textbf{GE}neration with \textbf{M}emory and \textbf{S}kills), a framework that pushes beyond the inherent limitations of foundational models on both general and downstream tasks. GEMS is built upon three core components. Agent Loop introduces a structured multi-agent framework that iteratively improves generation quality through closed-loop optimization. Agent Memory provides a persistent, trajectory-level memory that hierarchically stores both factual states and compressed experiential summaries, enabling a global view of the optimization process while reducing redundancy. Agent Skill offers an extensible collection of domain-specific expertise with on-demand loading, allowing the system to effectively handle diverse downstream applications. Across five mainstream tasks and four downstream tasks, evaluated on multiple generative backends, GEMS consistently achieves significant performance gains. Most notably, it enables the lightweight 6B model Z-Image-Turbo to surpass the state-of-the-art Nano Banana 2 on GenEval2, demonstrating the effectiveness of agent harness in extending model capabilities beyond their original limits.
1 Introduction
GEMS addresses the difficulty of applying multimodal generation models to complex instructions and specialized downstream tasks through three integrated pillars: Agent Loop, Agent Memory, and Agent Skill. Across nine tasks and multiple generative backends, it reports consistent gains, including a headline comparison on GenEval2.
- General-purpose multimodal models perform strongly on straightforward prompts but struggle with intricate instructions and specialized downstream applications.
- Existing iterative and multi-agent systems can provide insufficient guidance or excessive redundancy by relying on single-step updates or accumulated historical context.
- GEMS combines Agent Loop, persistent hierarchically compressed Agent Memory, and extensible on-demand Agent Skill to refine outputs and provide domain-specific expertise.
- 9 diverse tasks validate GEMS across five mainstream benchmarks and four specialized downstream tasks, with generalizability tested across multiple generative backends.
- 14.22 average gain on mainstream benchmarks and 14.03 on downstream tasks are achieved with Z-Image-Turbo, while the 6B model surpasses Nano Banana 2 on GenEval2.
- The architecture augments the user prompt with skills, iteratively refines it in the Agent Loop, and uses Agent Memory to manage historical context.
2 Related Works
Related work has progressed from basic inference-time prompt optimization toward reasoning- and agent-based systems that iteratively improve multimodal generation. Agent research similarly advances from self-correcting loops to collaborative multi-agent frameworks.
- Inference-time scaling methods include prompt rewriting, random search, Chain-of-Thought reasoning, and iterative optimization for multimodal generation.
- Agent systems extend language-model reasoning and execution through structured planning, interaction, self-correcting loops, and specialized roles communicating in multi-agent frameworks.
3 Method
GEMS organizes multimodal generation as an iterative pipeline that plans and decomposes prompts, generates and verifies images, then refines prompts using persistent memory and domain-specific skills. Hierarchical compression preserves trajectory information while limiting redundancy, and on-demand skill loading supports scalable specialization.
- GEMS comprises Agent Loop, Agent Memory, and Agent Skill, which collaborate on complex instruction following and specialized downstream tasks.
- Agent Loop: Agent Loop coordinates Planner, Decomposer, Generator, Verifier, and Refiner modules.
- Agent Loop: The Planner retrieves triggered domain skills, synthesizes an enhanced prompt, and sends the original prompt to the Decomposer for evaluation planning.
- Agent Loop: The Decomposer partitions the original prompt into atomic visual requirements represented as binary semantic or structural probes.
- Agent Loop: At each iteration, the model-agnostic Generator synthesizes an image from the current optimized prompt.
- Agent Loop: The Verifier uses an MLLM to assess each image against atomic criteria and produce binary feedback for refinement or termination.
- Agent Loop: When criteria remain unsatisfied, the Refiner combines the current prompt, image, verification feedback, internal reasoning, and prior memory to produce the next prompt.
- Agent Memory: Agent Memory stores the optimization trajectory while hierarchical compression distills verbose reasoning traces into concise experiences and retains factual states with strategic reflections.
4 Experiments
GEMS is evaluated across mainstream and downstream multimodal generation tasks, with ablations examining its components, iterative optimization, memory design, skills, and efficiency. It achieves consistent gains, including improved refinement trajectories and reduced iteration counts.
- Main Results: 14.22 average normalized-score increase on mainstream tasks with Z-Image-Turbo demonstrates consistent gains over prior inference-time scaling baselines.Qwen-Image-2512 further validates effectiveness across a different foundational architecture.
- Main Results: 14.03 average normalized-score improvement on downstream tasks with Z-Image-Turbo surpasses the best inference-time scaling baseline’s +8.92.Prompt-rewriting baselines degrade on some tasks, particularly text rendering.
- Ablation Studies: 63.5 GenEval2 score results from adding Agent Loop, Agent Memory, and Agent Skill to a 31.0 baseline, with respective gains of 21.4, 9.0, and 2.1 points.The ablation uses Z-Image-Turbo and averages results over three independent runs.
- Ablation Studies: 71.4% passed criteria on GenEval2 after rising from 62.2% across iterations, indicating directed optimization rather than merely random variation.GEMS shows a consistent upward trajectory, unlike the fluctuating gains of a basic Agent Loop Only approach.
- Memory Analysis: 2.5 points are gained by compressing raw reasoning logs into concise Experiences, whereas including full thoughts produces negligible gains.Historical prompts and feedback add 3.4 points, while generated images add a further 3.1 points.
- Efficiency and Skills: 2.80 average iterations, down from 3.26, follows Agent Memory and Agent Skill improvements while GEMS maintains lower overhead through early stopping.At approximately three generated images per task, GEMS substantially outperforms other inference-time scaling methods.
5 Conclusion
GEMS reframes text-to-image generation as iterative optimization by combining iterative refinement, persistent trajectory-level memory, and domain-specific skills. With Z-Image-Turbo, it improves performance across mainstream and downstream tasks and surpasses Nano Banana 2 on GenEval2.
- GEMS reframes text-to-image generation as an iterative optimization problem.
- The framework unifies iterative refinement, persistent trajectory-level memory, and domain-specific skills.
- 14.22 average gain across five mainstream benchmarks and 14.03 across four specialized downstream tasks are achieved with Z-Image-Turbo.
- Z-Image-Turbo surpasses Nano Banana 2 on GenEval2 despite using a 6B foundational model.
A Experiment Details
The appendix provides prompts for GEMS’s LLM-based modules and identifies the programmatic managers that support skills and memory.
- Detailed prompts are provided for the Planner, Decomposer, Verifier, Refiner, and Compressor modules.
- The Skill Manager and Memory Manager are implemented as programmatic modules rather than LLM agents.
A.2 Evaluation Details
Evaluation uses official Z-Image and Qwen-Image settings and includes configuration details for Nano Banana 2 to support comparison.
- Evaluations follow the official settings of Z-Image and Qwen-Image.
- Nano Banana 2 configuration is also presented for comprehensive comparison.
A.3 Benchmark Details
The appendix describes the evaluation details, downstream benchmarks, and inference-time scaling baselines used for comparison.
- Table 3 summarizes the evaluation details for the compared models.
- LongText-Bench, SpatialGenEval, CREA, and ArtiMuse evaluate text rendering, spatial intelligence, creative drawing, and aesthetic drawing, respectively.
- The study compares GEMS with inference-time scaling baselines including Rewrite, Promptist, Random Search, Maestro, and CRAFT.
- Rewrite enhances initial prompts with an MLLM, while Promptist automatically rephrases prompts using a fine-tuned language model and beam search.
- Random Search generates multiple images and selects the best according to verifier-assigned scores.
- Maestro iteratively generates two images per round and uses pairwise comparison to evolve outputs.
- CRAFT refines prompts from MLLM feedback and targeted updates without specialized memory management.
B Result Details
The results section provides fine-grained benchmark breakdowns to compare inference-time scaling strategies across specific generation capabilities.
- Scores are reported separately for GenEval, GenEval2, DPG-Bench, OneIG, WISE, SpatialGenEval, and CREA.The breakdown covers seven benchmarks and follows each benchmark’s official evaluation scripts.
- The analysis examines how different inference-time scaling strategies affect individual generation capabilities.
- Benchmark subsets are used to provide a more comprehensive performance comparison than aggregate results alone.
C Qualitative Results
Qualitative results compare GEMS with the baseline using identical prompts and provide fine-grained benchmark dimensions for mainstream and downstream evaluation.
- Qualitative comparisons: GEMS and the baseline are compared on generated outputs using the same prompts.Examples include an Earth-from-the-moon view and a liquid-water butterfly formed from a splash.
- Mainstream benchmark dimensions: GenEval measures single-object, two-object, counting, color, position, and attribute-binding capabilities.
- Mainstream benchmark dimensions: GenEval2 evaluates object, attribute, count, position, and verb dimensions.
- Mainstream benchmark dimensions: DPG-Bench and OneIG evaluate global/entity/relation factors and alignment/text/reasoning/style/diversity factors, respectively.
- Downstream benchmark dimensions: WISE, SpatialGenEval, and CREA cover scientific domains, spatial relations, and creative-quality dimensions with benchmark-specific categories.WISE includes cultural, temporal, spatial, biological, physical, and chemical dimensions; SpatialGenEval and CREA use their listed fine-grained categories.
D Limitations and Future Work
The paper identifies constraints in efficiency, workflow autonomy, modality coverage, and editing support, while outlining corresponding future directions.
- Efficiency: Agent Loop’s iterative workflow causes noticeable inference latency despite using lightweight, distilled Z-Image-Turbo.Future work targets workflow optimization to reduce computational overhead and improve efficiency.
- Agent autonomy: The current system depends on predefined workflows to coordinate agent collaboration and module interactions.The authors propose investigating tool interfaces for more autonomous memory and skill management.
- Modality scope: GEMS is primarily designed for image generation, with future research considering more complex multimodal domains such as video generation.
- Editing support: Z-Image-Turbo and Qwen-Image-2512 currently do not support image editing.The proposed direction is to use more versatile models for a unified reasoning, generation, and editing loop.