Source-linked AI summary
Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generation
Zekai Zhang, Jiahao Li, Jie Zhang, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Shengming Yin, Tianhe Wu, Xiaoyue Chen, Xiao Xu, Yan Shu, Yanran Zhang, Yixian Xu, Yuxiang Chen, Zhendong Wang, Zihao Liu, Zikai Zhou, Huishuai Zhang, Dongyan Zhao, Chenfei Wu
TL;DR
Real-world image requests often leave essential context implicit, unavailable, or dependent on prior interactions, limiting current text-to-image systems. Qwen-Image-Agent progressively constructs generation context through planning and grounding, and achieves state-of-the-art results across IA-Bench, MindBench, and WISE-Verified, including an 82.6% improvement over Qwen-Image-2.0 on MindBench.
Problem
Real-world image generation remains insufficiently studied beyond rendering because requests may be underspecified, require external knowledge, demand decomposition, or depend on prior context.
Method
Qwen-Image-Agent progressively constructs generation context through context-aware planning and grounding, while IA-Bench evaluates Plan, Reason, Search, and Memory capabilities.
Results
82.6% improvement over Qwen-Image-2.0 on MindBench accompanies state-of-the-art performance across IA-Bench, MindBench, and WISE-Verified.
Takeaways & Limitations
The work provides a context-centric perspective and practical guidance for building image agents that address real-world user needs beyond direct prompt rendering.
Takeaways & Limitations
Some context gaps remain too implicit to identify reliably, making performance dependent on the intelligence of the MLLM backbone.
Abstract
from arXiv · showhide
While text-to-image (T2I) models have achieved remarkable progress, they struggle with real-world requests that are often underspecified, implicit, or dependent on up-to-date knowledge. We identify this challenge as the Context Gap: the mismatch between the user context and the sufficient generation context for T2I models. To bridge this gap, we propose Qwen-Image-Agent, a unified agentic framework that integrates plan, reason, search, memory and feedback in a context-centric manner. Qwen-Image-Agent treats user input as partial context and progressively constructs the generation context through Context-Aware Planning and Context Grounding. Specifically, Context-Aware Planning identifies missing context and plans how it should be acquired and used, while Context Grounding gathers this context from reason, search, memory, and feedback. To evaluate agentic image generation, we further introduce Image Agent Bench (IA-Bench), a benchmark covering four core image agent capabilities: Plan, Reason, Search, and Memory. Experiments on IA-Bench, Mindbench and WISE-Verified show that Qwen-Image-Agent outperforms strong baselines and achieves state-of-the-art performance.
1 Introduction
Real-world image generation exposes a Context Gap: T2I models optimized for fully specified prompts must handle underspecified requests requiring intent inference, current knowledge, visual references, and interaction history. Qwen-Image-Agent addresses this gap with context-aware planning and grounding, while IA-Bench evaluates the core capabilities needed for agentic image generation.
- Motivation: Real-world requests are often underspecified, requiring T2I models to infer intent, retrieve current knowledge or visual references, and use interaction history.This reflects the mismatch between fully specified training prompts and practical deployment conditions.
- Motivation: The Context Gap is the mismatch between provided user context and the generation context required by T2I models.The paper frames this gap as a fundamental challenge and motivation for agentic image generation.
- Approach: Qwen-Image-Agent progressively constructs full generation context by integrating plan, reason, search, memory, and feedback through Context-Aware Planning and Context Grounding.Context-Aware Planning identifies missing context and determines how it should be acquired and used.
- Evaluation: IA-Bench evaluates Plan, Reason, Search, and Memory across 17 real-world tasks, 730 test instances, and 1801 fine-grained binary criteria.The benchmark addresses limitations of evaluations focused primarily on rendering or isolated knowledge and reasoning.
- Results: Experiments on IA-Bench, WISE-Verified, and MindBench show that Qwen-Image-Agent substantially outperforms strong agentic baselines and achieves state-of-the-art results.Ablation studies further verify complementary benefits from different grounded contexts.
2 Related Work
Recent image generation research adds planning, reasoning, search, memory, and feedback capabilities, while existing methods remain fragmented in how they handle generation context. Benchmarks have progressed from instruction following and text–image alignment toward knowledge-driven and reasoning-driven evaluation.
- Agentic Image Generation: Recent methods extend image generation and editing with planning, reasoning, search, memory, and self feedback capabilities.Planning decomposes complex intents; search improves grounding in open-world scenarios; memory supports long-horizon interactions; and feedback studies test-time scaling.
- Agentic Image Generation: Existing methods remain fragmented in identifying, acquiring, and using the context required for real-world image generation, whereas Qwen-Image-Agent unifies these capabilities.The unified framework integrates plan, reason, memory, search, and feedback in a single context-oriented approach.
- Image Generation Benchmarks: Image-generation benchmarks evolved from instruction following and text–image alignment to knowledge-driven and reasoning-driven settings.GenEval evaluates compositional attribute binding, DPGBench evaluates dense prompt following, and WISE and PhyBench evaluate domain knowledge and physical commonsense grounding.
3 Qwen-Image-Agent Framework
Qwen-Image-Agent bridges the context gap between incomplete user context and the complete generation context required for successful image rendering. It progressively constructs this context through Context-Aware Planning and Context Grounding, using planning, reasoning, search, memory, and feedback.
- Framework formulation: The framework treats image generation as conditional rendering and progressively builds the generation context before final rendering.The agent maintains a state, takes context-gathering actions, and accumulates intermediate results along a trajectory.
- Context-Aware Planning: Context-Aware Planning operates at information, content, and generation levels to identify missing context, construct prompts, and allocate context across scenarios.It rewrites prompts around subjects, attributes, layouts, styles, and textual elements, while managing context in multi-turn and multi-image generation.
- Information-level Planning: Information-level planning raises explicit questions about missing generation information and routes each question to reasoning, web search, or image search.This planning stage identifies the context gap before grounding gathers the required information.
- Context Grounding: Context Grounding collects and organizes missing context through reasoning, search, memory, and feedback.Reasoning makes implicit requirements explicit, search supplies current facts and visual references, and memory retrieves prior-turn or external knowledge.
- Grounding via Feedback: Feedback grounding evaluates generated images against a planned checklist and feeds failed attributes back into the context for subsequent generation.A VLM assesses each result against expected image attributes, converting failed items into feedback context.
4 IA-Bench
IA-Bench evaluates agentic image-generation capabilities beyond rendering, covering planning, reasoning, external knowledge search, and cross-turn memory. It provides a structured, checklist-based framework with 4 tasks, 17 subtasks, 730 instances, and 1801 evaluation items.
- Benchmark scope: IA-Bench addresses underspecified, knowledge-dependent, multi-step, and context-dependent image-generation requests through four capabilities: Plan, Reason, Search, and Memory.These capabilities target challenges beyond instruction following, visual fidelity, and aesthetic quality.
- Benchmark scope: The benchmark contains 4 tasks, 17 subtasks, 730 instances, and 1801 evaluation checklist items.Together, these components structure evaluation across planning, reasoning, search, and memory dimensions.
- Capability dimensions: Planning tasks test decomposition into visual arrangements, reasoning tasks test latent-constraint inference, search tasks test external knowledge grounding, and memory tasks test context reuse across turns.The task families include Composition, Enumeration, Multi-Panel; Math, Science, Commonsense, Maze, Map, Geometry; IP and Information; and User Profile and Conversation History.
- Benchmark construction: IA-Bench uses human-annotated prompts and refined checklists, filtering memorization-solvable instances and using dynamic references for memory-oriented interactions.Annotators generate checklist candidates with LLMs, then manually review and refine them for correctness and necessity.
- Evaluation protocol: Pass Rate measures strict all-checklist success, while Checklist Accuracy measures the average proportion of satisfied items, jointly capturing complete and partial compliance.A VLM evaluates whether each generated image satisfies its associated checklist items.
- Evaluation protocol: IA-score = 0.3 × Plan + 0.3 × Reason + 0.3 × Search + 0.1 × Memory.The weighted aggregate emphasizes Plan, Reason, and Search while treating Memory as a complementary measure of cross-step consistency and context retention.
5 Experiments
Experiments evaluate Qwen-Image-Agent across IA-Bench, WISE-Verified, and MindBench, showing strong agentic, world-knowledge, and reasoning performance. Qualitative results, ablations, and limitations further characterize its capabilities and remaining challenges.
- Benchmarks: IA-Bench evaluates Plan, Reason, Search, and Memory, while WISE-Verified measures semantic understanding and world knowledge and MindBench tests dynamic external knowledge and multi-step reasoning.These three benchmarks provide complementary coverage of core image-agent capabilities and knowledge-intensive generation.
- IA-Bench Results: Qwen-Image-Agent achieves the highest IA-score and improves Q-score from 17.4 to 45.4 over direct generation with Qwen-Image-2.0.It also performs strongly against closed-source and other agentic image-generation baselines across Plan, Reason, and Search.
- IA-Bench Results: Agentic generation models consistently outperform direct-generation models on Plan, Reason, and Search, while closed-source models retain an advantage in Memory.This comparison highlights both the value of agentic processing and a remaining weakness in agentic memory capabilities.
- Cross-Benchmark Results: Qwen-Image-Agent achieves state-of-the-art performance on both WISE-Verified and MindBench, surpassing Nano Banana Pro on WISE-Verified.WISE-Verified emphasizes world knowledge, whereas MindBench focuses on complex reasoning and external knowledge use.
- Qualitative Results: Qualitative comparisons show that Qwen-Image-Agent improves complex real-world generation by progressively transforming incomplete user context through its agentic pipeline.It can infer maze trajectories, retrieve stock information, generate specified spiral layouts, and verify object attributes and composition.
- Ablations: Removing grounded context lowers its corresponding evaluation dimension, while replacing GPT-5.5-0424 with Qwen-Plus and Qwen-VL-Max substantially degrades most metrics.The ablations support the context design and show that MLLM intelligence is critical to the overall system.
6 Conclusion
The paper identifies the context gap as a central challenge in real-world image generation and addresses it with Qwen-Image-Agent, a unified context-centric agentic framework. It also introduces IA-Bench to evaluate four core capabilities of agentic image generation.
- The context gap is identified as a central challenge in real-world image generation.
- Qwen-Image-Agent integrates plan, reason, search, memory, and feedback within a unified context-centric framework.
- IA-Bench systematically evaluates four core agentic image-generation capabilities: Plan, Reason, Search, and Memory.
A Appendix · A.1 Case Study
The case studies demonstrate Qwen-Image-Agent’s agentic image-generation capabilities across planning, reasoning, web and image search, feedback, multi-image generation, and memory. They show the system acquiring or organizing context to solve structured, knowledge-dependent, and multi-turn generation tasks.
- A.1 Case Study: Qwen-Image-Agent demonstrates reasoning by solving a maze through reasoning about the concrete path.
- A.1 Case Study: Qwen-Image-Agent demonstrates web search by retrieving external knowledge to generate a 45° top-down urban miniature scene.The requested scene uses vertical isometric 3D cartoon styling, realistic materials and lighting, weather-integrated architecture, and minimalist composition.
- A.1 Case Study: Qwen-Image-Agent demonstrates image search by retrieving visual references for Gary De'Snake from Zootopia 2 wearing a red and green scarf under a Christmas tree.
- A.1 Case Study: Qwen-Image-Agent demonstrates feedback through self-correction on a parking-lot composition containing 5 red cars, 3 blue cars, and 2 black cars.
- A.1 Case Study: Qwen-Image-Agent demonstrates multi-image generation by splitting and allocating generation context for a 3-slide renewable-energy deck.The requested deck uses Henri Matisse paper cut-outs, bold organic shapes, vibrant colors, and slide-specific renewable-energy content.
- A.1 Case Study: Qwen-Image-Agent demonstrates memory by selecting relevant memory context for a multi-turn product-image task involving a white mini portable projector.The first turn specifies a keystone correction dial, HDMI port, speaker grille, and the text "Brightness: 200 ANSI Lumens".