PaperlayerView exact revision on arXiv

Source-linked AI summary

GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning

Chengqi Duan, Rongyao Fang, Yuqing Wang, Kun Wang, Linjiang Huang, Xingyu Zeng, Hongsheng Li, Xihui Liu

arXiv:2505.17022v2cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

Complex prompts with multiple objects, precise spatial relations, and bound attributes remain difficult for visual generation models. GoT-R1 applies reinforcement learning to autoregressive visual generation with MLLM rewards for reasoning and images, reporting significant gains on T2I-CompBench and GenEval.

  • Problem

    Visual generation models struggle with complex prompts that require multiple objects to have precise spatial relationships and attributes, and template-supervised reasoning can remain unfaithful to prompts.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    GoT-R1 uses dual-stage reinforcement learning with MLLM rewards to assess intermediate reasoning and final images, enabling strategy discovery beyond predefined patterns.

    Source: §1 · pp. 2–3

  • Results

    GoT-R1 reports significant improvements over existing methods on T2I-CompBench and GenEval, particularly for precise spatial relationships and attribute binding.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The reported gains support GoT-R1 as an approach to improving compositional image generation for prompts involving spatial relations and attribute binding.

    Source: §1 · p. 3

    For prompts with more than ten items, accurate corresponding bounding-box generation declines, potentially reducing fidelity in very complex scenes.

    Source: C LIMITATIONS · p. 16

Abstract

from arXiv · show

Visual generation models have made remarkable progress in creating realistic images from text prompts, yet struggle with complex prompts that specify multiple objects with precise spatial relationships and attributes. Effective handling of such prompts requires explicit reasoning about the semantic content and spatial layout. We present GoT-R1, a framework that applies reinforcement learning to enhance semantic-spatial reasoning in visual generation. Building upon the Generation Chain-of-Thought approach, GoT-R1 enables models to autonomously discover effective reasoning strategies beyond predefined templates through carefully designed reinforcement learning. To achieve this, we propose a dual-stage multi-dimensional reward framework that leverages MLLMs to evaluate both the reasoning process and final output, enabling effective supervision across the entire generation pipeline. The reward system assesses semantic alignment, spatial accuracy, and visual quality in a unified approach. Experimental results demonstrate significant improvements on T2I-CompBench benchmark, particularly in compositional tasks involving precise spatial relationships and attribute binding. GoT-R1 advances the state-of-the-art in image generation by successfully transferring sophisticated reasoning capabilities to the visual generation domain. To facilitate future research, we make our code and pretrained models publicly available at https://github.com/gogoduan/GoT-R1.

1 INTRODUCTION

GoT-R1 applies reinforcement learning to autoregressive visual generation so models can discover semantic-spatial reasoning strategies beyond fixed templates. It combines reasoning-process and image rewards and reports benchmark gains.

  • Complex prompts challenge models with precise spatial relations and attribute binding, while fixed supervised templates can yield prompt-unfaithful reasoning.Source: §1 · p. 1
  • GoT-R1 adapts reinforcement learning to autoregressive visual generation, aiming for autonomous semantic-spatial reasoning beyond predefined patterns.Source: §1 · pp. 2–3
  • Its dual-stage MLLM reward framework evaluates intermediate reasoning and final images across semantic alignment, spatial accuracy, and image quality.Source: §1 · pp. 2–3
  • MLLM rewards assess prompt-to-reasoning semantics and layout, reasoning-to-image fidelity, and prompt-to-image quality; GRPO supports strategy discovery.Visualizing bounding boxes on a blank canvas improves reliability of the prompt-to-reasoning spatial reward.Source: §1 · p. 2
  • GoT-R1 significantly improves performance on T2I-CompBench and GenEval, especially on compositional tasks requiring precise spatial relations and attribute binding.Source: §1 · p. 3

2 RELATED WORK

Prior work spans diffusion and autoregressive generation, unified multimodal models, and reinforcement learning for reasoning. These lines of work leave open how reasoning capabilities translate into compositional image generation.

  • Diffusion and autoregressive generators have advanced, yet both still struggle with complex scene understanding, precise spatial relations, and multi-object attribute binding.Source: §2 · p. 3
  • Unified MLLMs show strong understanding, but prior unified or generative models had not demonstrated that reasoning improves generation quality.Source: §2 · p. 3
  • RL improves language-model reasoning; GRPO normalizes group rewards without a separate critic, while T2I-R1 (Jiang et al., 2025) uses BiCoT-GRPO and vision-expert rewards.Source: §2 · p. 3

3 METHOD

GoT-R1 extends Generation Chain-of-Thought with reinforcement learning and a unified MLLM reward framework. Its method combines staged training with rewards for intermediate reasoning and generated-image alignment.

3.1 PRELIMINARY

  • GoT decomposes complex prompts into semantic descriptions and spatial coordinates in an explicit reasoning chain before image generation.The method builds on this representation to guide object attributes and spatial arrangements.Source: §3.1 · p. 4
  • Hand-crafted reasoning templates constrain GoT’s strategy discovery, while supervised fine-tuning can produce unfaithful chains that bottleneck later image generation.Source: §3.1 · p. 4
  • GRPO samples outputs, scores them, normalizes rewards within each group into advantages, and updates the policy using an optimization objective.The group-based approach uses relative performance differences and avoids a separate critic model.Source: §3.1 · pp. 4–5

3.2 GOT-R1 FRAMEWORK

  • GoT-R1 uses a unified autoregressive MLLM and two training stages: supervised fine-tuning on GoT annotations, then GRPO-based reinforcement learning over sampled reasoning chains and images.The first stage initializes templated reasoning; the second explores freer chains using multi-dimensional rewards.Source: §3.2 · p. 5

3.3 MLLM-BASED DUAL-STAGE MULTI-DIMENSIONAL REWARD

  • The dual-stage reward framework evaluates prompt-image alignment, prompt-reasoning faithfulness, reasoning-image fidelity, and visual quality, combining intermediate-process and final-output supervision.Prompt-reasoning evaluation separates semantic and layout rewards; the framework uses MLLMs for multi-dimensional judgments.Source: Figure 3 · p. 5, §3.3 · p. 6
  • For spatial reasoning evaluation, GoT coordinates are rendered as bounding boxes for MLLM assessment; reasoning-image fidelity uses average IoU between planned and grounded object boxes.The rendered-box approach addresses limited sensitivity to textual coordinates and spatial relationships.Source: §3.3 · pp. 6–7, Figure 4 · p. 6

4 EXPERIMENT

Experiments report gains on compositional benchmarks and image quality, alongside analyses of self-explored reasoning and the reward design. Ablations examine both reward components and training settings.

4.1 TRAINING SETTINGS

  • Training uses 70,000 steps of GoT-T2I pretraining followed by 1,000 GRPO steps, with models based on Janus-Pro-1B and Janus-Pro-7B.The GRPO prompt set combines T2I-CompBench training prompts and LAION-Aesthetics; the reported setup uses groups of 16 candidates.Source: §4.1 · p. 7

4.2 QUANTITATIVE EVALUATION

  • On COCO 2014, GoT-R1-7B improves CLIP Score and Aesthetic Score and receives 77% human preference over baseline models.Human evaluation used 300 randomly selected prompts.Source: Table 4 · p. 7, §4.2 · p. 7
  • GoT-R1-7B achieves the highest scores in five of six T2I-CompBench categories, with up to 15% improvement after 1,000 GRPO fine-tuning steps.The comparison spans diffusion models with frozen encoders, two-stage layout-guided models, and autoregressive models enhanced with LLMs or MLLMs.Source: §4.2 · p. 7, Table 1 · p. 8
  • GoT-R1-7B scores 0.75 overall on GenEval; two-object generation improves from 0.69 to 0.94, and attribute binding from 0.43 to 0.68.The overall score is reported as a new state of the art over Janus-Pro-GoT-7B.Source: §4.2 · p. 7, Table 2 · p. 8

4.5 ABLATION STUDY ON REWARD DESIGN

  • Removing RRI or RPI degrades performance, RPI performs best among single rewards, and Rsem and Rspa primarily support attribute binding and spatial consistency, respectively.Using only RRI can misalign reasoning and provide harmful guidance to image generation.Source: Table 3 · p. 8, §4.5 · pp. 9–10

4.3 QUALITATIVE EVALUATION

  • Compared with Janus-Pro-7B and Janus-Pro-7B-GoT, GoT-R1-7B better aligns with complex prompts and produces detailed, aesthetically appealing images.The examples include multiple attributes, relative spatial relationships, object numeracy, and unnatural prompts.Source: §4.3 · pp. 7–8, Figure 5 · p. 9

4.4 ANALYSIS ON SELF-EXPLORED GENERATION CHAIN-OF-THOUGHT

  • GPT-4o overwhelmingly prefers GoT-R1-7B’s self-explored reasoning over predefined GoT across the evaluated T2I-CompBench categories and LAION-5B prompts.The evaluation sampled 100 prompts each from Color, Spatial, Complex, and LAION-5B, assessing prompt relevance, object and bounding-box accuracy, and text clarity and fluency.Source: Table 5 · p. 9, §4.4 · p. 9

4.5 ABLATION STUDY ON REWARD DESIGN

  • Training ablations compare product versus summed rewards, visualized versus textual layout scoring, and MLLM rewards versus CLIP and Grounding DINO; the reported findings affirm the chosen settings.The ablations are evaluated on T2I-CompBench under guidance scale 5.Source: Table 6 · p. 9, §4.5 · p. 10

5 CONCLUSION

GoT-R1 uses reinforcement learning to improve semantic-spatial reasoning in autoregressive visual generation, with gains reported on T2I-CompBench and GenEval. Its dual-stage reward framework evaluates both reasoning and generated images.

5 CONCLUSION

  • GoT-R1 applies reinforcement learning to autoregressive visual generation, helping models discover reasoning strategies beyond predefined templates for compositional prompts.The framework builds on the Generation Chain-of-Thought paradigm.Source: §5 · p. 10
  • Its MLLM-based dual-stage, multi-dimensional rewards assess intermediate reasoning and final images for semantic alignment, spatial accuracy, and image quality.Source: §5 · p. 10
  • GoT-R1 reports significant improvements over existing methods on T2I-CompBench and GenEval, particularly for precise attribute binding and spatial relationships.Source: §5 · p. 10

6 ETHICS AND REPRODUCIBILITY STATEMENT

The work uses public datasets and restricts experiments to research benchmarks and synthetic evaluation. It describes reproducibility measures and plans to release code, checkpoints, and evaluation scripts.

6 ETHICS AND REPRODUCIBILITY STATEMENT

  • Experiments use publicly available T2I-CompBench trainset and GoT datasets, without private or sensitive data.Source: §6 · p. 10
  • The study limits experiments to research benchmarks and synthetic evaluation because text-to-image generation may be misused for misleading or harmful content.Source: §6 · p. 10
  • The authors report model architectures, training setups, and hyperparameters, validate robustness through multiple runs and ablations, and plan to release code, checkpoints, and evaluation scripts.Source: §6 · p. 10

A QUALITATIVE EVALUATION

Figure 6 presents text-to-image samples from Janus-Pro-7B, Janus-Pro-GoT-7B, and GoT-R1-7B across various prompts, for qualitative comparison of image quality and prompt adherence.

A QUALITATIVE EVALUATION

B ADDITIONAL QUANTITATIVE ANALYSIS

Additional analyses report that the Qwen2.5VL-7B reward model aligns more closely with human preference than CLIP Score, training runs are stable, and single-reward training shows reward hacking with degraded compositional performance.

B ADDITIONAL QUANTITATIVE ANALYSIS

C LIMITATIONS

The paper acknowledges constraints involving position prediction in object-heavy scenes, inference latency, and dependence on the reward model’s capability.

  • Position prediction accuracy decreases for prompts with more than ten items, which can affect the fidelity of very complex scenes.Source: C LIMITATIONS · p. 16
  • Generating the explicit GoT reasoning chain can increase total generation time by approximately 30% compared with direct generation without reasoning.Source: C LIMITATIONS · p. 16
  • Framework performance is inherently tied to the capability of the MLLM used as the reward function.Source: C LIMITATIONS · p. 16

D MLLM-BASED REWARD EVALUATION PROMPTS

The evaluation prompt appendix describes prompts for assessing semantic consistency, spatial layout, prompt-image alignment, reasoning-image grounding, and reasoning-chain comparison.

E REWARD CURVE IN GRPO TRAINING

The total reward rises during GRPO training and gradually stabilizes; the figure rescales rewards to make the progression easier to visualize.

Loading 2505.17022v2…