Source-linked AI summary
T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT
Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, Hongsheng Li
TL;DR
Text-to-image generation has only preliminarily explored chain-of-thought reasoning despite the complexity of cross-modal alignment and fine-grained visual synthesis. T2I-R1 introduces semantic-level and token-level CoT coordinated by BiCoT-GRPO, and reports +13% on T2I-CompBench and +19% on WISE while surpassing FLUX.1.
Problem
CoT exploration for image generation remains preliminary, despite requiring cross-modal prompt interpretation and fine-grained visual-detail synthesis.
Method
T2I-R1 combines semantic-level high-level planning and token-level patch generation through BiCoT-GRPO in a unified model.
Results
+13% on T2I-CompBench and +19% on the WISE benchmark are reported, surpassing FLUX.1.
Takeaways & Limitations
T2I-R1 better understands complex prompts, reasons about user intentions, and handles uncommon scenarios with greater robustness.
Takeaways & Limitations
Applying the paradigm to video generation requires further exploration because dense frame-level rewards, unified video models, and inference-time efficiency remain unresolved.
Abstract
from arXiv · showhide
Recent advancements in large language models have demonstrated how chain-of-thought (CoT) and reinforcement learning (RL) can improve performance. However, applying such reasoning strategies to the visual generation domain remains largely unexplored. In this paper, we present T2I-R1, a novel reasoning-enhanced text-to-image generation model, powered by RL with a bi-level CoT reasoning process. Specifically, we identify two levels of CoT that can be utilized to enhance different stages of generation: (1) the semantic-level CoT for high-level planning of the prompt and (2) the token-level CoT for low-level pixel processing during patch-by-patch generation. To better coordinate these two levels of CoT, we introduce BiCoT-GRPO with an ensemble of generation rewards, which seamlessly optimizes both generation CoTs within the same training step. By applying our reasoning strategies to the baseline model, Janus-Pro, we achieve superior performance with 13% improvement on T2I-CompBench and 19% improvement on the WISE benchmark, even surpassing the state-of-the-art model FLUX.1. Code is available at: https://github.com/CaraJ7/T2I-R1
1 Introduction
The paper applies reinforcement learning and bi-level chain-of-thought reasoning to autoregressive text-to-image generation, separating semantic planning from token-level image synthesis. T2I-R1 combines these reasoning levels through BiCoT-GRPO and reports improvements over baseline models, including FLUX.1 comparisons.
- 1 Introduction: Image generation CoT is divided into semantic-level planning and token-level patch generation.Semantic-level CoT handles global structure and prompt reasoning, while token-level CoT handles local pixel details and visual coherence.
- 1 Introduction: T2I-R1 uses a unified language model for visual understanding and generation instead of requiring a separate prompt-interpretation model.The paper motivates this design by noting that separate models increase computational cost, complexity, and deployment challenges.
- 1 Introduction: The approach targets more human-aligned results by reasoning about prompt intentions and handling uncommon scenarios more robustly.Figure 2 presents examples involving reasoning or uncommon scenarios.
- 1 Introduction: BiCoT-GRPO jointly optimizes both CoT levels with an ensemble of vision-expert rewards.The framework integrates both reasoning processes within the same training step.
2 Related Work
Related work connects unified image generation and understanding in multimodal language models with reinforcement-learning approaches for large reasoning models. These lines of work motivate reasoning-enhanced autoregressive text-to-image generation.
- 2 Related Work: Unified multimodal models increasingly combine image understanding with language-model processing, while image generation from such models remains under exploration.The related-work discussion describes multiple branches of image-generation methods, including approaches using external image-generation components.
- 2 Related Work: Reinforcement-learning methods such as rule-based rewards and GRPO have been used to elicit extended reasoning before final answers in large language models.The cited related work emphasizes rewards focused on final-answer correctness and adherence to predefined formats.
3 Method
BiCoT-GRPO jointly optimizes semantic-level planning and token-level image generation through a two-stage process, using group-relative rewards and vision-expert evaluation. Its reward ensemble assesses human preference, object presence and relationships, and visual attributes.
- GRPO optimization: GRPO samples groups of responses, normalizes their rewards into group-relative advantages, and applies a clipped objective with a KL penalty.The token-level policy-gradient loss is normalized over generated tokens to balance rewards against overly long semantic reasoning.
- Semantic-level CoT: Semantic-level CoT precedes image generation and plans objects, attributes, spatial relationships, and interactions.It can also infer intended objects or actions for indirect, unusual, or ambiguous prompts.
- Token-level CoT: Token-level CoT treats sequential patch-by-patch image-token generation as visual reasoning.Each current patch is generated based on previous patches, paralleling progressive canvas completion.
- BiCoT-GRPO: BiCoT-GRPO extends GRPO to jointly optimize semantic-level and token-level CoT reasoning for image generation.The model first performs global semantic planning, then generates image tokens patch by patch as local reasoning.
- Generation pipeline: The two-stage pipeline generates semantic CoT first, then conditions image-token generation on the prompt, semantic CoT, and image-start token.The generated tokens are decoded into an image, and each response combines semantic and token sequences.
- Ensemble of generation rewards: The reward ensemble combines human-preference, object-detection, and visual-question-answering experts to evaluate generated images from multiple aspects.GPT-4o mini extracts objects and attributes for customized reward inputs, while multiple rewards regularize against hacking one reward model.
4 Experiment
Experiments evaluate T2I-R1 across benchmark comparisons, reward-model choices, and ablations of the two CoT levels. The method improves benchmark performance, while joint semantic- and token-level reasoning also preserves diversity and improves aesthetic quality.
- Main Results: 13% average improvement on T2I-CompBench and 19% on WISE over the baseline model.
- Main Results: T2I-R1 leads in five of six T2I-CompBench subtasks and achieves a spatial score of 0.3378.
- Main Results: T2I-R1 achieves the highest overall WISE score of 0.54 and outperforms FLUX.1-dev by 4%.
- Reward Analysis: The HPS reward performs well for attribute binding, while the object detector provides limited improvement there and weaker relation comprehension.
- Reward Analysis: The final model uses the H + D + V ensemble because human evaluation identified it as having the highest visual quality.
- Ablation Study: Semantic-level CoT increases diversity and benchmark performance, while joint optimization yields better aesthetic quality than semantic-level CoT alone.
5 Conclusion
The conclusion presents T2I-R1 as a reasoning-enhanced text-to-image model combining high-level planning with patch-level generation through reinforcement learning. It reports benchmark gains and improved handling of complex prompts and uncommon scenarios.
- T2I-R1 combines semantic-level CoT for high-level planning with token-level CoT for patch-by-patch generation.
- BiCoT-GRPO integrates both CoT levels within the same reinforcement-learning training step.
- Qualitative analysis indicates better understanding of complex prompts, user intentions, and uncommon scenarios.
A.1 Experiment Setup
The experiments use multiple text-to-image benchmarks and a Janus-Pro-7B base model, with prompts prepared for reward computation. Additional visualizations compare token-level-only and dual-CoT optimization.
- Training Settings: The training dataset contains 6,786 text prompts without images, sourced from T2I-CompBench training data and another cited source.
- Training Settings: Janus-Pro-7B is used as the base model with a learning rate of 1e-6 and beta of 0.01.
- Benchmark: Evaluation covers T2I-CompBench, WISE, GenAI-Bench, and TIIF-Bench.
- Benchmark: T2I-CompBench contains 6,000 compositional prompts across attribute binding, object relationships, and complex compositions.
- Additional Visualizations: Figures compare image diversity from token-level-CoT-only optimization with optimization of both semantic-level and token-level CoT.
B.1 More Results
Additional evaluations on GenAI-Bench and TIIF-Bench show that T2I-R1 improves the baseline and achieves leading results across basic and advanced prompts. The advanced-prompt margin is especially notable relative to FLUX.1.
- GenAI-Bench: T2I-R1 substantially improves the baseline model on GenAI-Bench and achieves the highest overall scores for basic and advanced prompts.
- GenAI-Bench: T2I-R1 surpasses FLUX.1 on both basic and advanced prompt types, with a notable margin on advanced prompts.
- Evaluation Coverage: Tables 5 and 6 report evaluation results on GenAI-Bench and TIIF-Bench, respectively.
- Interpretation: The advanced-prompt advantage is associated in the paper with the high-level reasoning capability provided by semantic-level CoT.
B.2 More Illustration of Ablation Study
The ablation study shows that semantic-level and token-level CoT contribute differently, while jointly optimizing both is necessary for the strongest overall behavior. Zero-shot semantic-level CoT alone provides little benefit compared with trained coordination.
- CoT ablations: Semantic-level CoT generally improves performance across both tested benchmarks, with a particularly significant gain on WISE.The improvement is attributed to textual reasoning that clarifies objects or phenomena before image generation.
- CoT ablations: Training solely with token-level CoT substantially reduces generated-image diversity.The paper demonstrates this effect qualitatively and quantifies diversity using ten generated images per prompt from T2I-CompBench.
- CoT ablations: Jointly optimizing semantic-level and token-level CoT produces better aesthetic quality than optimizing semantic-level CoT alone.The comparison supports the necessity of combining both reasoning levels.
- Zero-shot analysis: Zero-shot semantic-level CoT brings very marginal improvement, whereas T2I-R1 produces satisfying results.The baseline may miss prompt objects or generate reasoning that does not fit its image-generation ability.
B.3 More Details about Reward Analysis
The reward analysis evaluates image quality through human rankings of outputs generated with different reward-model combinations. The experiments also report the computational setup and training-hyperparameter reference used for the study.
- Human evaluation: Eight graduate students ranked four images generated with reward-model options V, O, H + D, and H + D + V according to visual appeal.Ranks were converted to scores of 3, 2, 1, and 0 points.
- Reward analysis: Multiple rewards may improve quality through implicit regularization that prevents overfitting to a single reward model.Individual reward models were reported to provide unsatisfactory quality despite high benchmark scores.
- Training setup: All experiments were conducted on 8 H800 GPUs, with training lasting about 16 hours.Detailed training hyperparameters are provided in Table 7.
C Limitations and Future Work
The paper identifies video generation as the main direction requiring further exploration. It notes unresolved challenges involving rewards, unified models, and inference time when extending the text-to-image paradigm.
- Future work: Applying this paradigm to video generation requires further exploration.The paper treats video generation as outside the currently demonstrated text-to-image scope.
- Future work: Video extension lacks a unified understanding-and-generation model, so BiCoT-GRPO cannot be used directly.The paper also notes that dense rewards for each generated frame remain an open question.
- Future work: Video generation inference is currently too long for the GRPO paradigm, leaving the balance between training time and effect for future study.This is presented as an additional unresolved challenge for extending the method.