PaperlayerView exact revision on arXiv

Source-linked AI summary

ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL

Yu Zhang, Yunqi Li, Yifan Yang, Rui Wang, Yuqing Yang, Dai Qi, Jianmin Bao, Dongdong Chen, Chong Luo, Lili Qiu

arXiv:2505.24875v2cs.CVcs.CL

TL;DR

Transferring step-by-step reasoning from text tasks to image generation is identified as an important next step. REASONGEN-R1 trains an autoregressive image generator with CoT-supervised fine-tuning followed by GRPO, and outperforms previous models across several benchmarks.

  • Problem

    Transferring step-by-step reasoning from text-based tasks to image generation is identified as an important next step.

    Source: §1 · p. 2

  • Method

    REASONGEN-R1 trains an autoregressive image generator on instruction–CoT–image samples with SFT, then applies GRPO using Qwen-2.5-VL alignment rewards and adaptive entropy loss.

    Source: §1 · p. 2

  • Results

    REASONGEN-R1 outperforms previous models, including Janus-Pro and other state-of-the-art systems, across multiple metrics; reported gains are GenEval (+6%), DPG-Bench (+1.69%), and T2I-Benchmark (+13.38%).

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The framework establishes a baseline for future “think-and-generate” content creation.

    Source: §1 · p. 2

    Reliance on large-scale pretrained models such as GPT-4.1 may introduce SFT-data biases that affect robustness and fairness, particularly in sensitive or underrepresented contexts.

    Source: §6 · p. 11

Abstract

from arXiv · show

Although chain-of-thought reasoning and reinforcement learning (RL) have driven breakthroughs in NLP, their integration into generative vision models remains underexplored. We introduce ReasonGen-R1, a two-stage framework that first imbues an autoregressive image generator with explicit text-based "thinking" skills via supervised fine-tuning on a newly generated reasoning dataset of written rationales, and then refines its outputs using Group Relative Policy Optimization. To enable the model to reason through text before generating images, We automatically generate and release a corpus of model crafted rationales paired with visual prompts, enabling controlled planning of object layouts, styles, and scene compositions. Our GRPO algorithm uses reward signals from a pretrained vision language model to assess overall visual quality, optimizing the policy in each update. Evaluations on GenEval, DPG, and the T2I benchmark demonstrate that ReasonGen-R1 consistently outperforms strong baselines and prior state-of-the-art models. More: aka.ms/reasongen.

1 Introduction

The introduction frames text-based reasoning for autoregressive image generation as an open challenge and presents REASONGEN-R1 as a two-stage SFT-plus-RL framework. It reports stronger benchmark performance and describes a rationale-annotated dataset, VLM rewards, and adaptive entropy loss as parts of the approach.

1 Introduction

  • The paper identifies a gap in autoregressive image generators’ ability to produce textual reasoning alongside images, making straightforward RL supervision potentially ineffective.It motivates reasoning before image creation by analogy with artists’ structural and sequential planning.Source: §1 · p. 2
  • The framework combines SFT with CoT and GRPO-based RL, using an instruction→CoT→image sequence to supervise textual rationales and image outputs.Its 200k-sample dataset pairs LAION aesthetics-subset images with GPT-4.1-mini-annotated reasoning trajectories.Source: §1 · p. 2
  • REASONGEN-R1 uses Qwen-2.5-VL for binary prompt–image consistency rewards and adaptive entropy loss to stabilize interleaved-modality RL training.The paper reports sensitivity to entropy explosion and vanishing in this setting.Source: §1 · p. 2
  • REASONGEN-R1 reports gains of +6% on GenEval, +1.69% on DPG-Bench, and +13.38% on T2I-Benchmark, and Figure 2 shows it outperforming Janus-Pro-7B across three instruction-following benchmarks.The paper also states that extensive experiments confirm the framework’s efficiency and effectiveness.Source: §1 · p. 2, Figure 2 · p. 3
  • The authors position integrating reasoning through a two-stage SFT-plus-GRPO framework as a baseline for future “think-and-generate” content creation.They describe the dataset, multimodal reward models, and adaptive entropy loss as technical contributions.Source: §1 · p. 2

2 Related Work

Related work covers unified autoregressive generation, chain-of-thought reasoning in language models, and RL-based reasoning. It identifies the integration of RL specifically for reasoning in image generation as largely unexplored.

2.1 Unified Auto-Regressive Generation Models

  • Unified autoregressive models can generate language and images, but separate text and image phases limit continuous cross-modal reasoning.The paper names Janus-Pro as an example of this modality split.Source: §2.1 · p. 3

2.2 Chain-of-Thought in LLM

  • CoT decomposes complex tasks into intermediate steps, with prior LLM work reporting gains in performance, interpretability, and controllability; this work integrates rationales and visual tokens in one sequence.The proposed sequence is trained end-to-end for interpretability and performance.Source: §2.2 · p. 3

2.3 Reinforcement Learning in LLM and LVLM reasoning

  • Although RL has been applied to image generation, its use specifically to refine reasoning within image generation remains largely unexplored.Related LLM and VLM work uses GRPO variants that normalize rewards across generated outputs without a separate critic model.Source: §2.3 · pp. 3–4

3 Method

REASONGEN-R1 first trains an autoregressive image generator to produce textual reasoning before an image, then applies RL to improve prompt analysis and image generation. The method combines a constructed caption dataset, interleaved SFT, group-relative policy optimization, adaptive entropy control, and VLM-based image rewards.

3 Method

  • REASONGEN-R1 has two stages: SFT equips a base autoregressive image generator with textual reasoning, then RL further improves prompt analysis and image output.The method adapts GRPO for image generation to align generated images with the rationale and prompt.Source: §3 · p. 4, §3.2 · p. 4

3.1.1 Dataset Construction

  • The dataset selects 200,000 LAION-Aesthetic V1 images, crops them to square format, and builds concise, augmented-concise, and detailed caption pairs.GPT-4.1 mini generates concise captions; GPT-4.1 nano generates augmented prompts and detailed captions from those concise captions.Source: §3.1.1 · p. 4

3.1.2 SFT Training

  • SFT teaches Janus-Pro-7B to generate a reasoning caption, emit its image-start token, and then produce image tokens in one sequence.The sequence begins with a concise image prompt followed by the detailed reasoning caption.Source: §3.1.2 · p. 4

3.2 Reinforcement Learning

  • GRPO compares advantages across groups of sampled responses and uses a clipped policy objective for the image-generation setting.The paper describes the objective as mirroring PPO’s clipped surrogate, with a KL penalty, and separately notes removing KL loss in its final algorithm.Source: §3.2.1 · p. 5

3.2.1 Reinforcement Learning Algorithm

  • The RL algorithm uses adaptive entropy loss; its target entropy and learnable coefficient are adjusted to address entropy vanishing, explosion, and resulting mode collapse.The paper changes α’s parameterization to arcsin(ϕ), allowing α to take positive or negative values.Source: §3.2.1 · p. 5

3.2.2 Reward Design

  • A pretrained Qwen-2.5-VL model scores image–prompt consistency, and the image-level reward is propagated through the full prompt→reasoning→image sequence.This assigns credit to preceding rationales based on the quality of the visual output.Source: §3.2.2 · pp. 5–6

4 Experiment

The experiments evaluate instruction adherence across three benchmarks and examine SFT, reward-model size, entropy regularization, and textual CoT. They also analyze how CoT content relates to generated images.

4.1.1 Evaluation Benchmarks

  • The benchmarks assess object-focused alignment, dense-prompt generation, and compositional prompts involving attributes, relations, and complex scene layouts.Source: §4.1.1 · p. 6

4.1.2 Dataset Settings

  • The training prompts combine expanded GenEval prompts, 5,000 GPT-4.1-generated DPG prompts, and 11,003 prompts from the T2I-CompBench++ training split.The expanded GenEval set contains 12,367 prompts after filtering overlap with its test set.Source: §4.1.2 · pp. 6–7

4.1.3 Training Settings

  • SFT ran for one epoch, while RL ran for 300 steps and selected the checkpoint with the highest validation reward.Source: §4.1.3 · p. 7

4.2 Main Result

4.3.1 Effectiveness of SFT training

  • The SFT-plus-RL model outperforms the w/o SFT baseline by 18%, while SFT alone is insufficient to achieve final performance.The authors attribute the need for RL to GPT-annotated CoT traces not always matching reasoning trajectories conducive to high-quality image synthesis.Source: §4.3.1 · p. 8, Table 4 · p. 8

4.3.2 Reward Model Size Matters

  • Using a smaller 3B rewarder yields poor post-RL performance, which the authors attribute to inadequate reward signals.Source: §4.3.2 · p. 8, Table 4 · p. 8

4.3.3 Stable Training with Adaptive Entropy Loss

  • Without entropy loss, RL entropy explodes after 100 steps; a fixed -0.002 penalty drives entropy dangerously low by step 80, while adaptive loss maintains an optimal range.The fixed penalty leads to mode collapse and a sharp reward decrease; no entropy loss also degrades performance.Source: §4.3.3 · pp. 8–9, Figure 4 · p. 8

4.4 Analysis on Textual CoT

5 Conclusion

REASONGEN-R1 combines CoT-based supervised fine-tuning with reinforcement learning to improve autoregressive image generation. The authors report stronger benchmark performance and identify reward calibration and adaptive entropy loss as important to stable RL training.

5 Conclusion

  • The framework combines CoT reasoning, supervised fine-tuning, and GRPO-based reinforcement learning to improve autoregressive image generation.The authors report better adherence to complex instructions and improvements in image quality and consistency with textual prompts.Source: §5 · p. 10
  • REASONGEN-R1 outperforms previous models, including Janus-Pro and other state-of-the-art systems, across GenEval, DPG-Bench, and T2I-Benchmark metrics.The authors also emphasize the importance of a well-calibrated reward model and adaptive entropy loss for stable and effective RL training.Source: §5 · p. 10
  • The authors propose the framework as a direction for multimodal generation and identify multi-step planning and fine-grained content control as future research areas.They plan to release the dataset and training code.Source: §5 · p. 10

6 Limitation

The authors identify three limitations: benchmark coverage may not represent real-world task diversity, SFT data may inherit pretrained-model biases, and adaptive entropy loss remains task-sensitive.

6 Limitation

  • Evaluation focuses on a specific benchmark set, leaving generalization to more complex or domain-specific tasks for future investigation.The authors note that these benchmarks may not fully represent real-world task diversity.Source: §6 · p. 11
  • Reliance on pretrained models such as GPT-4.1 may introduce biases from the SFT data, potentially affecting output robustness and fairness in sensitive or underrepresented contexts.Source: §6 · p. 11
  • The task-specific sensitivity of the adaptive entropy loss parameter needs further study, despite its use to mitigate mode collapse.Source: §6 · p. 11

A.1 Dataset Construction Details

Dataset construction uses three OpenAI API calls to produce concise captions, prompt variations, and detailed captions for CoT supervision. The detailed captions are generated from concise captions alone, matching the information available to the model during SFT.

A.1 Dataset Construction Details

  • The dataset construction uses three OpenAI API calls, each with a distinct role.Source: §A.1 · p. 15
  • GPT-4.1 Small generates concise image captions describing object counts, colors, positions, and other details.The captioning prompt asks for a short sentence that faithfully captures important image information.Source: §A.1 · p. 15
  • GPT-4.1 Nano augments each concise caption with paraphrases, tags, varied captions, and object prompts to diversify SFT prompt patterns.Source: §A.1 · p. 15
  • Detailed captions generated from concise captions alone serve as ground-truth CoT supervision, avoiding visual information unavailable to the model during SFT.The detailed-caption generation step does not provide GPT with the corresponding image.Source: §A.1 · p. 15

A.2 SFT Training Details

SFT training diversifies concise prompts and adds a bridging instruction before the ground-truth CoT. RL uses separate modality-specific entropy targets, a vision-language reward model, and implementation settings for rollout and evaluation.

A.2 SFT Training Details

  • Training uniformly samples a prompt augmentation type, including the original concise caption, to reduce overfitting to one prompt format.Tags and object prompts are comma-joined, while paraphrases and varied captions are sampled individually.Source: §A.2 · p. 16
  • A bridging instruction, “Output a richly detailed prompt,” is inserted between the concise prompt and ground-truth CoT to reduce the base-to-target distribution gap.Source: §A.2 · p. 16

A.3 RL Algorithm Details

  • RL maintains independent adaptive entropy regularizers for image and text tokens, with targets of 7.0 and 2.0 respectively, based on average post-SFT rollout entropy.Source: §A.3 · p. 16
  • Qwen-2.5-VL-7B grades prompt–image alignment with a binary consistency score, requiring a perfect match for a positive reward.The reward prompt asks the model to inspect required elements, counts, colors, and positions.Source: §A.3 · p. 16
  • RL uses verl and classifier-free guidance scale 1.0 for faster rollouts, while inference and evaluation use scale 5.0, matching Janus-Pro 7B’s default.Source: §A.3 · pp. 16–17

B.1 Analysis on Text CoT

Table 6 presents additional ReasonGen-R1 visualizations involving CoT substitution. The examples pair detailed scene descriptions with short image prompts across food, animal, and nature scenes.

B.1 Analysis on Text CoT

  • Table 6 collects additional ReasonGen-R1 visualization results for CoT substitution.The examples pair detailed scene descriptions with short prompts across several scene types.Source: Table 6 · p. 17, §B.1 · p. 17
  • Other examples pair a plant-left-of-cow prompt with a detailed cow scene, and bird prompts with descriptions of a cardinal alone or beside a wildflower.Source: §B.1 · p. 18
  • Pizza examples pair elaborate descriptions of fruit or meat toppings with the same short prompt, “A photo of a pizza.”Source: §B.1 · pp. 18–19

B.2 Experiment Settings

Table 7 summarizes hyperparameter configurations for supervised fine-tuning and reinforcement learning. The settings are framed as balancing model capacity, sequence coverage, and compute efficiency.

B.2 Experiment Settings

  • The configurations cover both supervised fine-tuning and reinforcement-learning stages of the pipeline.Source: §B.2 · p. 19, Table 7 · p. 19
  • The stated design aims to balance model capacity, sequence coverage, and compute efficiency.Source: §B.2 · p. 19
  • Table 7 presents the hyperparameter configurations used for supervised fine-tuning and reinforcement learning.Source: Table 7 · p. 19, §B.2 · p. 19

B.3 Case Visualization

This section presents ReasonGen-R1 case visualizations of written reasoning used to guide image composition and expand short prompts into detailed scene descriptions. Examples cover long and detailed prompts, counting prompts, spatial relationships, and complex attributes.

B.3 Case Visualization

  • The rollouts include long and detailed prompts sampled from DPG-bench and counting prompts generated using Geneval.Source: §B.3 · p. 19, Table 8 · p. 20, Table 9 · p. 29
  • Across the visualized categories, REASONGEN-R1’s CoT aligns closely with generated image content and guides scene planning and composition.Source: §B.3 · p. 19
  • For terse prompts, CoT adds object attributes, lighting, textures, backgrounds, and mood; “A photo of two persons” is refined into a more specific scene description.The example specifies a young woman and a man, natural light, clothing, an ambient background, and an emotional tone.Source: §B.3 · p. 20
  • Spatial-relationship visualizations include prompts such as “A photo of a sheep left of a bird” and “A photo of a cup left of a vase.”Their accompanying descriptions elaborate on the objects, lighting, and scene mood.Source: Table 10 · p. 33, §B.3 · p. 34
  • Complex-attribute visualizations include prompts specifying object colors and combinations, such as a green teddy bear with a blue bench and a red snowboard with a blue bicycle.The paired descriptions expand these prompts into fuller scene narratives.Source: Table 11 · p. 39, §B.3 · pp. 40–41
Loading 2505.24875v2…