Source-linked AI summary
Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step
Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Rui Huang, Haoquan Zhang, Manyuan Zhang, Jiaming Liu, Shanghang Zhang, Peng Gao, Hongsheng Li, Pheng-Ann Heng
TL;DR
The paper addresses the open question of whether CoT-style reasoning can verify and reinforce autoregressive image generation step by step. It systematically evaluates test-time verification, DPO preference alignment, and their integration, introducing PARM and PARM++ as specialized reward models. The resulting strategies improve Show-o by +24% on GenEval and surpass Stable Diffusion 3 by +15%.
Problem
Whether multi-step reasoning strategies can be effectively applied to autoregressive image generation remains under-explored.
Method
The study evaluates ORM/PRM test-time verification, iterative DPO preference alignment, and their integration on Show-o, while introducing PARM and PARM++ for adaptive stepwise assessment and reflection-based self-correction.
Results
+24% on GenEval is achieved over the Show-o baseline, surpassing Stable Diffusion 3 by +15%.
Takeaways & Limitations
CoT reasoning strategies can be adapted and combined to improve autoregressive image generation, with PARM and PARM++ providing tailored assessment and self-correction.
Takeaways & Limitations
ORM lacks fine-grained stepwise evaluation, while PRM struggles with blurry early images and visually similar later-stage outputs.
Abstract
from arXiv · showhide
Chain-of-Thought (CoT) reasoning has been extensively explored in large models to tackle complex understanding tasks. However, it still remains an open question whether such strategies can be applied to verifying and reinforcing image generation scenarios. In this paper, we provide the first comprehensive investigation of the potential of CoT reasoning to enhance autoregressive image generation. We focus on three techniques: scaling test-time computation for verification, aligning model preferences with Direct Preference Optimization (DPO), and integrating these techniques for complementary effects. Our results demonstrate that these approaches can be effectively adapted and combined to significantly improve image generation performance. Furthermore, given the pivotal role of reward models in our findings, we propose the Potential Assessment Reward Model (PARM) and PARM++, specialized for autoregressive image generation. PARM adaptively assesses each generation step through a potential assessment approach, merging the strengths of existing reward models, and PARM++ further introduces a reflection mechanism to self-correct the generated unsatisfactory image, which is the first to incorporate reflection in autoregressive image generation. Using our investigated reasoning strategies, we enhance a baseline model, Show-o, to achieve superior results, with a significant +24% improvement on the GenEval benchmark, surpassing Stable Diffusion 3 by +15%. We hope our study provides unique insights and paves a new path for integrating CoT reasoning with autoregressive image generation. Code and models are released at https://github.com/ZiyuGuo99/Image-Generation-CoT
1 Introduction
The paper investigates whether CoT-style step-by-step verification and preference reinforcement can improve autoregressive image generation. Using Show-o, it studies test-time reward verification, DPO alignment, their integration, and specialized reward models PARM and PARM++.
- Autoregressive image generation remains under-explored as a setting for multi-step reasoning, despite its discrete-token, iterative decoding process.
- The study evaluates ORM/PRM-based test-time verification and iterative DPO preference alignment on Show-o using the GenEval benchmark.It also examines how these strategies combine.
- ORM improves performance substantially, whereas PRM provides minimal benefit in the investigated image-generation setting.
- Integrated preference alignment and test-time verification methods produce greater improvements, indicating complementary effects.The combinations include reward-model-guided DPO, verification after DPO, and both together.
- PARM adaptively assesses generation steps by judging evaluability, estimating future potential, and ranking final paths, combining fine-grained and outcome-level assessment.It is designed to address blurry early images, convergent later paths, and the limitations of global reward judgments.
- The resulting reasoning strategies improve Show-o by +24% on GenEval and surpass Stable Diffusion 3 by +15%.
- PARM++ adds reflection-based self-correction, detecting prompt-image misalignment and iteratively refining the generated image.On the Show-o baseline, it improves over PARM by +4% on GenEval.
2 Related Work
Prior work established autoregressive and multimodal models for language, understanding, and image generation, while this study extends verifier-based reasoning strategies to image generation.
- Scaling test-time computation has been explored for complex language and multimodal understanding tasks through reasoning paths and reward models.
- The paper investigates whether verifier-based strategies can enhance image generation and introduces PARM for this domain.
- Post-training alignment methods use preference models and reinforcement learning to align model outputs with target preferences.
- Autoregressive image-generation models learn dependencies among image tokens, offering an alternative to diffusion-based synthesis.
3 Our Investigation
The study asks whether Chain-of-Thought reasoning can verify and reinforce image generation step by step, extending a strategy widely used in language and multimodal understanding.
- The central question is whether CoT reasoning can verify and reinforce image generation step by step.
3.1 Overview
The investigation targets autoregressive image generation because discrete-token decoding creates reasoning paths suitable for preference optimization and step-wise reward verification.
- Autoregressive image generation converts images into discrete tokens and predicts outputs iteratively from prior outputs.
- This representation supports DPO post-training because it permits cross-entropy optimization over generated tokens.
- The study uses Show-o as its baseline and evaluates text-to-image generation on GenEval, which tests quality, alignment, attributes, and object co-occurrence.
3.2 ORM vs PRM as Test-time Verifiers
The study compares outcome- and process-level reward models for best-of-N test-time verification, finding that fine-tuned ORM benefits most while PRM faces difficulties with intermediate images.
- ORM scores final candidate images for prompt alignment and selects the candidate with the highest probability of a positive response.
- ORM and PRM are implemented with LLaVA-OneVision, while ranking datasets are curated to fine-tune their verification abilities.
- PRM scores candidates at intermediate steps and follows the most confident path through step-level best-of-N selection.
- +10%: fine-tuned ORM achieves the highest gain over the 53% GenEval baseline in test-time verification.
- +2%: fine-tuned PRM provides only marginal GenEval improvement, whereas ORM evaluates final images with richer visual and semantic information.
- Fine-tuned reward models outperform zero-shot versions and improve more steeply as N increases in best-of-N selection.
3.3 Test-time Verifiers vs Preference Alignment
This section compares test-time verification with DPO preference alignment for autoregressive image generation. Iterative DPO progressively improves performance and surpasses the strongest fine-tuned ORM verifier.
- DPO Alignment: 13K prompts support the initial preference dataset, while iterative DPO constructs a refined dataset of 7K paired samples.The aligned model generates updated ranking data, excluding prompts whose images receive uniform labels.
- DPO Alignment: DPO trains Show-o on paired preferred and dispreferred images, encouraging higher likelihood for preferred generations.The policy is initialized from Show-o, while a frozen Show-o reference policy provides the comparison baseline.
- Results: +9% on GenEval follows initial DPO alignment, increasing to +11% with iterative training.The authors attribute the additional gain to learning from a refined preference dataset.
- Results: Initial DPO matches fine-tuned ORM verification, whereas iterative DPO surpasses all test-time verifiers by +2% over fine-tuned ORM.Test-time verification uses best-of-N selection with N = 20 in Table 1.
3.4 DPO Alignment plus Test-time Verifiers
The authors combine preference alignment with test-time verification through three integration strategies. Across these strategies, the methods complement one another, with verifier use during both training and inference producing the largest gains.
- Integration Strategies: The three integrations combine DPO with reward-model guidance, verification after DPO, or both forms of reward-model use.The comparison evaluates training-time guidance and best-of-N selection after alignment.
- Results: The third integration outperforms the first by +8% and the second by +3%.This configuration applies the verifier during both training and test-time decoding.
- Results: Using verifiers during both training and test time yields the maximum enhancement among the evaluated integration methods.The result supports complementary roles for preference alignment and inference-time path selection.
4 Potential Assessment Reward Model
PARM is a reward model specialized for autoregressive image generation that combines adaptive step-wise potential assessment with best-of-N′ selection. It improves verification and, when combined with iterative DPO, substantially raises Show-o’s GenEval performance.
- Motivation: ORM selects strong final outputs but lacks fine-grained step-wise evaluation, while PRM struggles with blurry early images and similar late-stage images.These limitations motivate a reward model tailored to intermediate autoregressive image states.
- PARM Design: PARM first judges visual clarity, then assesses whether clear intermediate images have potential to become high-quality final outputs.Paths failing clarity are skipped, while paths failing potential assessment are truncated.
- PARM Design: PARM performs best-of-N′ selection among surviving high-potential paths, defaulting to the path with the lowest ‘no’ probability when N′ = 0.This preserves ORM’s global final-output selection while adding intermediate-path pruning.
- Results: +6% over fine-tuned ORM is achieved by PARM, which also scales with increasing N and outperforms iterative DPO.PARM additionally attains higher integration scores than fine-tuned ORM when combined with post-training.
- Results: 77% on GenEval is reached by integrating PARM with iterative DPO during post-training and test time, improving Show-o by +24% and surpassing Stable Diffusion 3 by +15%.The largest gains occur on Two Obj., Colors, Position, and Attribute binding.
- Generalization: The approach consistently improves LlamaGen-3B and Janus-Pro-7B without retraining or architectural changes.This demonstrates plug-and-play transfer to sequential autoregressive image generators.
5 Potential Assessment Reward Model ++
PARM++ extends PARM with an external reflection stage that detects image-text misalignment and guides iterative self-correction. Reflection substantially improves GenEval quality, although the self-correction fine-tuning itself slightly reduces baseline accuracy.
- Reflection Mechanism: PARM++ adds reflection to examine final image-prompt alignment and generate detailed discrepancy explanations for correction.The reflection mechanism is handled by the reward model because image generators output images rather than freeform review text.
- Training: The reflection process runs for at most 3 iterations and uses a 520K-point training set combining PARM data with 120K reflection instances.Negative examples contain GPT-4o annotations describing image-text discrepancies.
- Training: PARM++ fine-tunes Show-o for iterative refinement using prompts, low-quality images, high-quality images, and annotated reflection reasons.Show-o supports simultaneous text and image inputs, enabling refinement from textual feedback.
- Results: +10% on GenEval results when PARM++ reflection is enabled, addressing unrealistic elements, incorrect numbers, wrong colors, and improper layouts.Without reflection, PARM++ performs slightly worse than PARM because of reflection-specific training data.
- Limitation: -2% GenEval accuracy follows self-correction fine-tuning before reflection is applied.The authors describe this as a trade-off caused by training for new capabilities affecting the model’s original knowledge.
6 Conclusion
The paper demonstrates that CoT reasoning strategies can improve autoregressive image generation and introduces tailored reward models for step-wise assessment and self-correction.
- Different CoT reasoning strategies, including test-time verification, preference alignment, and their integration, effectively improve autoregressive image generation.
- PARM evaluates step-wise generation for adaptive reward scoring in autoregressive image generation.
- PARM++ incorporates reflection to enable self-corrected image generation.
- The qualitative results compare Show-o with the best-performing strategy, which integrates PARM with iterative DPO for reward guidance and test-time verification.