Source-linked AI summary
Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning
Lei Zhang, Junjiao Tian, Zhipeng Fan, Kunpeng Li, Jialiang Wang, Weifeng Chen, Markos Georgopoulos, Felix Juefei-Xu, Yuxiang Bao, Julian McAuley, Manling Li, Zecheng He
TL;DR
Image generation models can produce plausible but incorrect scenes, while partially formed images are difficult to evaluate for spatial and semantic correctness. The paper introduces process-driven generation, which interleaves textual planning, visual states, inspection, and refinement with dense supervision. It reports gains from 79% to 83% on GenEval and from 70% to 76% on WISE for BAGEL-7B, alongside higher GenEval accuracy than PARM with 8x lower training-data and inference cost.
Problem
One-shot models remain brittle on visual logic, and incomplete intermediate images are difficult to evaluate for preserved spatial structure and semantic coherence.
Method
Process-driven generation trains a unified multimodal model to co-evolve textual plans and visual states through Plan, Sketch, Inspect, and Refine with step-wise supervision.
Results
79% to 83% on GenEval and 70% to 76% on WISE are reported for BAGEL-7B, while GenEval accuracy reaches 0.83 vs. 0.77 for PARM with 8x lower training-data and inference cost.
Takeaways & Limitations
The approach makes image generation explicit, interpretable, and directly supervisable by enforcing correctness during intermediate steps rather than only at the final output.
Abstract
from arXiv · showhide
Humans paint images incrementally: they plan a global layout, sketch a coarse draft, inspect, and refine details, and most importantly, each step is grounded in the evolving visual states. However, can unified multimodal models trained on text-image interleaved datasets also imagine the chain of intermediate states? In this paper, we introduce process-driven image generation, a multi-step paradigm that decomposes synthesis into an interleaved reasoning trajectory of thoughts and actions. Rather than generating images in a single step, our approach unfolds across multiple iterations, each consisting of 4 stages: textual planning, visual drafting, textual reflection, and visual refinement. The textual reasoning explicitly conditions how the visual state should evolve, while the generated visual intermediate in turn constrains and grounds the next round of textual reasoning. A core challenge of process-driven generation stems from the ambiguity of intermediate states: how can models evaluate each partially-complete image? We address this through dense, step-wise supervision that maintains two complementary constraints: for the visual intermediate states, we enforce the spatial and semantic consistency; for the textual intermediate states, we preserve the prior visual knowledge while enabling the model to identify and correct prompt-violating elements. This makes the generation process explicit, interpretable, and directly supervisable. To validate proposed method, we conduct experiments under various text-to-image generation benchmarks.
1 Introduction
The paper proposes process-driven image generation, replacing one-shot synthesis with an interleaved Plan–Sketch–Inspect–Refine loop grounded in evolving visual states. Dense supervision addresses ambiguous intermediates and improves benchmark performance and efficiency.
- Motivation: One-shot generation can produce plausible but incorrect images because models must resolve spatial layouts, object relations, and attributes in a single pass.Textual chain-of-thought remains visually blind to spatial misalignments and cannot dynamically evolve object states.
- Process-driven generation: Process-driven generation reformulates synthesis as a co-evolving trajectory of textual plans and visual states executed through Plan, Sketch, Inspect, and Refine.The model constructs images incrementally rather than hallucinating a final image in one step.
- Supervision: Scene-graph subsampling creates logically ordered incremental prompts, while dual-stream process critiques teach self-assessment from textual and visual states.These mechanisms address ambiguous intermediates and help models identify alignment errors before final output.
- Results: 79% to 83% (+4%) on GenEval composition object alignment and 70% to 76% (+6%) on WISE world knowledge reasoning are reported for BAGEL-7B.The comparison uses the base BAGEL-7B as the reference model.
- Results: 0.83 vs. 0.77 on GenEval is achieved with an 8x reduction in both training data and inference cost compared with PARM.The paper attributes this balance to semantic partitioning, which supervises concrete visual states instead of latent noise.
- Results: +6% is reported for self-sampled critiques over symbolic corrections in ablation studies.The study identifies diverse editing instructions and self-sampled critiques as important drivers of performance.
2 Related Work
Related work extends unified multimodal modeling and interleaved reasoning, but existing approaches often constrain visual understanding or separate reasoning from image generation. Process-driven generation is positioned against methods that verify or refine before or after sampling.
- Unified multimodal models: Early unified multimodal models use discrete visual tokenizers to model images as token sequences but suffer from constrained fine-grained visual understanding.Examples include Chameleon, Emu3, and Show-o.
- Unified multimodal models: Another line of work couples a pretrained language model with an external diffusion module.
- Interleaved reasoning: Recent interleaved-reasoning methods apply verification or prompt refinement before generation or after image sampling, rather than reasoning throughout generation.The paper distinguishes these text-oriented or outcome-driven approaches from its process-driven formulation.
3 Method
The method reframes image generation as an interleaved textual–visual trajectory that repeatedly plans, sketches, inspects, and refines localized updates. It trains this process with structured intermediate states and critiques to preserve coherence and correct conflicts.
- Framework: Process-driven generation alternates textual reasoning steps with intermediate visual states, progressively constructing the final image.The unified multimodal model generates a trajectory rather than committing to the entire scene in one pass.
- Four-stage generation cycle: Each cycle follows Plan, Sketch, Inspect, and Refine to produce localized updates and correct discrepancies against the prompt.Planning specifies an incremental instruction, sketching drafts the update, inspection identifies mismatches, and refinement generates a corrected update.
- Interleaved states: Intermediate textual states encode painting instructions, global descriptions, and refinement signals, while visual states comprise rough sketches and polished representations.Special textual forms distinguish planning and inspection outputs, and visual outputs explicitly mark modality transitions.
- Data construction and supervision: The method addresses ambiguous partial images with a multi-stage dataset pipeline that constructs visual states and textual critiques for supervised planning, assessment, and refinement.The pipeline uses scene-graph structures for consistent intermediate visual generation and self-sampling for textual critiques.
- Data construction and supervision: Scene-graph subsampling updates a specific local region while preserving the remaining scene’s spatial and semantic coherence, with instruction rewriting expanding editing behaviors.The dataset also includes conflict and image–instruction alignment reasoning subsets to teach textual and visual misalignment detection and correction.
- Model training: The model conditions on the prompt and the full preceding chain of interleaved visual and textual reasoning trajectories, using a weighted combination of training objectives.The adopted unified multimodal backbone is fine-tuned for the interleaved process-driven generation task.
4 Experiment
Experiments show that process-driven reasoning improves compositional and world-knowledge text-to-image generation while reducing training and inference costs. Ablations attribute these gains to diverse editing instructions, self-sampled critiques, and complementary semantic- and visual-level supervision.
- Implementation Details: 30K multi-turn samples, 15K conflict samples, and 15K alignment samples form the process-based interleaved reasoning dataset.The dataset combines intermediate images with step-level instructions, conflict trajectories, and positive/negative image–instruction pairs.
- Quantitative Evaluation: 0.76 overall on WISE demonstrates the strongest reported performance, with substantial gains in Time, Biology, and Chemistry.WISE evaluates world-knowledge reasoning in text-to-image generation, where unified models otherwise struggle in temporal and scientific domains.
- Quantitative Evaluation: 0.83 on GenEval exceeds PARM’s 0.77 while using 62K samples instead of 688K and 131 sampling steps instead of 1000.The reported comparison corresponds to an 11× reduction in training data and nearly 8× lower inference cost.
- Ablation Study and Analysis: Self-sampled critiques reach 0.87, 0.72, and 0.69 on Color, Position, and Attribute, outperforming scene-graph corrections.The authors attribute the improvement to critiques drawn from the model’s own distribution and failure modes.
- Ablation Study and Analysis: Combining instruction-intermediate conflict and image-instruction alignment supervision achieves the highest performance across tasks.The two mechanisms target distinct failure modes, improving Position by +5% and Counting by +12%, respectively.
5 Conclusion
The paper presents process-driven generation as a co-evolving loop in which unified multimodal models build images through planning, sketching, inspection, and refinement. The approach reports improved fidelity, detail, and benchmark performance over the BAGEL-7B base model.
- Conclusion: The method builds images stroke by stroke through textual planning, visual sketching, self-inspection, and refinement.It trains BAGEL-7B end-to-end to emit interleaved text and image tokens.
- Conclusion: 0.83 on GenEval and 0.76 on WISE improve BAGEL-7B from 0.79 and 0.70, respectively.The reported gains are +5% absolute on GenEval and +6% absolute on WISE.
- Conclusion: Generated images are described as having high visual fidelity, fine-grained details, and strong aesthetic appeal.The visualization uses prompts sampled from the Gen-Eval and WISE benchmarks.