Source-linked AI summary
DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generation
Dongzhi Jiang, Renrui Zhang, Haodong Li, Zhuofan Zong, Ziyu Guo, Jun He, Claire Guo, Junyan Ye, Rongyao Fang, Weijia Li, Rui Liu, Hongsheng Li
TL;DR
Text-to-image CoT methods have relied on generator-only approaches or coarse textual plans. DraCo interleaves visual drafts with textual verification and refinement; it outperforms other CoT methods across three benchmarks, including +8% on GenEval.
Problem
Existing T2I CoT methods either use unified MLLMs chiefly as generators or rely on textual planning, which can be too coarse for detailed image generation.
Source: §1 · p. 2Method
DraCo generates a low-resolution visual draft, uses the model to check it against the prompt, then selectively corrects and refines it.
Source: §1 · p. 2Results
Across three benchmarks, DraCo outperforms other methods with CoT, including +8% on GenEval.
Source: §3.2 · p. 7, §4 · p. 9Takeaways & Limitations
DraCo offers a direction for enhancing visual generation through interleaved multimodal reasoning in unified MLLMs.
Source: §4 · p. 9Application to other fields has not been extensively studied; low-resolution drafts are not directly or optimally suited to video, 3D assets, or scenes, and video generation remains computationally expensive.
Source: E. Limitations and Future Work · p. 14
Abstract
from arXiv · showhide
Recent unified multimodal large language models (MLLMs) have shown impressive capabilities, incorporating chain-of-thought (CoT) reasoning for enhanced text-to-image generation. However, existing approaches remain limited, either treating the model merely as a standalone generator or relying on abstract textual planning. To this end, we propose Draft-as-CoT (DraCo), a novel interleaved reasoning paradigm that fully leverages both textual and visual contents in CoT for better planning and verification. Our method first generates a low-resolution draft image as preview, providing more concrete and structural visual planning and guidance. Then, we employ the model's inherent understanding capability to verify potential semantic misalignments between the draft and input prompt, and performs refinement through selective corrections with super-resolution. In this way, our approach addresses two fundamental challenges: the coarse-grained nature of textual planning and the difficulty in generating rare attribute combinations. To support training, we curate DraCo-240K, aiming to enhance three atomic capabilities spanning general correction, instance manipulation, and layout reorganization. Supported by DraCo-CFG, a specialized classifier-free guidance (CFG) strategy for interleaved reasoning, DraCo achieves a tremendous increase on GenEval (+8%), Imagine-Bench (+0.91), and GenEval++ (+3%), significantly outperforming direct generation and other generation methods empowered by CoT.
1. Introduction
Prior text-to-image CoT methods either use the unified model chiefly as a generator or rely on textual plans that are too coarse for dense image details. DraCo combines visual drafting with textual verification and selective refinement.
1. Introduction
- Prior T2I CoT methods use reward models or text-only plans; text can be too abstract for detailed image planning, while rare attributes and object combinations are underrepresented in training data.The paper gives “a white orange” as an example of a rare combination models may struggle to generate.Source: §1 · p. 2
- DraCo generates a low-resolution visual draft, checks it for semantic misalignment with the prompt, then selectively corrects and refines it with super-resolution.The draft supplies visual planning, while the model's understanding capability supports verification.Source: §1 · p. 2
- DraCo-240K is curated to train the model to identify mistakes and precisely control draft corrections through diverse operations.The dataset targets three atomic correction capabilities and uses pipelines combining MLLMs, editing models, and segmentation models.Source: §1 · p. 3
- DraCo-CFG is designed for interleaved reasoning, explicitly strengthening draft-image visual semantics and verification-based correction instructions for final generation.The paper identifies these as the two major conditions for final generation.Source: §1 · p. 3
- 8% improvement on GenEval and 4% above the text-CoT-based method; DraCo also shows superior performance on ImagineBench and GenEval++.The reported GenEval comparison is against Bagel, a strong baseline.Source: §1 · p. 3
2. Method
DraCo uses Bagel’s unified visual understanding and generation in a draft–verify–refine pipeline, supported by DraCo-240K and a tailored classifier-free guidance strategy.
2.1. Preliminary
- Bagel combines a ViT vision encoder, a VAE encoder, and two transformer branches for visual understanding and generation.The ViT processes image tokens for understanding, while Rectified Flow produces VAE tokens for visual generation.Source: §2.1 · p. 3
2.2. Draft-as-CoT
- DraCo first sketches a low-resolution draft, verifies semantic misalignments using visual understanding, then refines it into a high-resolution image with guidance balancing draft consistency and prompt-directed correction.Draft-only visual features are used for verification; the CFG distinguishes unconditional, draft-only, and fully conditioned inputs.Source: §2.2 · p. 4
2.3.1. Necessity of Dataset Construction
- Bagel’s pilot correction failures include ignoring valid instructions and generating new images instead of adhering to draft semantics.The authors report that existing datasets do not address these challenges.Source: §2.3.1 · p. 4
2.3. DraCo-240K
- DraCo-240K contains over 240K interleaved reasoning instances targeting general correction, instance manipulation, and layout reorganization.The dataset addresses pilot-study difficulties following correction instructions and preserving draft semantics.Source: §2.3.1 · p. 4
2.3.2. Construction Details
- The construction pipelines create image pairs varying edited objects, same-category instance counts or attributes, or object layout while retaining relevant shared image content.The process also generates prompts and verifications from the pairs, then organizes training examples by whether modification is needed.Source: §2.3.2 · pp. 5–6, Figure 4 · p. 5
2.3.3. Training Set
- Training data includes modification-needed and no-modification-needed examples, plus text-to-image examples at draft and final-output resolutions.Qwen3-VL generates prompts and verifications from image pairs.Source: §2.3.3 · p. 6, §2.3.2 · p. 6
2.4. Training Loss
- Supervised fine-tuning applies binary cross-entropy to verifications and mean squared error to final-image VAE tokens, with condition dropout supporting DraCo-CFG.Training drops all conditions or preserves only the ViT feature, each with 5% probability.Source: §2.4 · p. 6
3. Experiment
DraCo is evaluated on three text-to-image benchmarks, where it outperforms other methods with chain-of-thought planning. Ablations examine draft-feature use and draft resolution.
3.1. Experimental Setting
- GenEval, GenEval++, and ImagineBench cover object existence, attributes, counting, positions, complex combinations, and unusual object-attribute combinations.Evaluation uses one image per prompt and the benchmarks’ official scoring methods.Source: §3.1 · pp. 6–7
- Bagel first receives text-to-image fine-tuning for valid 384×384 generation, followed by 16K full-parameter DraCo fine-tuning steps.Training freezes the ViT encoder and connector and uses EMA weights.Source: §3.1 · p. 7
3.2. Main Results
- 0.86 overall on GenEval and 0.40 on GenEval++; DraCo outperforms other CoT methods across all three benchmarks.On ImagineBench, it improves 0.91 points over Bagel without reasoning and 0.18 over text-only reasoning; it leads in five of six GenEval subtasks.Source: §3.2 · p. 7, Table 1 · p. 7, Table 2 · p. 8
3.3. Ablation Study
- DraCo-CFG is compared with Bagel’s original CFG using Table 3 and Figure 6.The supplied passages identify the comparison but do not report its outcome.Source: §3.3 · p. 7, Table 3 · p. 8, Figure 6 · p. 8
- 0.84 with draft VAE features is 2% lower than without them, and the features can cause artifacts through over-adherence to the draft.Reported artifacts include unnatural horse lighting, an improper horse-leg/sofa intersection, and a disconnected clock frame.Source: §3.3 · p. 8, Table 3 · p. 8, Figure 6 · p. 8
- 0.76 at 128×128 and 0.75 at 1024×1024 are unsatisfactory draft-resolution results for different reasons.The smaller draft limits planning and verification; the larger one increases token length and reduces training samples per fixed number of iterations.Source: §3.3 · p. 8, Table 3 · p. 8
4. Conclusion
DraCo combines visual and textual chain-of-thought for text-to-image generation, targeting coarse textual planning and rare attribute combinations. Its experiments report gains across three benchmarks.
4. Conclusion
- DraCo interleaves visual and textual chain-of-thought to address coarse-grained textual planning and rare attribute combinations.The conclusion describes rare-combination difficulty as associated with training-data biases.Source: §4 · p. 8
- +8% on GenEval, +0.91 on Imagine-Bench, and +3% on GenEval++ are the reported gains; DraCo-240K targets three correction capabilities, alongside DraCo-CFG.DraCo-CFG strengthens draft visual semantics and verification correction instructions.Source: §4 · p. 9
Appendix Overview
The appendix outlines related work on unified multimodal models and image-based reasoning, then contrasts prior image-reflection and editing approaches with DraCo.
Appendix Overview
- The appendix is organized into related work, DraCo-CFG details, dataset details, experiment details, limitations and future work, and qualitative examples.Source: Appendix Overview · p. 13
A. Related Work
- Unified multimodal generation approaches include external diffusion models [9] [50] [51], autoregressive image generation [6] [60] [63], and unified autoregressive-diffusion frameworks [8] [77].Bagel uses mixture transformer architectures for text and image VAE tokens.Source: A. Related Work · p. 13
- Visual-generation reasoning includes semantic-level CoT [21] and reflection methods [41] [74], with some methods refining prompts or editing generated images.DraCo differs from cited editing methods by pre-planning at draft resolution; those methods edit a full-resolution first image and may restrict editing content.Source: A. Related Work · p. 13
B.1. Original CFG of Bagel
Bagel’s original CFG uses multimodal contexts whose dropped conditions can couple image, edit, and CoT guidance. DraCo-CFG’s prompts specify draft generation and subsequent verification and modification.
B.1. Original CFG of Bagel
- Bagel’s “thinking before editing” setup conditions on image features and an edit instruction, generates textual CoT, then generates the edited image.Its CFG contexts do not fully decouple image, edit, and CoT conditions; dropping pairs can emphasize CoT twice.Source: §B.1 · p. 13
- Sequential CFG rounds couple stext and simg, and the text-only latent Mtext is controlled by both scales.Source: §B.1 · p. 14
B.2. System Prompt
- DraCo-CFG uses different system prompts to emphasize specific conditions; its draft prompt asks for a larger image with the same content and better detail.The full-condition prompt directs the model to generate a scratch image, check prompt alignment, and modify it if needed.Source: §B.2 · p. 14
C. More Dataset Details
DraCo-240K covers general correction, instance manipulation, and layout reorganization, with validation procedures for prompts involving counts and spatial relations.
C. More Dataset Details
- DraCo-240K includes general corrections such as object replacement and background modification, instance-level attribute changes, and layout changes such as swapping object positions.Instance manipulation also covers adding or removing objects and changing attributes; layout tasks include adding or removing objects relative to existing ones.Source: C. More Dataset Details · p. 14
- Prompts for instance manipulation and layout reorganization are cross-validated because current MLLMs can struggle with numeracy and spatial information.GroundingDINO detections are checked against prompt object counts and positions, and only aligned samples are retained.Source: C. More Dataset Details · p. 14
- Draft images are downsampled to 384 × 384 for training, and verification includes a conclusion specifying what should change.The draft images are roughly the same size as the final images before downsampling.Source: C. More Dataset Details · p. 14
D. More Experiment Details
Bagel is fine-tuned to produce valid low-resolution drafts before DraCo training, while retaining generation training at both draft and final resolutions.
D. More Experiment Details
- Bagel receives a text-to-image fine-tuning stage before DraCo training to generate valid small-resolution drafts.The stage uses 1024 × 1024 generated images resized to 384 × 384, including images generated from GPT-4o.Source: D. More Experiment Details · p. 14
- Training uses equal shares of 384 × 384 and 1024 × 1024 images to support generation at both resolutions.The model is fine-tuned for 14K steps, and DraCo training continues from the EMA weight.Source: D. More Experiment Details · p. 14
E. Limitations and Future Work
The authors identify limits in extending DraCo beyond text-to-image generation and in the study of human involvement in data curation and training.
E. Limitations and Future Work
- DraCo’s low-resolution draft is not directly or optimally suited to other modalities, including video, 3D assets, and scenes.For video generation, using a low-resolution draft still seems computationally expensive; modality-specific drafts and their target challenges remain unstudied.Source: E. Limitations and Future Work · p. 14
- The role of humans in improving data curation and the training loop is not well studied.The authors note that human involvement could help better align generation and correction methods.Source: E. Limitations and Future Work · pp. 14–15
F. Qualitative Examples
The qualitative examples show DraCo’s drafts, verification, and final outputs across correction tasks, and compare its outputs with Bagel and T2I-R1.
F. Qualitative Examples
- Verification analyzes draft–prompt inconsistencies, proposes a correction, and identifies what should remain unchanged before summarizing the final correction.The examples include position correction, rare attribute generation, precise object editing, and numeracy correction.Source: F. Qualitative Examples · p. 15
- One example removes a bottom-left pizza after verification identifies eight pizzas in the image versus seven in the prompt.The prompt requests “A photo of seven pizzas,” and the correction instruction is to remove the bottom-left pizzas.Source: F. Qualitative Examples · p. 16
- Another example adds a blue cup beside a blue toilet while preserving the toilet’s position.Verification identifies the missing cup and specifies that the toilet should remain unchanged.Source: F. Qualitative Examples · p. 16
- DraCo produces high-quality outputs with precise prompt alignment, while T2I-R1 examples show artifacts or oversaturation and Bagel examples do not accurately follow prompts.The comparisons include rare attribute generation and object-editing cases.Source: F. Qualitative Examples · p. 16