Source-linked AI summary
IV-CoT: Implicit Visual Chain-of-Thought for Structure-Aware Text-to-Image Generation
Zixuan Li, Haokun Lin, Yicheng Xiao, Zhiwei Li, Xinyang Song, Zelong Zheng, Yong He, Heng Yao, Ke Ding, Chao Yu, Chuan Yuan, Qi Li, Zhenan Sun
TL;DR
Structure-aware text-to-image generation still struggles to preserve object counts, spatial relations, attribute bindings, and layouts because structural planning and appearance rendering are entangled. IV-CoT separates these roles through a latent structural-to-semantic cascade with training-only sketch supervision, improving GenEval from 0.86 to 0.88 and T2I-CompBench from 0.5448 to 0.5743.
Problem
Structure-aware text-to-image generation remains limited in preserving object counts, spatial relations, attribute bindings, and layouts under entangled conditioning.
Method
IV-CoT uses a latent structural-to-semantic query cascade with training-only sketch supervision to plan structure before rendering appearance in one forward pass.
Results
GenEval improves from 0.86 to 0.88 and T2I-CompBench from 0.5448 to 0.5743 versus OpenUni-L-1024, while retaining single-pass inference.
Takeaways & Limitations
IV-CoT supports structure-aware prompt following while keeping visual planning latent, efficient, and represented through recoverable, manipulable structural queries.
Takeaways & Limitations
IV-CoT does not explicitly model readable scene-text rendering and has not been evaluated for image editing scenarios.
Abstract
from arXiv · showhide
Unified multi-modal large language models (MLLMs) have achieved strong text-to-image generation quality, but still struggle with structure-aware prompt following, where object counts, spatial relations, attribute bindings, and coarse layouts must be preserved. We attribute this limitation in part to the entanglement of structural planning and appearance rendering within a single conditioning stream. To address this issue, we propose Implicit Visual Chain-of-Thought (IV-CoT), a latent visual reasoning framework for query-conditioned image generation. IV-CoT decomposes the visual conditioning queries into a structural-to-semantic cascade, where structural queries first form a latent visual plan and semantic queries then render appearance conditioned on this plan. To guide the structural queries, we introduce training-only sketch supervision, which encourages them to capture structure from sketches without requiring sketch extraction or intermediate decoding at inference time. IV-CoT performs implicit CoT reasoning in a single forward pass and achieves superior results on GenEval and T2I-CompBench. Visualizations and analyses demonstrate that the learned structural and semantic queries play complementary roles in structure-aware generation.
1 Introduction
IV-CoT addresses structure-aware prompt-following failures by separating latent structural planning from appearance rendering in a causal query cascade. Training-only sketch supervision shapes structural queries without requiring intermediate outputs at inference, while the method improves reported benchmarks and preserves single-pass generation.
- Motivation: Unified MLLM generators can produce plausible images that swap attributes, omit objects, or violate layouts when prompts require complex structures.Structure-aware prompt following includes preserving object shapes, materials, attributes, counts, and spatial arrangements.
- Motivation: Entangled conditioning compresses scene structure, object identity, attributes, and appearance together, making structural and visual controls difficult to distinguish.The paper motivates separating what determines scene structure from what controls visual appearance.
- IV-CoT: IV-CoT places structural queries before semantic queries so they encode a latent plan for object form, count, layout, and coarse spatial relations before appearance rendering.Semantic queries attend to the structural plan to render appearance and fine-grained details.
- Training: Training-only sketch supervision teaches structural queries to capture contours, shapes, counts, and layouts while suppressing color, texture, and lighting factors.A retained structural objective regularizes image-generation training, and sketches are not required as external controls at inference.
- Results: 0.88 GenEval and 0.5743 T2I-CompBench improve over 0.86 and 0.5448, while IV-CoT retains single-pass inference with 9-15× lower latency than explicit CoT methods.The same OpenUni-L-1024 backbone is used for the reported benchmark comparison.
2 Method
IV-CoT separates structural planning from appearance rendering through an ordered structural-to-semantic query cascade in a query-conditioned MLLM-DiT generator. Sketch supervision trains structural queries to encode visual plans, while semantic queries refine appearance and inference performs this reasoning in one forward pass without intermediate image decoding.
- Backbone: The unified MLLM-DiT backbone maps MLLM query states into DiT conditioning, which guides iterative denoising from noisy image latents to decoded images.The original flat query sequence entangles layout, shape, attributes, color, and texture, motivating explicit role separation.
- Query Cascade: IV-CoT partitions visual query inputs into structural and semantic groups, whose causal ordering creates a one-way structural-to-semantic dependency.Structural queries attend to the prompt and structural inputs, while semantic queries can additionally attend to structural query states.
- Structural Training: Sketch-supervised structural training uses PiDiNet-extracted sketches and frozen MLLM-DiT components to force structural queries to encode contours, shapes, counts, and coarse layouts.Only the structural query inputs are trained in this stage, and sketch latents share the diffusion VAE latent space with image latents.
- Image Training: Image-generation training jointly optimizes the diffusion generator and query inputs while retaining structural loss to prevent structural queries from drifting toward appearance-only cues.The semantic branch completes identity, color, material, and texture details conditioned on structural queries.
- Inference: During inference, IV-CoT produces structural and semantic queries in one MLLM pass, then standard diffusion decoding generates the image without sketch extraction, intermediate images, or test-time search.The structural plan remains internal to the query sequence, while the visible output follows the usual image-generation process.
3 Experiment
Experiments show that IV-CoT improves structure-aware text-to-image generation while preserving inference efficiency. Ablations and visual analyses attribute the gains to complementary structural supervision, query cascading, and separable latent query functions.
- Main performance: 0.88 on GenEval and 0.5743 on T2I-CompBench improve over OpenUni-L-1024’s 0.86 and 0.5448, especially on structure-sensitive dimensions.Gains include position and color attribution in GenEval and spatial relation, shape, texture, and color in T2I-CompBench.
- Efficiency and qualitative results: 1.693 seconds per sample gives IV-CoT the highest T2I-CompBench overall score while achieving 9.01×, 9.89×, and 14.98× lower latency than T2I-R1, GoT-R1, and TWIG-RL.Qualitative comparisons also show better preservation of object count, spatial placement, attribute binding, and visual organization at comparable image quality.
- Ablation study: Full IV-CoT outperforms controlled variants, while more queries provide limited gains and removing either sketch supervision or the ordered cascade weakens performance.The ablation results indicate that structural supervision and the structural-to-semantic cascade are complementary rather than interchangeable.
- Query analysis: Structural queries produce recoverable sketch-like plans and actively determine layout and coarse shape, while semantic queries attend more broadly to object interiors and appearance regions.Randomly replacing structural queries disrupts object layout, contours, and coarse shape; attention patterns suggest soft rather than hard functional specialization.
- Zero-shot recombination: Without additional training, recombined queries often preserve prompt A’s coarse layout or object configuration while adopting salient appearance attributes from prompt B.This behavior suggests partially controllable structure-appearance separation and manipulable latent visual plans.
4 Related Work
Related work improves compositional image generation through explicit intermediate reasoning and latent or continuous deliberation. IV-CoT instead uses structure-first queries in the MLLM-DiT interface without intermediate decoding or test-time latent search.
- Explicit reasoning for image generation: Recent image-generation methods improve compositionality by externalizing intermediate reasoning.
- Latent and continuous reasoning: Latent and continuous reasoning carries intermediate deliberation through hidden states, soft thought tokens, or latent visual and multimodal tokens.
- Latent and continuous reasoning: IV-CoT structures the MLLM-DiT conditioning interface with structure-first queries, without intermediate decoding or test-time latent search.
5 Conclusion
IV-CoT is a structure-first latent reasoning framework that separates structural and semantic conditioning queries for text-to-image generation. Training-only sketch supervision and a structural-to-semantic cascade enable latent visual planning without intermediate sketch or rationale decoding at inference time, improving structure-aware prompt following efficiently.
- IV-CoT organizes text-to-image conditioning queries into structural and semantic roles within a structure-first latent reasoning framework.
- Training-only sketch supervision guides structural queries, while a structural-to-semantic cascade keeps visual planning in latent representations.The method does not require intermediate sketch decoding or textual rationale decoding at inference time.
- Experiments and analyses on GenEval and T2I-CompBench show improved structure-aware prompt following with high efficiency.
Limitations
The work targets structure-aware prompt following but is not specifically optimized for readable text rendering in images. Its current objectives do not explicitly model the fine-grained alignment, spelling consistency, and typography-aware supervision required for accurate scene text.
- Limitations: IV-CoT is not specifically optimized for rendering readable text within images.Sketch supervision captures contours, layouts, and object configurations, but accurate scene-text rendering requires character-level alignment, spelling consistency, and typography-aware supervision absent from the current objectives.
A Implementation Details
IV-CoT is instantiated on OpenUni-L with separate structural and semantic query groups, trained on a combined 128,393-pair dataset. Its optimization uses AdamW with bfloat16 mixed precision, warmup, and cosine learning-rate decay.
- Backbone and query configuration: IV-CoT uses OpenUni-L, combining a 2B InternVL3 MLLM with a 1.6B Sana diffusion generator and 512 concatenated visual queries.The 512 queries comprise two groups of 256; semantic queries use the pretrained OpenUni checkpoint, while structural queries use the Stage-1 checkpoint.
- Training data: Training uses 128,393 image-text pairs combined from BLIP3o, ShareGPT-4o-Image, and Echo-4o.
- Optimization: Optimization uses bfloat16 mixed precision and AdamW with learning rate 2 × 10−5, β = (0.9, 0.95), weight decay 0.05, and gradient clipping at 1.0.The learning rate warms up for the first 10% of training steps, then follows cosine decay to 1 × 10−7; the default random seed is 42.
B Additional Generation Samples
Figure 7 presents additional IV-CoT generation samples spanning diverse objects, scenes, and visual styles. These examples illustrate broad generation capability alongside coherent visual structures.
- Additional Generation Samples: Figure 7 shows IV-CoT samples covering diverse object categories, scenes, and visual styles.The examples provide additional generation samples from IV-CoT.
- Additional Generation Samples: The samples illustrate broad generation capability while preserving coherent visual structures.
C Latency Measurement
The paper measures end-to-end generation latency under a standardized single-GPU setup, averaging inference time across 100 prompts while excluding model loading.
- Measurement Setup: Latency is measured on one NVIDIA A800 80GB GPU with batch size 1, averaging wall-clock inference time over 100 prompts.Reported time excludes model loading and includes text processing, method-specific reasoning or intermediate generation, and final image synthesis.
D Attention Analysis
The attention analysis characterizes how structural and semantic queries share cross-attention during rendering. Across denoising steps and layers, structural attention shifts from deeper global-layout recovery toward shallower local object regions as coarse structure emerges.
- Relative attention proportion: Relative attention maps measure normalized allocation between structural queries Qs and semantic queries Qm, not absolute attention magnitude.The query groups contain equal numbers of queries, so group-wise normalization is not confounded by query-group size.
- Layer- and step-wise visualization: Figure 8 visualizes intermediate images and complementary Qs and Qm attention proportions across denoising steps and diffusion-transformer layer groups.Columns represent increasing denoising steps, while rows group consecutive layers.
- Layer- and step-wise visualization: Early in denoising, structural patterns are more visible in deeper layers, suggesting global information aggregation for coarse-layout recovery.The latent image state is still noisy at these early steps.
- Layer- and step-wise visualization: As denoising progresses, structural patterns also appear in shallower layers, indicating increasing alignment with local object regions after coarse structures emerge.The semantic-query maps show complementary patterns.
E Use of AI Assistants
AI assistants supported language polishing, wording suggestions, and submission-form preparation, while the authors reviewed and verified all technical content, experiments, analyses, claims, and final text.
- AI assistants were used for language polishing and wording suggestions.
- They also assisted with preparation of the submission form.
- The authors reviewed and verified all technical content, experiments, analyses, claims, and final text.