Source-linked AI summary
CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation
Haodong Li, Chunmei Qing, Huanyu Zhang, Dongzhi Jiang, Yihang Zou, Hongbo Peng, Dingming Li, Yuhong Dai, ZePeng Lin, Juanxi Tian, Yi Zhou, Siqi Dai, Jingwei Wu, Pheng-Ann Heng
TL;DR
Existing T2I reasoning methods rely on natural-language planning that is too imprecise for complex layouts and dense text. CoCo uses executable code to generate a deterministic visual draft, then refines it, with CoCo-10K supporting the workflow. It reports substantial gains over direct generation and other methods across challenging benchmarks.
Problem
Natural-language planning lacks the precision required for complex spatial layouts, structured visual elements, and dense textual content in text-to-image generation.
Method
CoCo generates executable code encoding spatial layouts and structural constraints, renders it as a draft, and performs fine-grained editing to produce the final image.
Results
CoCo substantially outperforms existing methods across StructT2IBench, Long-Text-Bench, and OneIG-Bench, including a 68.83% gain over Bagel on StructT2IBench.
Takeaways & Limitations
Executable code provides an explicit intermediate representation for structured T2I generation, while draft-to-final supervision yields higher-fidelity outputs with stronger semantic alignment.
Abstract
from arXiv · showhide
Recent advancements in Unified Multimodal Models (UMMs) have significantly advanced text-to-image (T2I) generation, particularly through the integration of Chain-of-Thought (CoT) reasoning. However, existing CoT-based T2I methods largely rely on abstract natural-language planning, which lacks the precision required for complex spatial layouts, structured visual elements, and dense textual content. In this work, we propose CoCo (Code-as-CoT), a code-driven reasoning framework that represents the reasoning process as executable code, enabling explicit and verifiable intermediate planning for image generation. Given a text prompt, CoCo first generates executable code that specifies the structural layout of the scene. The code is then executed in a sandboxed environment to render a deterministic draft image. Subsequently, the model refines this draft through fine-grained image editing to produce the final high-fidelity result. To support this training paradigm, we construct CoCo-10K, a curated dataset containing structured draft-final image pairs designed to teach both structured draft construction and corrective visual refinement. Empirical evaluations on StructT2IBench, OneIG-Bench, and LongText-Bench show that CoCo achieves improvements of 68.83%, 54.8%, and 41.23%, respectively, over direct generation, while also outperforming other CoT-enhanced generation methods. These results demonstrate that executable code is an effective and reliable reasoning paradigm for precise, controllable, and structured text-to-image generation.
1 Introduction
CoCo addresses the limited precision of natural-language planning in complex text-to-image generation by using executable code to create verifiable visual drafts and guide refinement. The paper introduces CoCo-10K and reports improvements across structured, text-intensive, and general T2I tasks.
- Motivation: Natural-language planning lacks the precision needed for complex spatial layouts, structural elements, and dense textual content.The limitation is especially evident in scientific diagrams and images requiring accurate layout, symbol placement, and semantic consistency.
- Motivation: Executable code encodes layouts, structural constraints, and textual placements deterministically, producing an observable draft for verification and targeted refinement.The rendered draft serves as an explicit visual scaffold that supports higher visual fidelity and stronger semantic alignment.
- Method: CoCo uses code as an explicit intermediate representation, renders it in a sandboxed environment, and applies fine-grained image editing to obtain the final result.This workflow replaces abstract planning with structured code that specifies spatial layouts, structural constraints, and object relationships.
- Training Data: CoCo-10K contains Text–Code pairs and Text–Draft–Final image triplets for learning executable layout planning and draft-guided refinement.The dataset addresses the lack of supervision for code generation and fine-grained correction signals.
- Results: 68.83% gain on StructT2IBench over Bagel, with a further 64.48% improvement over text-CoT-based approaches.The experiments also report robust generalization on Long-Text-Bench and OneIG-Bench.
- Results: CoCo substantially outperforms existing methods on tasks requiring precise layouts, complex text rendering, and structured visual generation.The reported validation covers StructT2IBench, LongText-Bench, and OneIG-Bench.
2 Related Work
Related work spans unified multimodal architectures, multimodal reasoning paradigms, and benchmarks for structured or text-intensive image generation. These studies motivate CoCo’s use of executable code as a structured medium connecting semantic intent with visual realization.
- Unified Multimodal Models: Unified multimodal models combine visual understanding and generation, but generation-side data and targeted training limitations constrain structured synthesis and complex text rendering.The cited discussion identifies this as a remaining challenge for UMMs.
- Multimodal Reasoning: Recent multimodal reasoning advances include LongCoT, visual outcomes within interleaved reasoning, and unified designs integrating text and image inputs and outputs.These developments extend reasoning beyond simple image understanding.
- Benchmarks: T2I benchmarks increasingly evaluate text rendering, long instructions, designer-level constraints, structured layouts, and text-intensive visual generation.Earlier benchmarks focused primarily on prompt–image alignment, while newer protocols probe more demanding capabilities.
- CoCo: CoCo introduces executable code as a structured reasoning medium to bridge semantic intent and visual realization.Its framework decomposes generation into code generation, draft rendering, and draft-guided refinement.
3 Method
CoCo decomposes text-to-image generation into executable code generation, deterministic draft rendering, and draft-guided refinement. CoCo-10K supplies paired supervision for learning structured draft construction and selective visual correction.
- CoCo Pipeline: CoCo generates executable code that encodes spatial layouts, object relationships, textual rendering, and canvas configurations.
- CoCo Pipeline: The generated code runs in a sandbox to produce a draft image capturing the specified semantic layout, object placement, text, and structural relationships.
- CoCo Pipeline: Draft-guided refinement improves visual fidelity while preserving the draft’s structural layout and semantic alignment.
- CoCo-10K: CoCo-10K combines Editing and Synthesis datasets to supervise structured perception, draft construction, and draft-guided refinement.
- CoCo-10K: The Editing Dataset pairs charts with corrected versions that modify values, labels, or formatting while preserving the overall structure.
- CoCo-10K: The Synthesis Dataset generates prompts, executable structural code, programmatic drafts, and visually refined outputs for scientific and text-intensive visuals.
4 Experiment
CoCo is evaluated on structured visuals and text-rendering tasks against generation-only, unified MLLM, and CoT-based methods. It achieves strong benchmark performance, while ablations show that draft refinement and executable-code supervision support reliable rendering and adaptive layouts.
- Evaluation: CoCo is evaluated on StructT2IBench, OneIG-Bench, and LongText-Bench against generation-only, unified MLLM, and unified MLLM with CoT planning methods.StructT2IBench covers structured visuals, while OneIG-Bench and LongText-Bench assess text rendering.
- Main Results: 73.52% overall accuracy on StructT2IBench surpasses GPT-Image’s 49.58% baseline.CoCo also leads on Chart, Graph, Math, and Table tasks requiring precise layouts and structured reasoning.
- Main Results: 0.853 overall score on OneIG-Bench and 0.754 on LongText-Bench demonstrate strong text-rendering performance across English and Chinese tasks.OneIG-Bench scores are 0.895 for English and 0.811 for Chinese; LongText-Bench scores are 0.755 and 0.753.
- Ablation Study: Text–Code pairs promote executable reasoning, while Text–Draft Image–Final Image triplets teach refinement from coarse drafts to higher-fidelity images.The training mixture combines complementary supervision sources for code generation and draft-guided refinement.
- Ablation Study: 100% compilation success after fine-tuning contrasts with Bagel’s 9.06% success rate on LongText-Bench.Bagel compiled only 29 of 320 generated programs, whereas CoCo’s training pipeline produced executable code consistently.
- Analysis: CoCo adapts canvas size and aspect ratio to prompt semantics despite training data fixed at 1024 resolution.Poster prompts often produce wider layouts, while charts and diagrams typically produce square or near-square canvases.
5 Conclusion
CoCo uses executable code as an intermediate reasoning representation for structured text-to-image generation. It renders a draft from code, refines that draft through image editing, and reports improvements on challenging benchmarks.
- Conclusion: CoCo formulates executable code as Chain-of-Thought for text-to-image generation.The code explicitly specifies spatial layouts and textual elements before image generation.
- Conclusion: The model executes structured code to create an intermediate draft image, then performs fine-grained editing to obtain the final result.CoCo-10K provides Text-Code pairs and Text-Draft Image-Final Image triplets for this workflow.
- Conclusion: CoCo reports substantial improvements on StructT2IBench, OneIG-Bench, and LongText-Bench for precise, controllable, and structured generation.The conclusion attributes the reported capability to executable code as an intermediate reasoning representation.