Source-linked AI summary
CoF-T2I: Video Models as Pure Visual Reasoners for Text-to-Image Generation
Chengzhuo Tong, Mingkun Chang, Shenglong Zhang, Yuran Wang, Cheng Liang, Zhizheng Zhao, Ruichuan An, Bohan Zeng, Yang Shi, Yifan Dai, Ziming Zhao, Guanbin Li, Pengfei Wan, Yuanxing Zhang, Wentao Zhang
TL;DR
Text-to-image generation lacks clearly defined visual reasoning starts and interpretable intermediate states, while existing approaches rely on indirect pixel correction or textual planning. CoF-T2I uses pretrained video models for three-step progressive visual refinement, supported by a 64K trajectory dataset and frame-wise encoding. It significantly outperforms the base video model and reaches 0.86 on GenEval and 7.468 on Imagine-Bench, although broader video- and 3D-related extensions remain unexplored.
Problem
Text-to-image generation lacks a clear visual reasoning starting point and interpretable intermediate states for progressive visual refinement.
Method
CoF-T2I uses pretrained video models to generate three progressively refined frames, trained with 64K CoF-Evol-Instruct trajectories and independently encoded frames.
Results
CoF-T2I significantly outperforms the base video model and achieves 0.86 on GenEval and 7.468 on Imagine-Bench.
Takeaways & Limitations
Video foundation models’ intrinsic Chain-of-Frame reasoning shows promise for advancing high-quality text-to-image generation.
Takeaways & Limitations
The study does not systematically explore extensions to broader video-related or 3D-related task domains.
Abstract
from arXiv · showhide
Recent video generation models have revealed the emergence of Chain-of-Frame (CoF) reasoning, enabling frame-by-frame visual inference. With this capability, video models have been successfully applied to various visual tasks (e.g., maze solving, visual puzzles). However, their potential to enhance text-to-image (T2I) generation remains largely unexplored due to the absence of a clearly defined visual reasoning starting point and interpretable intermediate states in the T2I generation process. To bridge this gap, we propose CoF-T2I, a model that integrates CoF reasoning into T2I generation via progressive visual refinement, where intermediate frames act as explicit reasoning steps and the final frame is taken as output. To establish such an explicit generation process, we curate CoF-Evol-Instruct, a dataset of CoF trajectories that model the generation process from semantics to aesthetics. To further improve quality and avoid motion artifacts, we enable independent encoding operation for each frame. Experiments show that CoF-T2I significantly outperforms the base video model and achieves competitive performance on challenging benchmarks, reaching 0.86 on GenEval and 7.468 on Imagine-Bench. These results indicate the substantial promise of video models for advancing high-quality text-to-image generation.
1. Introduction
CoF-T2I repurposes pretrained video models as pure visual reasoners for text-to-image generation, using progressive frame evolution instead of textual planning. The method introduces structured supervision and achieves competitive benchmark performance.
- Motivation: Video models support frame-by-frame visual inference, but their use for text-to-image generation remains underexplored because clear starting points and interpretable intermediate states are lacking.Existing approaches rely on external verifiers or textual planning, which can make pixel-level correction indirect and lossy.
- CoF-T2I: CoF-T2I generates a compact three-frame sequence that progresses from coarse layout through refinement to a high-fidelity final image.Only the final frame is fully decoded and used as the output image.
- Data: CoF-Evol-Instruct provides 64K progressive visual-refinement trajectories spanning initial semantic composition to final aesthetic refinement.Its quality-aware pipeline supplies defect-aware, step-wise supervision for training.
- Results: CoF-T2I significantly outperforms its video backbone and achieves competitive results on challenging prompt-following and compositional-reasoning benchmarks.The reported benchmarks include GenEval and Imagine-Bench.
2. Methodology
CoF-T2I reframes text-to-image generation as progressive visual reasoning over video-model latent trajectories, using intermediate frames for refinement and decoding only the final frame. The method is trained with frame-wise representations and CoF-Evol-Instruct, a quality-aware dataset of structured three-frame refinement sequences.
- CoF-T2I: CoF-T2I uses a video generation backbone to model a three-frame latent trajectory that progresses from coarse semantics to fine-grained aesthetics.The trajectory represents distinct reasoning steps, with the terminal latent producing the output image.
- CoF-T2I: Only the terminal latent z3 is decoded into the final image, while the preceding latent states encode progressive visual refinement.The model learns the joint distribution of the full latent sequence and projects z3 through the decoder.
- Frame-wise Representation: Frame-wise latent encoding independently represents each frame to reduce motion artifacts introduced by native spatiotemporal video-VAE compression.The native compression may introduce implicit flow and dynamic inconsistencies, motivating independent frame encoding.
- Training and Inference: Training applies flow matching to latent sequences so later frames refine earlier frames through end-to-end optimization from Gaussian-noise initialization.The model predicts denoising targets for z1:3 corresponding to the CoF sequence and generates the full latent sequence at inference.
- CoF-Evol-Instruct: The dataset separates semantic correction from aesthetic improvement because existing single-image and low-quality multi-step datasets do not provide reliable intermediate supervision.Its fixed length of three preserves these refinement stages while maintaining causal progression from defective drafts to high-quality outputs.
- CoF-Evol-Instruct: CoF-Evol-Instruct contains 64K three-frame CoF sequences curated through a quality-aware pipeline for progressive, diverse, and consistent visual supervision.The pipeline samples images across model tiers, filters failures, and expands valid samples using forward refinement, bidirectional completion, or backward synthesis.
3. Experiments
CoF-T2I is evaluated on GenEval and Imagine-Bench against standard image generators and multimodal models using textual intermediate reasoning. Results show strong benchmark performance, progressive gains across reasoning frames, and benefits from intermediate supervision and independent frame encoding.
- Evaluation: GenEval evaluates object-centric prompt following, while Imagine-Bench tests imaginative prompts involving controlled concept transformations and compositional reasoning.Both benchmarks are evaluated under their official protocols with overall and category-wise scores.
- Main Results: 0.86 overall on GenEval, with CoF-T2I surpassing BAGEL-Think by 0.04 and T2I-R1 by 0.07.The comparison includes standard image models and unified multimodal models using textual Chain-of-Thought.
- Main Results: 7.468 overall on Imagine-Bench versus 5.939 for Wan2.1, including 7.797 versus 5.383 in the Multi-Object category.The gains are reported especially for complex composition and imaginative instructions.
- Intermediate Supervision: The Target-Only SFT comparison tests whether intermediate frames contribute beyond final-frame supervision, using identical training settings.The reported disparity indicates benefits from explicitly learning the generative trajectory.
- Reasoning Trajectory: GenEval rises from 0.56 at F1 to 0.79 at F2 and 0.86 at F3 across the three-step reasoning chain.The passage reports monotonic gains across all subtasks and describes iterative visual self-correction.
- Robustness and Encoding: CoF-T2I improves GenEval across Wan2.1-T2V backbones at 1.3B and 14B parameters, while independent frame encoding outperforms the continuous causal video-VAE variant scoring 0.83.The relative gain is reported as more pronounced for the 1.3B model.
4. Conclusion
The paper presents CoF-T2I as a text-to-image foundation model that uses pretrained video backbones for pure visual reasoning. It combines multi-step frame evolution with CoF-Evol-Instruct and reports strong benchmark performance for progressive image refinement.
- Conclusion: CoF-T2I repurposes pretrained video generation backbones as pure visual reasoners for text-to-image generation.The model performs multi-step frame evolution at inference time without textual planning.
- Conclusion: CoF-T2I progressively corrects semantic errors and refines perceptual quality through frame evolution, with the final frame serving as the generated image.The approach is framed as an explicit visual reasoning process.
- Conclusion: CoF-Evol-Instruct provides step-wise supervision for progressive visual refinement.The conclusion identifies the dataset as part of the model's training framework.
- Conclusion: Experiments on GenEval and Imagine-Bench show strong improvement over the base model and competitive performance on challenging prompt-following and compositional reasoning settings.The conclusion positions video-derived temporal reasoning as a direction for future text-to-image generation.
B. More Dataset Details
The dataset categorizes prompts by the dominant constraint that must change while other content remains invariant. Categories cover object properties, composition, quantity, spatial relations, and global context.
- Prompt Categorization: Prompt categories are defined by the primary constraint that must change while other content remains invariant.This categorization makes semantic-stage edits targeted and controllable.
- Attribute Binding: Attribute Binding changes an object's intrinsic property, such as color, shape, size, or material, while preserving object identity and scene structure.
- Object Combination: Object Combination concerns multi-object composition, including co-existence, unusual juxtaposition, and hybrid concepts.
- Quantity Control: Quantity Control targets the specific number of object instances required by the prompt.
- Spatial Arrangement: Spatial Arrangement specifies relative relations such as left/right, above/below, in/on, or front/behind while preserving object identities.
- Context Manipulation: Context Manipulation modifies scene-level conditions such as era, time, background, location, or environment.
B.2. Implementation Details.
The implementation uses quality-aware routing and a closed-loop planner-editor-verifier pipeline to construct and validate CoF image transitions. Resolution and retry policies vary by refinement stage to balance efficiency and precision.
- Quality-based Routing: Quality-based Routing classifies anchor images as Semantically Misaligned (F1), Visually Unrefined (F2), or High Fidelity (F3).The classification jointly considers semantic alignment and aesthetic quality.
- Quality Assessment: The quality assessor receives a prompt and generated image and assigns one of three labels using semantic alignment and aesthetic quality.F1 denotes semantic errors, F2 semantic correctness with low aesthetics, and F3 semantic correctness with high aesthetics.
- Unified Editing Primitive: The Unified Editing Primitive uses a planner, editor, and verifier to produce category-conditioned stage transitions.The planner creates an editing instruction, the editor applies it, and the verifier checks the result.
- Verification Logic: Forward transitions must improve semantics or aesthetics, while backward transitions must explicitly degrade or introduce errors without changing unrelated content.The verifier outputs success only when the intended transition is clearly executed.
- Resolution Strategy: Planner and verifier inputs use 512 × 512 resolution for F1–F2 transitions and preserve 1024 × 1024 for F2–F3 transitions.The policy reduces computation for semantic checks while retaining detail for aesthetic assessment.
- Retry Policy: The pipeline retries rejected edits up to K = 3 times before falling back to direct regeneration with Qwen-Image.
C. More Experiment Details
This section adds ImagineBench results and describes implementation details for continuous video VAE encoding and the system prefix used during training and inference.
- Additional ImagineBench results and implementation details cover continuous video VAE encoding and the system prefix.
C.1. Additional Results on Imagine-Bench
The section evaluates target-only supervision and analyzes how ImagineBench quality changes across the three-frame reasoning chain.
- Target-only SFT evaluates fine-tuning solely on final output frames without intermediate reasoning supervision.Per-type average scores are reported in Table 6.
- Table 7 reports per-type ImagineBench scores for F1 → F2 → F3 to assess generation quality at each reasoning step.The table is intended to show progressive refinement from draft to final output.
C.2. Implementation Details
The implementation adapts the video VAE to three-frame reasoning chains and uses a fixed prompt prefix to induce Chain-of-Frame refinement.
- Three-frame chains are padded to five frames by repeating the final frame for compatibility with the native 1 + 4n temporal compression schedule.The transformation is (F1, F2, F3) → (F1, F2, F3, F3, F3).
- Only the last decoded frame is retained as the final text-to-image output; all other decoded frames are discarded.
- A fixed system prefix instructs the model during training and inference to generate a short refinement chain preserving concept and composition.The prompt requests step-by-step image improvement from the user prompt.
D. Limitations and Future Work
The authors identify broader task extensions and reinforcement learning as unresolved directions, while presenting qualitative trajectory and baseline comparisons.
- Extension beyond text-to-image, including text-to-video and text-to-3D synthesis, has not been systematically explored.
- Text-to-video extensions may require longer temporal sequences, greater computation, dynamic coherence, and avoidance of unintended motion artifacts.
- Reinforcement learning remains underexplored for this video-based visual reasoning framework.Future integration is proposed to support more adaptive visual refinements and potentially improve robustness across diverse prompts.
- Figure 7 visualizes complete reasoning trajectories and compares CoF-T2I with Wan2.1-T2V-14B and BAGEL-Think across diverse generation scenarios.The comparisons include imaginative object combinations, attribute binding, and spatial arrangement.