Source-linked AI summary
VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System
Haodong Li, Tianfei Ren, Xiaoxiao Ma, Chunmei Qing, Zhen Fang, Sipeng He, Ziyu Guo, Haoyu Wu, Juanxi Tian, Yihang Zou, Ruichuan An, Dongzhi Jiang, Boxue Yang, Ji Xie, Xu Huang, Wenhao Yan, Jialv Zou, Zhengrong Yue, Yaxin Luo, Xiaotong Li, Yuzhu Wang, Junyan Ye, Jinjing Zhao, Zehui Chen, Lin Chen, Renye Yan, Feng Zhao, Pheng-Ann Heng
TL;DR
Text-to-video models struggle to make dynamics physically consistent because prompts compress complete spatiotemporal processes into sparse language. VideoCoCo externalizes that process as executable Blender code, simulates it, and edits the resulting draft into photorealistic video, achieving state-of-the-art average performance on PhyGenBench and VBench-2.0.
Problem
Text prompts leave physical evolution largely unstated, creating a gap between compressed intent and the complete spatiotemporal process required for physically consistent video.
Method
VideoCoCo uses executable Blender code to simulate a deterministic spatiotemporal draft, then a generative video engine converts it into photorealistic video through draft-conditioned editing.
Results
0.475 to 0.558 on PhyGenBench; VideoCoCo achieves state-of-the-art average performance on both PhyGenBench and VBench-2.0.
Takeaways & Limitations
Executable code provides an effective, controllable, and inspectable process-level intermediate representation for physically consistent video generation.
Takeaways & Limitations
The approach adds inference latency and depends on Blender’s expressiveness, making highly complex phenomena such as turbulent fluids challenging to synthesize zero-shot.
Abstract
from arXiv · showhide
Text-to-video models have achieved remarkable visual quality, yet they still struggle to generate physically consistent dynamics because the temporal evolution of a scene must be inferred implicitly from a highly compressed text prompt. Existing chain-of-thought approaches introduce intermediate plans or visual states, but these representations are typically non-executable or temporally sparse, limiting their ability to instantiate and control the complete spatiotemporal process. To address this limitation, we introduce VideoCoCo, an agentic dual-engine framework in which executable Blender code serves as a process-level chain of thought. Given a text prompt, a coding agent synthesizes a Blender program that explicitly specifies the scene and its temporal evolution. The executable simulation engine runs the program to produce a deterministic spatiotemporal draft, which is subsequently transformed into a photorealistic video by a generative video engine through draft-conditioned editing. This decomposition separates process-level reasoning from high-fidelity visual realization. To adapt the video editor to simulated drafts, we construct VideoCoCo-3K, a curated dataset of draft-instruction-target triplets. VideoCoCo improves the OmniWeaving baseline from 0.475 to 0.558 on PhyGenBench and from 52.18 to 77.88 on VBench-2.0, achieving the best average score on both benchmarks. These results demonstrate that executable code provides an effective, controllable, and inspectable intermediate representation for physically consistent video generation.
1 Introduction
VideoCoCo addresses the difficulty of recovering physically consistent spatiotemporal processes from compressed text prompts by using executable Blender code as a process-level chain of thought. Its dual-engine framework simulates deterministic drafts, edits them into photorealistic videos, and is supported by the VideoCoCo-3K aligned dataset.
- Motivation: Compressed text prompts leave physical evolution largely unstated, requiring video models to recover complete spatiotemporal processes while synthesizing appearance.
- Motivation: Existing video chain-of-thought methods use plans, layouts, keyframes, visual states, or trajectory search, but their intermediate representations remain largely non-executable or temporally sparse.
- VideoCoCo framework: VideoCoCo uses a coding agent to synthesize Blender code that explicitly specifies a scene and its temporal evolution.
- VideoCoCo framework: The simulation engine runs Blender code in a sandboxed environment to produce a deterministic spatiotemporal draft, which the generative video engine transforms into a photorealistic video through draft-conditioned editing.
- VideoCoCo-3K: VideoCoCo-3K provides draft–instruction–target triplets generated through an agentic data-construction pipeline to adapt the video engine to simulated drafts.
- Results: VideoCoCo achieves state-of-the-art average performance on PhyGenBench and VBench-2.0, substantially improving OmniWeaving, especially for thermal and material dynamics.
2 Related Work
Related work spans advances in diffusion-based text-to-video generation, physically plausible video generation, and visual chain-of-thought reasoning. These approaches range from stronger video backbones and world representations to explicit physical controls and intermediate reasoning representations.
- Text-to-Video Generation: Diffusion models dominate text-to-video generation, with progress from early temporally coherent synthesis to latent-diffusion methods and large-scale video diffusion backbones.Representative systems include Open-Sora, HunyuanVideo, Wan, CogVideoX, and Step-Video-T2V.
- World Representations: Video diffusion models have also been explored as general visual and world representations, alongside reconstruction alignment, explicit 3D geometry coupling, and scalable multi-agent, multi-view modeling.
- Physically Plausible Generation: Physically plausible video generation follows explicit or implicit paradigms: explicit methods use simulators, trajectory guidance, or structured constraints, but may generalize poorly beyond predefined phenomena.Implicit methods aim to reduce reliance on hand-specified structure by learning physical priors from data.
- Visual Chain-of-Thought: Visual chain-of-thought methods for video generation externalize reasoning through textual plans, layouts, or keyframes, or embed it within sequences as intermediate visual states.Examples include VChain, VideoDirectorGPT, VideoRLVR, and ChEaP.
3 Method
VideoCoCo uses executable Blender code to externalize a prompt’s physical process into a deterministic, temporally dense draft, then uses draft-conditioned editing to realize that process photorealistically. VideoCoCo-3K supplies aligned draft–instruction–target triplets for adapting the video editor.
- Dual-Engine Framework: VideoCoCo separates process-level reasoning from visual realization with an executable simulation engine and a generative video engine.The simulation engine produces a deterministic spatiotemporal draft, while the generative engine converts it into a photorealistic video.
- Code-as-CoT Program Synthesis: Blender Python explicitly declares the scene, objects, physical properties, interactions, and temporal evolution, making the reasoning executable rather than a textual plan or sparse keyframes.The program is self-contained and commits the agent to a concrete process.
- Sandboxed Execution and Draft Rendering: Sandboxed Blender execution yields a deterministic, temporally dense white-clay draft in which each frame represents an instantiated state from initial condition to final outcome.The controlled runtime also returns rendering errors as diagnostic signals for code revision.
- Draft-Conditioned Video Editing: The instruction agent specifies subjects, materials, lighting, and cinematic style without redefining motion, while the draft anchors the process for the draft-conditioned video editor.This assigns motion to the draft and appearance to the instruction, so the editor restyles an instantiated process rather than imagining one from scratch.
- VideoCoCo-3K Dataset: VideoCoCo-3K provides draft–instruction–target triplets pairing simulated white-clay processes with photorealistic videos that share their spatiotemporal evolution.The dataset is constructed by generating drafts and instructions, then obtaining photorealistic targets from a high-fidelity teacher editor.
4 Experiments
VideoCoCo improves physical consistency on PhyGenBench and VBench-2.0 by combining executable Blender drafts with adapted video editing. Ablations show drafting supplies the physical dynamics, while lightweight LoRA adaptation further improves photorealistic realization.
- PhyGenBench: VideoCoCo raises OmniWeaving’s PhyGenBench average from 0.475 to 0.558, achieving the best overall result.It leads on mechanics (0.558), optics (0.613), and material (0.525), while ranking second on thermal (0.511).
- PhyGenBench: +0.133 and +0.078 are VideoCoCo’s largest PhyGenBench gains over OmniWeaving on material and thermal, respectively.These categories are identified as weaknesses of appearance-driven generators lacking explicit dynamics references.
- VBench-2.0: 25.70 points separate VideoCoCo from OmniWeaving on VBench-2.0 average plausibility, rising from 52.18% to 77.88%.VideoCoCo achieves the best average among all systems and leads mechanics (92.31%) and thermotics (72.92%).
- Ablation: 0.558 is the best PhyGenBench ablation average, achieved by LoRA tuning after tuning-free editing already improves the base from 0.475 to 0.506.Holding executable draft generation fixed isolates the contributions of drafting and editor adaptation.
- Ablation: 0.558 vs. 0.535 shows LoRA tuning outperforming full fine-tuning despite updating fewer parameters.The adaptation restyles simulation drafts while preserving motion, retaining the base generator’s visual priors.
- Qualitative results: VideoCoCo realizes correct dynamics with photorealistic appearance in examples where OmniWeaving produces plausible frames that violate the intended process.The qualitative examples include dry-ice sublimation and vacuum-induced bottle collapse.
5 Conclusion
VideoCoCo uses executable code as a process-level chain of thought in an agentic dual-engine framework, separating spatiotemporal simulation from photorealistic editing to enforce physical consistency. It achieves state-of-the-art physical plausibility, while incurring additional inference latency and remaining limited by Blender’s expressiveness.
- Conclusion: VideoCoCo uses executable code as a process-level chain of thought within an agentic dual-engine framework for physically consistent text-to-video generation.The framework was introduced to enforce physical consistency through executable code.
- Conclusion: Decoupling spatiotemporal simulation from photorealistic editing enables VideoCoCo to achieve state-of-the-art physical plausibility on PhyGenBench and VBench-2.0.
- Conclusion: The approach adds inference latency and is bounded by Blender’s expressiveness, making highly complex phenomena such as turbulent fluids challenging to synthesize zeroshot.
- Conclusion: Future work will integrate specialized physical engines such as Taichi and investigate knowledge distillation to internalize executable priors into end-to-end video models.