VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System

• Text prompts leave physical principles unstated, requiring models to infer a complete spatiotemporal process while synthesizing appearance; the paper calls this mismatch Causal Opacity.Source: §1 · p. 2
• Prior video chain-of-thought methods use text, layouts, or keyframes [1] [2], generated visual states [3] [4], or trajectory search and revision [5] [6], but their intermediates remain descriptive or selective.They do not instantiate a complete process that can be directly inspected and revised.Source: §1 · p. 3
• A coding agent creates a Blender program specifying the scene and its temporal evolution; simulation produces a deterministic draft that a video engine transforms into a photorealistic video through draft-conditioned editing.This division separates process-level reasoning from visual realization.Source: §1 · p. 3
• VideoCoCo-3K is a dataset of draft–instruction–target triplets, built through an agentic pipeline to adapt the video engine to simulated drafts while retaining their spatiotemporal structure.Source: §1 · p. 3
• VideoCoCo achieves state-of-the-art average performance on PhyGenBench and VBench-2.0, substantially improving OmniWeaving, with particularly large gains in thermal and material dynamics.Source: §1 · p. 3

2.1 Text-to-Video Diffusion Models

• Text-to-video diffusion progressed from foundational coherent synthesis [8] [9] through latent tuning [10] [11] [13] to large backbones [14] [15] [16] [17] [18], yet plausible dynamics remain unresolved [26] [27] [28] [29] [30] [31] [32] [33] [34] [35].Video diffusion has also been explored as a general visual or world representation [19] [20] [21], alongside reconstruction alignment, 3D coupling, and scalable multi-agent, multi-view modeling [22] [23] [24].Source: §2.1 · pp. 3–4

2.2 Physics-Aware Video Generation

• VideoCoCo grounds prompt-specific dynamics in executable simulation code, producing deterministic drafts that guide a video editor toward realistic results.Explicit simulator-based methods [36] [37] [38] [39] [40] can generalize poorly beyond predefined phenomena, while learned physical priors [41] [42] [43] [44] [45] offer limited control over prompt-specific processes; auto-rubrics provide another generative reward signal [46].Source: §2.2 · p. 4

2.3 Chain-of-Thought for Visual Generation

• VideoCoCo extends executable Code-as-CoT from image previews to deterministic temporal drafts that give video editors dense guidance over target dynamics.Video CoT approaches externalize plans or keyframes [1] [2], embed intermediate states [3] [4], or search and revise trajectories [5] [6]. DraCo uses drafts for text-to-image generation [48], CoCo represents intermediates as executable code [49], and MetaPoint and VideoCoF add structured spatial control and temporal reasoning [50] [51].Source: §2.3 · p. 4

3.1 Executable Simulation Engine

• Blender programs explicitly specify scene objects, physical properties, and temporal evolution; sandbox execution yields a deterministic, temporally dense draft that fixes what happens and when.The code is executable and inspectable, unlike textual plans or sparse keyframes; the draft is intentionally low-fidelity, leaving appearance to the next engine.Source: §3.1 · pp. 6–7

3.2 Generative Video Engine

• The generative video engine uses the draft to preserve process dynamics and an instruction agent’s appearance-focused guidance to specify subjects, materials, lighting, and style.The instruction agent reads both the prompt and draft, and is discouraged from redefining the draft’s motion; the editor restyles the instantiated process rather than imagining it from scratch.Source: §3.2 · p. 7

3.3 VideoCoCo-3K: Data for Draft-Conditioned Editing

• VideoCoCo-3K aligns simulated drafts with editing instructions and photorealistic targets, addressing the lack of training data pairing white-clay drafts with matching processes.The triplets are generated using the pipeline’s simulation and instruction agents and a teacher editor; evaluation prompts and near-duplicates are excluded, while prompts and Blender programs are retained as metadata.Source: §3.3 · p. 8

3.4 Editor Adaptation and Inference

• The editor is trained on VideoCoCo-3K with drafts and instructions as conditions, then inference automatically converts a prompt into a Blender draft, editing instruction, and final video.The editor starts from OmniWeaving and uses a conditional denoising objective; full fine-tuning and LoRA are considered, and inference requires no ground-truth draft, human annotation, or auxiliary input.Source: §3.4 · p. 8

4.1 Experimental Setup

• Evaluation uses PhyGenBench’s four physical categories and VBench-2.0’s mechanics, thermotics, and material dimensions, with higher scores better on every metric.PhyGenBench follows its official protocol with GPT-4o as the judge; VBench-2.0 uses a pre-registered subset balancing temporal and perceptual/semantic factors. VideoCoCo renders deterministic Blender drafts in a sandbox and refines them with an editor trained on VideoCoCo-3K.Source: §4.1 · p. 10

4.2 Code-as-CoT Improves Physical Consistency

• 0.558 average PhyGenBench consistency with VideoCoCo exceeds OmniWeaving’s 0.475 and Wan2.2-TI2V-5B’s 0.544, the strongest open baseline.VideoCoCo leads on mechanics (0.558), optics (0.613), and material (0.525), and ranks second on thermal (0.511), where Wan2.2-TI2V-5B scores 0.533. Its largest gains over OmniWeaving are material (+0.133) and thermal (+0.078).Source: §4.2 · pp. 10–11
• 77.88% average VBench-2.0 physical plausibility with VideoCoCo improves on OmniWeaving’s 52.18% by 25.70 points, the best average among all systems.VideoCoCo scores best on mechanics (92.31%) and thermotics (72.92%), improving the base by 29.52 and 20.84 points, respectively; it ranks second on material (68.42%), behind CogVideoX-1.5 (83.19%).Source: §4.2 · p. 11

4.3 Both Drafting and Tuning Contribute

• 0.506 average PhyGenBench consistency with tuning-free editing rises from the base’s 0.475; LoRA reaches 0.558, exceeding full tuning’s 0.535.The tuning-free editor uses the unmodified OmniWeaving model, isolating the executable draft’s contribution; adapting the editor on VideoCoCo-3K adds a complementary gain. LoRA’s low-rank adaptation preserves visual priors while learning to restyle drafts and preserve motion; full tuning is more prone to overfitting the limited triplet data.Source: Table 3 · p. 10, §4.3 · p. 11
• Figure 3 shows VideoCoCo following executable drafts to realize intended dynamics photorealistically in dry-ice sublimation and vacuum-induced bottle-collapse prompts.OmniWeaving produces plausible frames in these examples but violates the intended process.Source: §4.3 · p. 11

• VideoCoCo uses executable code as a process-level chain of thought, decoupling spatiotemporal simulation from photorealistic editing to achieve state-of-the-art physical plausibility on PhyGenBench and VBench-2.0.Source: §5 · pp. 11–12
• The approach adds inference latency and is limited by Blender's expressiveness, making highly complex phenomena such as turbulent fluids challenging to synthesize zeroshot.Source: §5 · p. 12
• Future work will explore specialized physical engines such as Taichi and distill executable priors into end-to-end video models to eliminate inference-time simulation.Source: §5 · p. 12