Source-linked AI summary
Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V
Zhenghan Chen, Zekai Shao, Xin Lin, Leting Cheng, Siming Chen
TL;DR
Existing code-based benchmarks provide limited coverage of complete video creation across creative categories and input modalities. Code2Video-Compass brings together T2V, I2V, and V2V tasks and dual-level evaluation; evaluations reveal persistent challenges and declining performance as task complexity increases.
Problem
Existing code-based benchmarks focus on specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated.
Source: §2.2 · p. 2, §1 · p. 2Method
Code2Video-Compass combines 2,104 T2V, I2V, and V2V tasks across 10 categories and 24 subtypes with 12 objective and five subjective evaluation metrics.
Source: §1 · p. 2Results
Takeaways & Limitations
Code2Video-Compass serves as a diagnostic testbed for advancing reliable, coherent, and high-quality code-based video generation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.
1. Introduction
Code-based video generation offers fine-grained control but requires models to coordinate coherent narratives, lengthy interdependent code, animations, and evolving scene states. Code2Video-Compass addresses limited evaluation of complete code-generated videos across creative categories and input modalities.
Motivation
- Executable code explicitly represents visual structures and animation parameters, enabling fine-grained control, targeted editing, and component reuse.Generating complete videos also requires coherent narrative planning, lengthy code across interdependent files, and coordination of animations and evolving states.Source: §1 · p. 2
Research gap
- Existing code-based benchmarks focus on specialized applications and lack broad task coverage, input modalities, and combined objective and subjective evaluation.Prior examples include educational videos, data narratives, SVG animations, web animation reconstruction, and short motion-graphics clips.Source: §1 · p. 2
Benchmark and evaluation
- Code2Video-Compass evaluates complete code-based video generation across T2V, I2V, and V2V, using textual instructions and optional visual references.The benchmark combines 2,104 tasks across 10 categories and 24 subtypes with 12 objective and five subjective metrics.Source: §1 · p. 2
Benchmark findings
- Evaluation of 15 MLLMs finds that video quality and execution reliability do not always align, while performance declines as task complexity increases.The benchmark assesses construction, dynamic-animation, and state-dependency complexity.Source: §1 · p. 2
2. Related Work
Prior work spans broad evaluation of direct video generation and specialized code-based animation tasks. Comprehensive evaluation of complete code-generated videos across creative categories and input conditions remains limited.
2.1. Direct Video Generation
- Direct-video benchmarks assess perceptual quality, compositional and temporal constraints, conditioning, narrative progression, long-range consistency, or combined objective and subjective evaluation.The cited benchmarks include VBench, EvalCrafter, T2V-CompBench, TC-Bench, VBench++, KeyFrame-Compass, StoryBench, MSVBench, LongAV-Compass, and T2AV-Compass.Source: §2.1 · p. 2
2.2. Code-Based Video and Animation Generation
- Code-based generation research covers specialized applications, while its evaluation remains limited for complete videos across diverse creative categories and input conditions.Prior systems address educational videos, data storytelling, logo motion, layered HTML animations, and structured vector animations; evaluation includes aesthetics, execution, appearance, motion, and subjective review.Source: §2.2 · p. 2
- Table 1 compares representative benchmarks by source, generation paradigm, input settings, scale, and evaluation.Its caption defines source types, text/image/video input markers, average duration, dual evaluation, token counts, and event counts.Source: Table 1 · p. 3, §2.2 · p. 3
3. Code2Video-Compass
Code2Video-Compass evaluates complete code-generated videos across text, image, and video inputs, with a taxonomy-driven dataset and complementary objective and subjective metrics. Its construction links independent creative prompts and tasks derived from complete source videos to detailed instructions, annotations, and evaluation.
3.1. Task Formulation
- The benchmark defines T2V, I2V, and V2V as complete-video tasks requiring executable projects to realize instructions and reconstruct visual references through code.For I2V and V2V, models also determine where the reference content belongs in the complete video.Source: §3.1 · p. 3
3.2. Data Categories and Complexity
- The taxonomy covers 10 creative categories and 24 subtypes, with construction, dynamic-animation, and state-dependency complexity each defined at three levels.The dimensions distinguish visual-construction demands, temporal organization, and how long objects, data, and states must remain consistent.Source: §3.2 · p. 3
3.3. Data Construction
- The 2,104 tasks combine independently curated text and visual-reference tasks with aligned T2V, I2V, and V2V tasks derived from complete professional and user-created videos.Instructions specify visual elements, relationships, text and data, motion, event sequences, and cross-stage dependencies; tasks receive subject, event, motion, complexity, and QA-checklist annotations.Source: §3.3 · pp. 3–4
3.4. Dataset Statistics
- The dataset contains 1,719 paired tasks from 573 source videos and 385 independent tasks, totaling 2,104 tasks across T2V (718), I2V (693), and V2V (693).Coverage analysis counts each paired source once, producing 958 records: 573 paired sources and 385 independent tasks.Source: §3.4 · p. 4
3.5. Dual-Level Evaluation Framework
- The dual-level framework combines 12 objective metrics for operational performance, video quality, alignment, and reference fidelity with five subjective metrics for instruction following and holistic video quality.The subjective evaluations use an MLLM judge and five-point Likert scales for content coherence, visual design, motion quality, and viewer engagement.Source: §3.5 · pp. 4–5, §3.5.2 · p. 6
3.5.1. Objective Evaluation
- Objective measures cover execution success and token use, technical and aesthetic quality, motion smoothness, local and global text alignment, and reference fidelity.For V2V, sequence alignment uses subsequence DTW on frame embeddings, while motion fidelity compares normalized point trajectories using bidirectional best matching.Source: §3.5.1 · pp. 5–6
3.5.2. Subjective Evaluation
- The subjective framework checks instruction requirements with verifiable QA questions and rates holistic video quality across four aspects.The checklist covers content, attributes, composition, relations, dynamics, aesthetics, and task-specific requirements; holistic quality covers coherence, design, motion, and engagement.Source: §3.5.2 · p. 6
4. Experiments
The experiments evaluate 15 MLLMs on a fixed subset of 1,054 tasks across T2V, I2V, and V2V, with code generated in a single call and no execution feedback or repair. Claude Opus 5.5 achieves the best average rank in each setting, while execution success and metric-specific strengths vary across models.
4.1. Experimental Setup
- The fixed evaluation subset contains 1,054 tasks, and each model generates code in one API call without execution feedback or repair.The subset includes 360 T2V, 347 I2V, and 347 V2V tasks sampled from both task-construction strategies.Source: §4.1 · p. 6
4.2. Main Results
- Claude Opus 5.5 achieves the best average rank in T2V (1.60), I2V (1.08), and V2V (1.15), with proprietary models occupying the top three positions in every setting.Among open-weight models, DeepSeek-V4.1-Flash leads T2V, while MiMo-V2.6-Pro leads I2V and V2V.Source: §4.2 · p. 6, Table 2 · p. 7, Table 3 · p. 7
- Execution success ranges from 0.52 to 0.94 on T2V, 0.59 to 0.96 on I2V, and 0.69 to 0.97 on V2V.Source: §4.2 · p. 7
- In T2V, Claude Opus 5.5 leads IF and CC at 0.91 and 4.05, while Claude Sonnet 5.5 leads MQ and VE at 3.85 and 3.90.GPT-6.1 Sol leads VD at 3.88.Source: §4.2 · p. 7, Table 2 · p. 7
- In I2V, Claude Opus 5.5 leads both image-fidelity metrics, PF at 0.89 and SF at 0.66, and all five subjective metrics.GPT-6 Astra and GPT-6 Luna have similar PF and SF scores but differ in IF and MQ, so faithful keyframe reconstruction does not necessarily ensure strong instruction following or motion quality.Source: §4.2 · p. 7, Table 3 · p. 7
- In V2V, Claude Opus 5.5 leads clip-fidelity metrics SA (0.88), AF (0.79), and MF (0.44), as well as all five subjective metrics.Claude Sonnet 5.5 has comparable clip fidelity but lower IF and CC than Opus, highlighting a gap between reference-clip reconstruction and complete-video requirements.Source: §4.2 · pp. 7–8, Table 4 · p. 8
5. Insights and Discussions
The analyses show that video quality must be considered alongside execution reliability, that failure profiles and complexity effects differ, and that greater generation effort does not ensure better quality. The Gemini 3.8 Flash judge aligns with human ratings and shows strong repeat stability.
5.1. Quality and Reliability Analysis
- GPT-6 Astra and GPT-6.1 Sol score 4.19 and 4.15 on successfully rendered videos, but their full-set means fall to 3.62 and 3.61; Claude Opus 5.5 has the highest full-set mean at 3.87.Higher quality on successful runs can coexist with lower execution reliability.Source: §5.1 · p. 8, Figure 5 · p. 8
5.2. Failure Analysis
- Failure profiles vary by model: compilation errors and missing APIs or packages make up 83.4% of GPT-6 Astra’s failures, while runtime exceptions account for 75.0% of Claude Opus 5.5’s.Across 2,756 error cases, failures ranged from compilation and dependency handling to runtime execution; lengthy code across interdependent files remains a challenge.Source: §5.2 · p. 8, Figure 6 · p. 8
5.3. Complexity Analysis
- Performance generally declines as construction, dynamic-animation, and state-dependency complexity increases, but the steepest level-to-level declines differ across dimensions.SF falls 14.38% of its score range from L1 to L2 under construction complexity and 12.82% under dynamic-animation complexity; state-dependency declines are larger from L2 to L3 across all 15 metrics.Source: §5.3 · p. 9, Figure 7 · p. 9
5.4. Generation Effort and Cost
- Greater generation effort does not necessarily yield better video quality: GLM-5.3-Flash uses 90.0k output tokens and scores 2.94, versus GPT-6 Astra’s 13.9k tokens and score of 3.62.GPT-6 Astra incurs approximately 6.1 times GPT-6.1 Sol’s estimated API cost for nearly identical quality scores, while Claude Opus 5.5 scores 3.87 at lower estimated cost than Astra.Source: §5.4 · p. 9, Figure 8 · p. 9
5.5. MLLM Judge Validation
- On 200 outputs rated by three human raters, the Gemini 3.8 Flash judge reaches Pearson’s r of 0.796 for IF and 0.783 for holistic video quality, with ICC(A,3) of 0.983 and 0.970.The validation drew from 300 outputs across six models, with each output evaluated three times by the judge.Source: §5.5 · p. 9, Table 5 · p. 9
6. Conclusion
Code2Video-Compass benchmarks complete code-based video generation across T2V, I2V, and V2V with 2,104 tasks and a dual-level evaluation framework. Evaluation of 15 MLLMs reveals persistent challenges in reliability, alignment, reference fidelity, and increasing task complexity.
6. Conclusion
- Code2Video-Compass provides a general-purpose benchmark for complete code-based video generation across T2V, I2V, and V2V, with 2,104 tasks and a dual-level evaluation framework.Source: §6 · p. 9
- Evaluation of 15 MLLMs reveals persistent challenges in execution reliability, instruction following, reference fidelity, and handling increasing task complexity.Source: §6 · p. 9
- The benchmark serves as a diagnostic testbed for advancing reliable, coherent, and high-quality code-based video generation.Source: §6 · p. 9