PaperlayerView exact revision on arXiv

Source-linked AI summary

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma, Yuzhe Chen, Jiankang Chen, Kaiyu Jiang, Changyi Liu, Haoyu Zhang, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Bingyi Li, Xu Wang, Xuanyu Zheng, Bin Wen, Tingting Gao, Han Li, Wentao Zhang, Weiran Xu, Kongming Liang, Zhanyu Ma

arXiv:2610.35864v1cs.CV

TL;DR

Existing video benchmarks do not establish whether general agents can make and inspect complete audio-visual videos from code. VibeVideoBench evaluates this task across 239 prompts with hierarchical checks and judge panels; even leading systems still miss core requirements, and missing videos affect the rankings.

  • Problem

    Existing evaluations cover short silent generated clips, static artifacts, or specialized video pipelines, leaving general agents’ ability to check perceptual, temporal, and audio-visual outputs untested.

    Source: §1 · p. 2

  • Method

    VibeVideoBench evaluates agents making complete code-driven MP4s on 239 tasks using program checks, requirement-traced rubrics, quality scores, and human-validated judge panels.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Even the leading systems fail a core requirement on about a quarter of tasks, and counting missing videos as failures changes system rankings.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    Agents should secure a complete video early and verify the delivered file, since many lost videos result from late or slow final renders or renders never started.

    Source: §6 · p. 9

    Each system-task outcome is based on one run with a fixed one-hour, single-attempt budget, which may not yield an agent’s best video.

    Source: J. Limitations and Broader Impact · p. 29

Abstract

from arXiv · show

Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.

1. Introduction

Existing video and agent benchmarks do not test whether general agents can create and inspect complete code-driven videos across picture, sound, and time. VibeVideoBench evaluates that task end to end, and its results show that even leading systems often miss core requirements or fail to deliver a video.

1. Introduction

  • VibeVideoBench tests general multimodal agents on complete code-driven videos, addressing a gap left by short, silent clip benchmarks, static-artifact tests, and single-domain video pipelines.A video is a perceptual, temporal, audio-visual outcome for which existing agent benchmarks provide no unit test.Source: §1 · p. 2
  • The benchmark contains 239 English- and Chinese-language tasks across 12 categories, requiring finished MP4s made entirely with code in a one-hour sandbox.Tasks range from brand promos and music-driven pieces to classroom lessons and paper explainers.Source: §1 · p. 2
  • Tasks begin from real requests, videos, asset packs, lecture notes, or papers, then become self-contained specifications that fix facts, stages, timing, and audio policy while preserving creative freedom.Missing assets are searched for or synthesized, and humans curate the resulting tasks.Source: §1 · p. 2
  • Evaluation combines program checks, requirement-traced rubric checks, and five task-anchored quality dimensions, with three judge panels validated against blind human preferences.The rubric checks distinguish demonstrated processes from claimed results and are tagged by modality.Source: §1 · p. 2
  • Even the leading systems fail a core requirement on about a quarter of tasks, while three quarters of lost videos stem from final renders started too late, running too slowly, or never started.Judges align on system rankings with each other and with blind human preferences, but agree only moderately on individual videos.Source: §1 · p. 2

2. Related Work

Prior work evaluates generated pixels, static or interactive artifacts, specialized video pipelines, or agent outcomes that can be checked functionally. VibeVideoBench instead studies general agents that build complete audio-visual videos from code and evaluates their perceptual, temporal, and audio-visual results.

Video generation: the pixel paradigm

  • Pixel-generation benchmarks assess plausibility of short clips, while VibeVideoBench studies agents that program renderers, TTS, and ffmpeg to create editable videos with controlled content, length, and sound.The authors argue that evaluation of mostly silent footage does not transfer to this program-driven setting.Source: §2 · p. 3

Code as a visual medium: the program paradigm

  • Existing code-to-video systems focus on educational or paper videos, fixed renderers or genres, or single model calls rather than agents that choose tools and revise their own renders.TheoremExplainAgent [27], Code2Video [10], VisualEDU, and ManimAgent [23] are examples of specialized educational systems.Source: §2 · p. 3

Agent benchmarks

  • General agent benchmarks verify text, files, or functional behavior, whereas a video is a perceptual, temporal, audio-visual outcome without a unit test; AgenticVBench [6] instead evaluates post-production of supplied footage.Here every frame and sound must be constructed by code.Source: §2 · p. 3
  • Among the benchmarks compared, only VibeVideoBench has general-purpose agents build complete audio-visual videos from scratch with code across open domains and task-specific rubrics.The table uses ✓, ✗, and ◦ for yes, no, and partial coverage.Source: Table 1 · p. 3

3. VibeVideoBench

VibeVideoBench specifies code-built video tasks, constructs them from real-world sources, and equips each with evidence-traced rubrics. Its 239 tasks cover varied categories, languages, assets, and audio requirements.

3.1. Task formulation

  • Each task specifies a natural-language prompt, optional read-only assets, machine-readable delivery requirements, a four-channel audio policy, and the intended creative freedom.The agent must deliver a video, editable source project, and timing sheet; no ground-truth video is provided, and image- or video-generation models are disallowed.Source: §3.1 · p. 3

3.2. Task construction

  • Tasks are seeded from real requests, videos, asset packs, lecture notes, or papers, then rewritten under a shared contract and human-curated for checkability.The contract fixes relevant facts, identities, stages, dwell times, and audio policy while preserving creative freedom; missing assets are searched for or synthesized.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3. Statistics

  • The benchmark has 239 tasks in 12 categories and 43 subcategories, including 209 English and 30 Chinese tasks; 70% require audio and 49% require speech.Tasks span 8 seconds to 15 minutes, and one pass asks for 11.1 hours of finished video.Source: §3.3 · p. 4

3.4. Rubrics

  • Each task has program-level L1 checks, atomic requirement-quoted L2 checks, and five task-anchored L3 quality dimensions, with modality tags and iterative model and human review.The final rubric set contains 1,449 L1 checks, 12,985 L2 checks, and 1,195 L3 rubrics.Source: §3.4 · p. 4

4. Evaluation Protocol

The evaluation separates delivery compliance, task-specific requirement success, and perceived video quality, then validates judge-based assessments against blind human preferences. It reports delivery alongside delivered-video scores and also treats missing videos as zero.

4. Evaluation Protocol

  • Each system-task rollout runs once in a fresh sandbox with a common toolchain and one-hour budget; L1 checks are decided by programs alone.A delivery must be a decodable MP4 that can be judged.Source: §4 · p. 4
  • L1 checks verify delivery requirements such as resolution, frame rate, duration, source project, and required audio; non-deliveries fail every applicable check.Checks also cover narration and frozen or black stretches for lessons and papers.Source: §4 · p. 5
  • L2 strict marks a run successful only if no core check fails and at least one passes, while L3 rates five task-anchored quality dimensions from 1 to 5.Unknown and not-applicable checks do not count against L2 strict; overall L3 is the mean of the five dimensions.Source: §4 · p. 5
  • The primary display pairs delivery rate with delivered-only L2 and L3 scores, while a separate no-video-zero view combines reliability and quality.Table 2 also reports L2 strict and human Elo alongside its delivered-video measures.Source: §4 · p. 6, Table 2 · p. 6
  • Three independent judge panels assess each delivered video, with visual and audio checks routed to suitable models; blind pairwise human preferences validate system-level rankings.Human comparisons use two playable videos from the same task and measure delivered-video quality, not delivery.Source: §4 · p. 6

5. Experiments

Across nine agent systems and 2,151 rollouts, GPT-6 Astra and Claude Opus 5.5 lead, but the benchmark remains far from solved. Delivery failures, judge disagreement on individual videos, and weak cost–quality correspondence qualify the results.

5.1. Main results

  • Astra (4.13) and Opus (4.07) lead delivered-video overall scores, while blind human preferences place them first and second, respectively.The human ranking matches delivered-video rankings almost exactly (Spearman ρ=0.98).Source: §5.1 · p. 6
  • Strict success ranges from 26.6% to 76.3%; even the best system fails a core requirement on about a quarter of tasks.Counting missing videos as failures changes scores and ranks; Gemini 4 Argon falls from 4th to 7th when its 77% delivery rate is included.Source: §5.1 · p. 7

5.2. Why videos are not delivered

  • Three quarters of the 224 failures (169) involve final renders started too late, running too slowly, or never started.Opus finishes successful videos a median of 24 minutes before the deadline; a larger budget did not improve DeepSeek’s score materially (3.92 at 60 minutes versus 3.94 at 120).Source: §5.2 · p. 7

5.3. How agents make videos

  • Agents use distinct renderer stacks, and 95% of runs inspect rendered frames, but inspection volume does not predict delivered quality (ρ=−0.07, n=9).The pipeline is code-to-frames-to-ffmpeg in 96–100% of runs; renderer–system coupling means renderer comparisons are associations, not isolated effects.Source: §5.3 · pp. 7–8

5.4. Judges and humans

  • Judge panels rank systems similarly (Spearman 0.93–0.95), but agree with humans on individual pairs only 73.3% of the time.Agreement falls to 63% when panel score gaps are below 0.2; panels assign the same mean score (3.81) to pairs humans label both bad as to other pairs.Source: §5.4 · p. 8

5.5. Cost

  • Mean API cost spans 49× across systems, yet its rank correlation with overall score is not significant (ρ=0.55, p=0.12, n=9).DeepSeek-V4.1-Flash reaches 95% of Astra’s score at 5.8% of its cost; score per dollar ranges from 0.26 points per USD for Opus to 8.8 for GLM.Source: §5.5 · p. 8, Figure 5 · p. 8

6. Conclusion

VibeVideoBench evaluates code-driven video agents on realistic, checkable tasks with evidence-based checks and quality scores. Its construction and taxonomy define the task mix, while task selection and benchmark authoring introduce documented scope considerations.

6. Conclusion

  • VibeVideoBench comprises 239 tasks in 12 categories and evaluates complete audio-visual videos using program checks, task-specific rubric checks, and five quality dimensions judged by three panels.The panels were validated against blind human preferences.Source: §6 · p. 9

A.1. Source streams and funnel

  • Human selectors favored checkable tasks unevenly, keeping 88% of music and sound-driven drafts but only 17% of story and comedy drafts and 15% of game drafts.Two LLM screens graded every candidate highly, so they remain a weak proxy for human judgments of task value; pilot-task removal barely changed the leaderboard.Source: §A.1 · p. 13

A.2. Taxonomy

  • Each task has one of 12 categories and one or more of 43 cross-cutting subcategories, including presentation forms and knowledge domains.Tasks average 2.0 subcategories, and the per-task mapping is released with the benchmark.Source: §A.2 · p. 13

A.3. Task format and statistics

  • Task packages give agents a self-contained prompt, delivery requirements, optional inputs, audio policy, and creative freedom.Typical prompts also specify audience, structure, progression, and deliverables such as an editable project and timing sheet.Source: §A.3 · pp. 14–15

A.4. Example specifications (abridged)

The appendices illustrate task specifications and detail how rubrics, the sandbox, and judge panels operationalize benchmark evaluation. They describe traceable checks, task-specific quality anchors, controlled rollout conditions, and varying judge agreement and strictness.

A.4. Example specifications (abridged)

  • Example specifications demand task-specific event sequences, synchronized audio, or narrated explanations while leaving presentation choices open.Examples include a Mario level run, an architectural-process animation, a Lenz’s law lesson, and a paper explainer.Source: §A.4 · pp. 15–16

B.1. Structure of a rubric

  • Rubric checks identify goals, criticality, pass/fail/unknown conditions, required evidence, prompt quotations, allowed variations, and modality tags.The 239 tasks contain 12,985 checks; five quality dimensions use task-specific anchors at scores 1, 3, and 5.Source: §B.1 · p. 16, Table B.1 · p. 16

B.2. Authoring principles

  • Rubric authoring prioritizes traceability and evidence: optional prompt language stays optional, process requirements need visible evidence, and checks cannot prescribe production methods.Core and detail checks are fixed before outputs are seen, and insufficient evidence yields unknown rather than a default score.Source: §B.2 · p. 16

B.3. Rubrics for lessons and papers

  • Lessons and papers use checks for source-grounded content, correct narration, and synchronization at 4–6 key moments.Judges do not see the source notes or papers, so 330 checks were rewritten to state required content directly and six audit agents re-derived numbers and formulas.Source: §B.3 · pp. 16–17

B.4. Quality control

  • Rubrics underwent iterative independent review, fresh-context adjudication, and human verification until no confirmed findings remained.Over-specification was the most frequent defect, and review also uncovered and prompted fixes to task bugs.Source: §B.4 · p. 17

B.5. Modality tags

  • Checks are tagged as visual (69.4%), audio-visual (23.2%), or audio (7.4%), and those tags route checks in J3.Source: §B.5 · p. 17

C. Sandbox, Harnesses and Rollout Protocol

  • Agents run in identical fresh containers with an hour-long budget, native harnesses, pre-approved tools, and no follow-up turns.At the deadline the container is killed; collector checks cover deliverables and video technical specifications, while content is assessed separately.Source: C. Sandbox, Harnesses and Rollout… · p. 17, Table C.1 · p. 17

D.2. Agreement between panels

  • Judge panels show strong system-level ranking agreement but only moderate agreement on individual videos, and they differ in strictness.Per-video overall correlations range from r=0.44 to 0.50; panel strictness differences motivate comparing only the three-panel mean across systems.Source: §D.2 · p. 18, §D.3 · p. 18, Table D.1 · p. 18

D.4. Self-preference

Self-preference shifts some panel scores, but removing self-judging or restricting panels by family leaves the system ranking largely intact. The full leaderboard also shows that strict success is driven mainly by core-check failures, with non-delivery a substantial issue for some systems.

D.4. Self-preference

  • Two of three judges rate their own model about a sixth of a point higher than the other panels do, while panels also show comparable system-specific tastes.J2 rates Gemini 3.8 Flash +0.16 and J3 rates GPT-5.6 Sol +0.20; J1 shows no preference for Gemini 4 Argon.Source: §D.4 · p. 19
  • Removing each system’s own-model judge changes only the final two places (τ=0.94), while family-excluded, leniency-shifted scores preserve the order (τ=1).The Astra–Opus gap narrows from 0.034 to 0.018 in the family-excluded comparison.Source: §D.4 · p. 19

E.1. Full leaderboard

  • Strict success ranges from 73.6% for Opus and 72.1% for Astra to 23.8% for Gemini 3.8 Flash, with most failures among leading systems involving core checks.For Gemini 4 Argon, GLM and Gemini 3.8 Flash, 19–25% of runs fail strict success through non-delivery.Source: §E.1 · p. 19

E. Full Results

  • The full leaderboard combines delivery and specification compliance, rubric pass rates, five quality dimensions, human preference ratings and API cost across 239 tasks.It reports three-panel means and treats non-delivery as a failure for specification compliance.Source: §E.1 · p. 19

E.2. Per-panel leaderboards

The three panels largely agree on system ordering, although their strictness and some near-tie rankings differ. Delivery affects comparisons, while quality profiles, program checks and run behavior identify distinct constraints on performance.

E.2. Per-panel leaderboards

  • The top four systems are the same under every panel, while disagreements occur mainly among near-ties; panel strictness varies substantially.Opus strict success ranges from 51.9% to 93.3% across J1–J3.Source: §E.2 · pp. 19–20

E.3. Uncertainty

  • Astra leads Opus with P=0.91, and Gemini 4 Argon’s 23% non-delivery helps explain why it ranks below GPT-5.6 Sol despite more head-to-head wins.Astra has 127 head-to-head wins against Opus’s 89; Argon wins 142 of 239 comparisons against Sol.Source: §E.3 · p. 20

E.4. Quality profiles by dimension and category

  • Building & process is hardest at 3.07, while classroom lessons score highest at 4.19; Astra or Opus leads 10 of 12 categories.The systems retain their broad tiers across dimensions, with visual aesthetics highest for most and timing lowest for all but Gemini 4 Argon.Source: §E.4 · p. 20

E.5. Program checks (L1)

  • Among 1,628 playable core-task deliveries, duration is the most frequent program-check failure, occurring 34 times; container defects are rare in delivered lesson and paper videos.Rule-based checks fail in 14 of 306 lesson and paper runs, nearly all without a playable video.Source: §E.5 · p. 20

F.1. Outcomes and effort

  • Opus and Astra usually finish well before the deadline, while GPT-5.6 Sol and GLM often run into it; longer runs score lower within every system.The harnesses also shape tool use: Claude Code and Codex agents work almost entirely through the shell, while Gemini, Kimi and GLM rely more on dedicated file tools.Source: §F.1 · p. 21

F.2. Rendering stacks

Systems use different rendering stacks, and renderer-level outcomes are associated with those choices but cannot be separated from system identity. Self-inspection is not a reliable cross-system predictor of delivered quality, and temporary output files would not have prevented most truncated videos.

F.2. Rendering stacks

  • Vector and 2D-canvas pipelines deliver most often and score highest, while software-WebGL 3D scenes are slowest and score lowest; renderer and system are coupled.The figure’s quality profiles use five dimensions and per-category overall scores, with radial axes starting above zero.Source: §F.2 · p. 22, Figure E.1 · p. 22

F.3. Self-inspection

  • How much agents inspect rendered frames does not predict delivered quality across systems, and reported verification often fails to match specification compliance.Astra inspects least while scoring best; among videos that fail compliance, most of Astra’s reports and a minority of others’ still claim verification.Source: §F.3 · p. 22

F.4. Failure taxonomy

  • Temporary-file rendering would not have saved most truncated MP4s because they mainly result from stalled or unfinished renders, not overwriting a finished video.Source: §F.4 · p. 22

G. Cost

API spending varies widely across systems but does not track headline score significantly, and the cost–quality frontier runs from GLM through DeepSeek to Astra. Cost estimates also omit judging and undercount timed-out runs.

G. Cost

  • API cost spans 49×, from $0.32 for GLM to $15.38 for Opus, but its headline-score rank correlation is not significant (ρ=0.55, p=0.13, n=9).DeepSeek-V4.1-Flash reaches 95% of Astra’s score at 5.8% of its cost.Source: G. Cost · p. 23
  • The cost–quality Pareto frontier proceeds from GLM to DeepSeek to Astra; Astra uses the fewest output tokens at 61k per task, versus 77k–289k for the others.Source: G. Cost · p. 25
  • Timed-out runs are under-costed because accounting stops at one hour, some model prices are promotional, and judging cost is excluded.Source: G. Cost · p. 25

H. Human Preference Study

The human preference study validates system-level rankings of delivered videos, while the ablations probe judge reliability, teaching coverage, time budgets, and CPU budgets. The tests expose limits in detecting audio-visual timing and in judging whether videos are acceptable, while finding no measurable benefit from longer-than-one-hour budgets or more CPU cores in the tested settings.

H. Human Preference Study

  • Astra and Opus lead the human ranking and cannot be separated; human rankings closely match delivered-only judge rankings, but do not validate delivery-inclusive scores.Humans never see missing videos, which changes some system ranks when missing deliveries score zero.Source: H. Human Preference Study · p. 26, Table H.1 · p. 26, Table H.2 · p. 26
  • Human-labeled “both bad” pairs receive nearly the same mean judge score as decisive pairs, so judge scores do not certify acceptability.The means are 3.81 and 3.83, respectively; only 4% of both-bad pairs have both videos below 3.5.Source: H. Human Preference Study · p. 26

I.1. Judge stress test

  • Judges detect picture defects, foreign soundtracks, and swapped halves, but miss audio delays and often praise synchronization.Audio-visual timing is a blind spot of both judges and program checks; clean re-encoding also shifts scores by 0.1–0.2, within judge noise for single videos or close systems.Source: §I.1 · p. 27, Figure I.1 · p. 27

I.2. Do the videos teach? A grounded quiz

  • The grounded quiz ranking correlates with overall judge scores, but agrees only moderately on individual runs and separates top systems poorly.Its 404 questions across 34 lessons and papers primarily measure coverage rather than correctness.Source: §I.2 · p. 27, Figure I.2 · p. 28

I.3. Time budget

  • Quality saturates between 30 and 60 minutes for DeepSeek-V4.1-Flash, while 15-minute runs lose videos and score worse.Doubling the budget from 60 to 120 minutes adds 0.01 (p=0.78); this ablation covers one system with one run per cell.Source: §I.3 · p. 28, Figure I.3 · p. 28

I.4. CPU budget

  • More CPU cores do not measurably improve pooled scores or playable-video counts on the tested render-heavy tasks.The ablation covers six systems and 20 tasks; the 16-core arm ran later on a busier host.Source: §I.4 · pp. 28–29, Figure I.4 · p. 29

J. Limitations and Broader Impact

The evaluation is bounded by single-run, fixed-budget testing and a task set weighted toward explainers, English, and 1080p-class outputs. It primarily measures adherence rather than creativity, and its proprietary judges have a specific sensitivity gap.

Limitations

  • Single runs make individual task outcomes variable, so per-task scores should not be over-read.Confidence intervals cover task sampling only, although system-level means are stable.Source: J. Limitations and Broader Impact · p. 29
  • A one-hour, single-attempt budget may not reach an agent’s best video, and saturation within an hour was tested for only one system.The longest tasks may not share that saturation behavior.Source: J. Limitations and Broader Impact · p. 29
  • The rubrics emphasize specification adherence; creativity and originality enter only partly through aesthetics and are not assessed independently.Source: J. Limitations and Broader Impact · p. 29
  • Coverage leans toward explainers, is mostly English, and is limited to 1080p-class outputs, with few very long videos.The task set is 38% explainers and 209 of 239 tasks are English.Source: J. Limitations and Broader Impact · p. 29
  • All judge panels are proprietary models and are insensitive to audio-visual timing, despite stable rankings across three judge families.Absolute scores are panel-specific and are never mixed.Source: J. Limitations and Broader Impact · p. 29

Broader Impact

  • Factual tasks supply their facts and forbid fabricated claims, while the broader-impact discussion flags persuasive marketing and misinformation as risks.No task depicts real private individuals; classic-game tasks use newly designed levels and assets.Source: J. Limitations and Broader Impact · p. 29

Related papers

Loading 2610.35864v1…