PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeCap-Bench: Can Code Serve as a Good Video Caption?

Jiafu Tang, Yifan Xu, Eric Tang, Junjie Zhao, Shipeng Luo, Te Sun, Haoxuan Hu, Yiyan Ji, Xinlong Chen, Zihao Wang, Yang Shi, Yuanxing Zhang, Pengfei Wan, Jiaheng Liu

arXiv:2610.37341v1cs.CV

TL;DR

Fine-grained spatial structure and temporal dynamics can make natural-language video captions lengthy and potentially ambiguous. The paper explores executable code as a complementary caption representation and introduces CodeCap-Bench to evaluate MLLM code-captioning. Code captions perform better than natural-language captions in question answering and reconstruction, while models still struggle to preserve complex temporal dynamics.

  • Problem

    Natural language can require lengthy, potentially ambiguous descriptions to express fine-grained spatial structures and temporal dynamics.

    Source: §1 · p. 1

  • Method

    The paper explores executable code as a complementary video-caption representation and introduces CodeCap-Bench, a structured 2D benchmark with fine-grained annotations and multi-level fidelity evaluation.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Code captions better preserve visual information than natural-language captions in question answering and reconstruction; leading proprietary MLLMs show promise but struggle with complex temporal dynamics, while open-source models lag substantially behind.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    Executable code is a promising complementary representation for video captioning, with an explicit visual interpretation that can be checked through rendering.

    Source: §1 · p. 2

    The evaluation focuses on structured 2D videos and a pure-code setting in which visual content is generated programmatically.

    Source: §1 · p. 2, §3.1 · p. 3

Abstract

from arXiv · show

Video captioning aims to preserve visual information through compact descriptions, predominantly using natural language. However, precisely expressing fine-grained spatial structures and temporal dynamics in natural language often leads to lengthy and potentially ambiguous descriptions. In this work, we explore Code as Video Caption, a paradigm where executable programs serve as a complementary representation of visual content through graphical primitives, layout rules, and functions of time. To examine whether code captions effectively preserve video information, we compare them with natural language captions on downstream question answering and video reconstruction tasks. Our results show that code captions achieve superior reasoning accuracy and reconstruction fidelity. Despite these advantages, realizing the potential of code captioning depends on whether current multimodal large language models (MLLMs) can generate faithful code captions. To systematically assess this capability, we introduce CodeCap-Bench, which comprises 323 videos spanning five categories of structured 2D content, along with fine-grained annotations and a multi-level evaluation framework. Extensive experiments reveal that while leading proprietary MLLMs demonstrate promising code captioning capabilities, they still struggle to faithfully capture complex temporal dynamics, with open-source models lagging substantially behind.

1 Introduction

Natural-language captions can become lengthy and ambiguous when they express fine-grained spatial structures and temporal dynamics. The paper explores executable code as a complementary representation and introduces a benchmark to assess whether models can generate faithful code captions.

1 Introduction

  • Code captions make spatial layouts, temporal evolution, and visual state changes explicit through graphical rules and executable functions.The flight seat-selection example uses grid rules for seat positions, time functions for scrolling, and conditional logic for color changes and selection marks.Source: §1 · p. 1
  • Code captions support more accurate question answering and better video reconstruction than natural-language captions under the evaluated settings.Reconstruction comparisons assess preservation of appearance, motion patterns, and spatial structure.Source: §1 · p. 1
  • CodeCap-Bench covers 323 structured 2D videos across five categories, with fine-grained focal-point and object-level spatio-temporal annotations.Its evaluation framework measures global visual similarity, object-level spatio-temporal alignment, and fine-grained visual information retention.Source: §1 · pp. 2–3
  • Leading proprietary MLLMs show promise but struggle with complex temporal dynamics, while open-source models lag substantially behind.The paper also examines relationships among caption fidelity, generation reliability, and cost.Source: §1 · p. 3

2 Related Work

Prior work examines fine-grained video captioning, caption evaluation, and code-based reconstruction in particular domains. CodeCap-Bench evaluates executable video captions across diverse structured 2D content using rendered-video fidelity measures.

2 Related Work

  • Fine-grained captioning and evaluation work spans detailed event and motion descriptions [39], object- and camera-centered captions [3], and spatial-temporal captioning and retrieval.VDCscore [3], VidCapBench [6], and CapQuiz [44] assess natural-language captions through question-based approaches.Source: §2 · p. 3
  • Code-based reconstruction studies cover static images, animations, indoor videos, and interactive webpages, while CodeCap-Bench targets diverse structured 2D videos.Its evaluation emphasizes fine-grained visual information retention and spatio-temporal fidelity.Source: §2 · p. 3

3 CodeCap-Bench

CodeCap-Bench evaluates executable programs that generate video content and its temporal evolution, comparing the rendered output with the reference. Its structured 2D dataset pairs fine-grained annotations with complementary fidelity measures.

3.1 Task Formulation

  • The task generates a programmatic caption for a reference video and evaluates the rendered video against it for visual appearance, spatial structure, and temporal dynamics.The paper focuses on a pure-code setting without reusing reference frames or external visual assets.Source: §3.1 · p. 3

3.2 Evaluation Framework

  • Fidelity is measured through global visual similarity, object-level spatio-temporal IoU, and fine-grained retention of annotated static and dynamic content.The measures respectively assess overall visual consistency, key-object alignment, and preservation of focal-point content.Source: §3.2 · pp. 3–4

3.3 Data Construction

  • The benchmark collects and filters structured 2D videos, then provides human-reviewed focal-point and object-level spatio-temporal annotations.Filtering retains 2D layouts, shapes, text, and explicit trajectories while excluding content dominated by complex 3D viewpoints, detailed textures, or dense pixel-level dynamics.Source: §3.3 · pp. 4–5, Figure 2 · p. 4

3.4 Dataset Statistics

  • The dataset contains 323 videos averaging 19.0 seconds, with 16.8 annotated objects and 8.6 static and dynamic focal points per video.The mean focal-point interval is 3.3 seconds; CodeCap-Bench spans longer sequences than Animation2Code [19], whose average duration is 5.1 seconds.Source: §3.4 · p. 5, Figure 3 · p. 5

4 Experiments

Across CodeCap-Bench experiments, GPT-6-astra with Codex achieves the highest reported overall retention and object-level alignment. Results also show that dynamics are harder to preserve than static content and that agentic workflows do not uniformly improve fidelity.

4. Experiments

  • Experiments assess code-caption fidelity, agentic workflows, reasoning effort, reliability, and computational cost across seven MLLMs and multiple generation settings.Agentic models can inspect reference frames, generate code, and refine it using rendering feedback; non-agentic models generate code in one pass from sampled frames and the prompt.Source: §4 · p. 6, §4.1 · p. 6

4.2 Main Results

  • GPT-6-astra with Codex leads overall retention at 73.24 and object-level spatio-temporal IoU at 75.37, beating Opus-5 with Pi Agent by 14.40 retention points.Table 1 reports configurations at medium thinking effort.Source: §4.2 · p. 6
  • Across evaluated configurations, dynamic retention is lower than static retention: GPT-6-astra with Codex scores 80.72 static versus 68.61 dynamic, and GPT-6-sol scores 64.04 versus 43.92.This pattern holds across models and generation settings.Source: §4.2 · p. 7
  • Agentic generation improves video production success rates but not always visual fidelity; Qwen3.8-max with Pi Agent improves both, while GPT-6-sol and GPT-5.6-sol lose overall retention with tested harnesses.For Qwen3.8-max with Pi Agent, success rises from 82.0% to 93.2% and overall retention from 44.67 to 48.83.Source: §4.2 · p. 7
  • Across four models, overall retention is higher for desktop and application interfaces and data and scientific visualization than for visual design and media animation or icons and abstract motion graphics.GPT-6-astra leads in every category on object-level spatio-temporal IoU and overall retention.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 Further Analysis

The analysis finds that code captions outperform natural-language captions in question answering and video reconstruction, while examining how model effort, cost, tool use, and evaluation relate to caption fidelity. The experiments also show that gains and operational trade-offs vary across models.

4.3.1 Effect of Thinking Effort

  • GPT-6-sol gains fidelity at higher thinking effort, while Qwen3.8-max sees most gains from low to medium effort and smaller gains thereafter.For GPT-6-sol, medium-to-xhigh effort raises spatio-temporal IoU from 50.42 to 67.13 and retention from 47.53 to 63.29, while cost rises from USD 0.315 to USD 1.409. Qwen3.8-max's xhigh gains are 2.67 and 2.72 points for the two fidelity measures, with cost increasing from USD 0.922 to USD 1.129.Source: §4.3.1 · p. 7, Figure 5 · p. 8

4.3.2 Generation Cost and Efficiency

  • Higher monetary cost does not consistently mean higher caption fidelity, and lower cost does not necessarily mean greater runtime efficiency.GPT-6-astra scores 72.87 in visual information retention at lower estimated cost than Opus-5, which scores 58.84; Qwen3.8-max and DeepSeek-v4.1-flash take 42.1 and 50.1 minutes per video, versus 12.5 minutes for GPT-6-astra.Source: §4.3.2 · pp. 7–8, Figure 6 · p. 8

4.3.3 Tool Usage Behavior

  • Models distribute tool calls differently, and more calls do not necessarily correspond to higher caption fidelity.GPT-6-astra scores 72.87 retention with 47.4 calls per video, while Gemini-3.8-flash scores 47.73 with 99.6 calls.Source: §4.3.3 · pp. 8–9, Figure 7 · p. 9

4.3.4 Validity of the Evaluation Metric

  • Two judge models rank GPT-6-sol above DeepSeek-v4.1-flash, and GPT-6-sol judge scores correlate with human ratings at Pearson r=0.942.The human-alignment assessment samples 50 videos, with 10 from each category.Source: §4.3.4 · p. 9, Figure 8 · p. 9

4.3.5 Execution Trace Example

  • In the bubble-chart example, GPT-6-astra uses visual analysis and render comparisons to detect and correct a one-year slider offset.The model estimates bubble positions and radii, detects year transitions, and tracks the pointer before generating the animation.Source: §4.3.5 · p. 10, Figure 9 · p. 10

4.3.6 Comparing Code and Natural Language Captions

  • Code-rendered videos outperform natural-language captions in question answering and reconstruction fidelity across the evaluated caption models.Question-answering gains range from 2.83 to 10.38 percentage points; Opus-5 reaches 83.02% accuracy versus 83.96% from original videos. Reconstruction improves across all three metrics, with Appearance Similarity increasing by 0.20–0.23 and Keyframe Structural Similarity by 0.18–0.22.Source: §4.3.6 · pp. 10–11, Table 2 · p. 10, Table 3 · p. 11

5 Conclusion

The paper presents executable code as a complementary video-caption representation and introduces CodeCap-Bench to evaluate MLLM code captioning. Code captions better preserve visual information than natural-language captions in question answering and reconstruction, while models still struggle with complex temporal dynamics.

5 Conclusion

  • CodeCap-Bench evaluates code captioning for structured 2D videos using diverse samples, fine-grained annotations, and a multi-level fidelity framework.The experiments find that leading proprietary MLLMs show promise but struggle with complex temporal dynamics, while open-source models have substantial performance gaps.Source: §5 · p. 11
  • Executable code better preserves visual information than natural-language captions in question answering and video reconstruction.Agentic workflows improve rendering success rates but do not consistently enhance caption fidelity.Source: §5 · p. 11

A Evaluation Environment and Runtime Configuration

The experiments run tasks in isolated containers and render generated web pages into videos.

A Evaluation Environment and Runtime Configuration

B Analysis of Video Generation Failures

Generation timeouts, incomplete HTML delivery, and rendering timeouts account for most failed video-generation tasks. Failure burden and leading causes differ across models.

B Analysis of Video Generation Failures

  • 83.6% of failed tasks stem from generation timeouts, incomplete HTML delivery, or rendering timeouts.These causes span code generation, page delivery, and rendering.Source: B Analysis of Video Generation Failures · p. 16, Figure 10 · p. 16
  • Qwen3.8-max and DeepSeek-v4.1-flash account for 73.1% of failed tasks, but their leading failure causes differ.HTML delivery problems account for 45.5% of Qwen3.8-max failures, while generation timeouts account for 55.6% of DeepSeek-v4.1-flash failures; GPT models have failure rates below 1%.Source: B Analysis of Video Generation Failures · p. 16

C Prompt Details

The appendix specifies prompts for code-only video recreation, requiring timeline-aligned visuals and a single-file deliverable, and a scoring rubric for comparing generated clips with reference content.

C Prompt Details

  • The prompts cover code-only visual implementation, timeline alignment, a single-file deliverable, and focal-point scoring against the reference video.The scoring prompt requests an integer score from 0 to 5.Source: C Prompt Details · p. 16, §C.3 · p. 18

C.1 Agentic Code Generation Prompt

  • The agentic prompt requires rebuilding the entire video in HTML, CSS, and JavaScript, preserving composition and continuous motion while prohibiting reference imagery in the output.It also directs frame inspection and iterative comparison of browser screenshots with reference frames.Source: §C.1 · p. 17
  • The agentic deliverable must be a self-contained index.html that plays automatically, preserves aspect ratio, and exposes a seekable videoCompare interface.The interface must report the reference duration and dimensions and reliably render requested timestamps.Source: §C.1 · p. 17

C.2 Non-Agentic Code Generation Prompt

  • The non-agentic prompt uses supplied frames, timestamps, duration, and dimensions to align the recreation with the reference video’s timeline.When frames appear in collages, their layout and timestamp labels identify their positions in the timeline.Source: §C.2 · p. 17
  • The non-agentic prompt requires a complete HTML document using code-generated visuals, forbids reference-image assets, and specifies an isolated, single-file deliverable with a seek interface.Its required output is only the HTML document, without Markdown or explanation.Source: §C.2 · p. 18

C.3 Video Scoring Prompt

  • The scoring prompt compares the complete generated video with reference content, using surrounding states to locate repeated content and checking static layout as well as dynamic progression.Dynamic checks include intermediate changes, order, pacing, duration, coordination, and results; static checks include position, proportions, relationships, and hierarchy.Source: §C.3 · p. 18
  • Scores 5, 4, and 3 distinguish exact matches, very slight local deviations, and clear non-core deviations; scores 2 and 1 separate a complete substantive part from rough or scattered correspondence.The rubric also directs comparisons to focus on visible frames and the importance of what a difference changes, not its screen area.Source: §C.3 · p. 18
  • The rubric assigns a 0–5 score based on reproduction fidelity, with timing and duration differences affecting the rating and the highest matching score selected.A clear change to stage timing, pacing, or synchronization limits the score to 2.Source: §C.3 · p. 18

D Representative Case Comparisons

The representative comparisons show one video from each of five dataset categories, with reference and model outputs arranged at shared timestamps and crop regions. The figures provide category-specific examples of content and changes across sampled times.

D Representative Case Comparisons

  • One case from each of five dataset categories compares reference videos with outputs from four models using Pi Agent, showing four chronological timestamps and identical timestamps and crop regions across rows.Source: D Representative Case Comparisons · p. 19
  • The workspace, game, and character-animation examples sample interface changes, a launch and collapse, and character or scene transitions at four times each.Their sampled times are 3.0, 5.0, 11.5, and 16.0 seconds for workspace; 3.1, 5.0, 8.0, and 13.4 seconds for the game; and 7.7, 9.6, 13.3, and 15.6 seconds for animation.Source: Figure 11 · p. 19, Figure 12 · p. 20, Figure 13 · p. 20
  • The scientific-visualization and icon examples sample energy-conversion diagrams and graphical transitions, then varied icon shapes, colors, and arrangements.Their four sampled times are 0.5, 4.3, 26.5, and 27.3 seconds for scientific visualization, and 3.0, 15.3, 17.5, and 23.0 seconds for icon collections.Source: Figure 14 · p. 21, Figure 15 · p. 21

Related papers

Loading 2610.37341v1…