PaperlayerView exact revision on arXiv

Source-linked AI summary

BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender

Yolo Y. Tang, Daiki Shimada, Jiayue Meng, Jing Bi, Pinxin Liu, Yicheng Wang, Yunzhong Xiao, Zhangyun Tan, Zeliang Zhang, Chao Huang, Susan Liang, Qianxiang Shen, Luchuan Song, Ali Vosoughi, Mingqian Feng, Melika Filvantorkaman, Chenliang Xu

arXiv:2609.15478v1cs.CV

TL;DR

Existing video benchmarks mainly use question answering, which does not establish that an agent tracked a scene’s full spatiotemporal structure. BVB tests this through controlled Blender reconstruction and evaluates semantic retention alongside perceptual similarity. Across 51 configurations, models achieve high visual similarity but retain only about half of source-correct spatiotemporal answers, making semantic retention the main challenge.

  • Problem

    Video question answering can be correct without demonstrating full spatiotemporal understanding of a scene.

  • Method

    BVB has multimodal agents reconstruct real indoor videos as editable animated Blender scenes through a shared Mini-BVB sandbox, scoring them with Dual VQA and Latent Similarity.

  • Results

    Across 51 configurations, the best models achieve high visual similarity but retain only 53.7% of source-correct spatiotemporal answers.

  • Takeaways & Limitations

    Programmatic reconstruction is a viable test of agentic video understanding, while semantic retention remains the main unresolved challenge.

  • Takeaways & Limitations

    BVB covers indoor egocentric videos, and performance also depends on coding ability and familiarity with Blender.

Abstract

from arXiv · show

Multimodal agents can create complex videos in software such as Blender by coding without relying on diffusion models. Yet video understanding benchmarks still evaluate models mainly through question answering. If an agent truly understands a video, it can reconstruct it programmatically. We introduce BVB, Blender-VideoBench, a benchmark that tests this ability by asking agents to reconstruct real-world videos as animated Blender scenes. To ensure fair comparison, each agent programs the reconstruction through a lightweight harness, Mini-BVB, in an identical sandbox under a shared cost limit. The benchmark renders each reconstruction from its animated camera and evaluates it on two axes: (1) Dual VQA measures how many spatiotemporal facts the reconstruction preserves. (2) Latent Similarity measures how closely the reconstruction matches the source video perceptually. Our overall score, a square-root mean, favors balanced performance. We evaluate 51 configurations from 10 model families and analyze semantic retention, perceptual similarity, reasoning effort, and cost. The best model reaches 88.6 Latent Similarity but retains only 53.7% of the source-correct spatiotemporal answers. Additional reasoning improves visual similarity but does not close this gap in factual accuracy. In a blind study with 15 raters and five configurations, Latent Similarity correlates strongly with human preference. These results show that programmatic reconstruction is a viable test of agentic video understanding, and that semantic retention remains the main challenge.

1 Introduction

BVB reframes video understanding as programmatic reconstruction, requiring agents to preserve scenes’ spatial and temporal structure rather than answer questions alone. It standardizes reconstruction and evaluates both factual retention and perceptual similarity across diverse model configurations.

  • Benchmark motivation and design: Programmatic reconstruction is motivated as a stronger understanding test because faithful scenes must preserve object placement, camera trajectory, and event ordering.These properties cannot be reliably inferred from a single frame or answer prior.
  • Benchmark motivation and design: BVB asks multimodal agents to reconstruct real-world indoor videos as editable animated Blender scenes using a standardized, cost-controlled, asset-free setup.Agents inspect frames and execute Blender code through Mini-BVB, constructing geometry from primitives and animating the camera along the source trajectory.
  • Evaluation: Dual VQA measures retained source-correct spatiotemporal facts, while Latent Similarity compares reconstructed and source videos’ layout and motion using frozen V-JEPA features.The two axes capture complementary aspects of reconstruction quality rather than relying on question answering alone.
  • Empirical findings: 51 configurations across 10 model families show visually plausible but semantically incomplete reconstructions, with the best model reaching 70.07 Overall but only 53.7% semantic retention.Visual similarity is substantially higher than factual retention, and additional reasoning improves visual similarity without closing the factual-accuracy gap.
  • Empirical findings: A blind study with 15 raters found that the automatic configuration ranking matched human rankings exactly.This supports the use of the benchmark’s automatic evaluation for comparing reconstruction quality.

2 BVB: Blender-VideoBench

BVB standardizes agent interaction and scores rendered reconstructions along semantic and perceptual axes. Its square-root-mean aggregate penalizes imbalance, while the evaluation design isolates retained facts, visual agreement, and failed runs.

  • Benchmark construction: Each agent uses the same two-action Mini-BVB harness to inspect frames or execute Blender code in a Docker sandbox under a per-scene spend cap.External asset libraries are disallowed, and valid runs must observe the source and produce a renderable Blender file.
  • Evaluation metrics: Dual VQA retains the fraction of questions answered correctly on both reconstruction and source among questions answered correctly on the source.Conditioning on the source-correct subset measures retained content rather than the judge’s baseline accuracy.
  • Evaluation metrics: Latent Similarity compares source and reconstructed clips with a frozen V-JEPA encoder across layout and motion, averaging both into a perceptual score.The encoder is not fine-tuned on BVB, and failed renderable files receive zero on both axes.
  • Aggregate score: The square-root mean combines Dual VQA and Latent Similarity to favor configurations that perform strongly and evenly across both axes.Unlike the arithmetic mean, it penalizes cross-axis dispersion; unlike the geometric mean, it does not become zero when one axis is zero.

3 Experiments

BVB evaluates zero-shot multimodal reconstruction agents under a shared harness, sandbox, and cost ceiling across 288 scenes. Results show strong perceptual similarity but substantial semantic loss, with rankings separated among leading configurations and human preferences aligned with overall performance.

  • 3.1 Experimental Setup: All agents use the same Mini-BVB Harness and Docker sandbox, evaluate all 288 scenes and 5,130 questions, and ordinarily operate under a $3 per-scene ceiling.The shared setup includes Blender, Python, Bash, and FFmpeg, with the same system prompt unless otherwise noted.
  • 3.1 Experimental Setup: Table 1 reports a subset of results across proprietary and open-weight models, with global rank and reasoning-effort badges enabling comparison across configurations.The full benchmark contains 51 configurations from ten model families.
  • 3.2 Main Results: 53.7 is the best Dual VQA score, while Latent Similarity spans 57.2–88.6, showing that even top reconstructions lose nearly half of source-available spatiotemporal answers.Overall scores span 47.50–70.07 across 51 configurations, and neither evaluation axis determines the other.
  • 3.2 Main Results: GPT-6-Astra-high ranks first at 70.07 Overall, ahead of GPT-5.6-Sol-xhigh at 67.49 and Grok-4.6-xhigh at 67.17.The 2.58-point gap between the top two Overall scores is statistically significant, although their Dual VQA difference is not.
  • 3.3 Human Alignment: Fifteen blind raters produce the same configuration ordering as Overall, while their scene- and model-level preferences correlate strongly with Latent Similarity at Spearman ρ=0.83.The mean human ranking matches the Overall order exactly, with Spearman ρ=1.00.

4 Analysis

BVB shows that visual similarity and semantic retention diverge: reconstructions can look convincing while losing spatiotemporal facts, with difficulty and model strengths varying by task. Dual VQA and Latent Similarity therefore provide complementary evidence, while reasoning effort and cost do not reliably predict balanced quality.

  • 4.1 How large is the gap between looking right and being right?: 88.6 Latent Similarity contrasts with only 53.7% Dual VQA retention, leaving nearly half of source-correct spatiotemporal answers lost after reconstruction.The top model retains 981 of 1,827 source-correct questions; the remaining 846 are lost.
  • 4.2 What spatiotemporal information is retained after reconstruction?: Object size and route planning achieve the highest mean retention at 63.0% and 58.5%, while appearance order and object count fall to 16.1% and 29.3%.Single-object properties are easier than tasks requiring whole-scene structure, such as counting instances or tracking the camera trajectory.
  • 4.2 What spatiotemporal information is retained after reconstruction?: Task rankings vary across configurations: object count spans 10.5–48.5%, appearance order 4.0–36.0%, and no configuration dominates every task.Relative direction is more compressed at 44.9–57.4%, indicating both shared difficulty and task-specific model strengths.
  • 4.3 Are the metrics complementary and human-aligned?: Latent Similarity correlates with human preference at Spearman ρ=0.83, whereas Dual VQA correlates at ρ=0.11; Overall preserves the five-configuration human ordering at ρ=1.00.LS tracks perceptual preference, while DV prevents visually similar but factually incomplete reconstructions from receiving a high balanced score.
  • 4.3 Are the metrics complementary and human-aligned?: Higher reasoning effort generally improves LS more reliably than DV, and longer runtime does not consistently produce higher Overall.Additional reasoning may refine geometry, materials, and camera motion without verifying object counts or spatial relations; intermediate effort steps can reverse Overall rankings.
  • 4 Analysis: Configurations costing a fraction of the top price still exceed 90% of the best Overall, so cost and quality do not scale together.The top Overall configuration costs $1.258 per scene, while alternatives reach 96% at $0.778 and 91% at $0.024.

5 Related Work

Prior video-understanding benchmarks mainly use question answering, while related programmatic-vision systems generate executable programs for reasoning or reconstructing visual content. BVB builds on this latter direction by focusing on Blender-based reconstruction of video scenes.

  • Benchmarking video understanding: Most video-understanding benchmarks evaluate models through question answering, including benchmarks targeting spatial, long-form, perspective, and egocentric video understanding.The related benchmarks probe different video capabilities but retain the question-answering format.
  • Programs as visual representations: Programmatic visual-reasoning systems use executable code as a persistent artifact, either composing perception modules or rebuilding depicted content in Blender.Related systems include ViperGPT and VisProg for generated Python, VIGA and Kubrick for Blender reconstruction, and SceneActBench for shared Blender-loop evaluation.

6 Conclusion

BVB provides a controlled, reproducible test of spatiotemporal video understanding through editable Blender reconstructions, but current agents still lose many factual details despite recognizable visuals.

  • 6 Conclusion: BVB evaluates spatiotemporal understanding by asking agents to reconstruct real videos as editable Blender scenes under shared budgets and standardized sandbox conditions.Agents operate through Mini-BVB, with asset-free primitive-based scene construction and animated cameras.
  • 6 Conclusion: Agents must inspect video frames before building and revising scenes, then save an executable Blender artifact as the final output.The harness provides frame requests and sandbox commands, with no fixed turn or frame budget beyond the per-scene dollar limit.
  • 6 Conclusion: The benchmark evaluates all 51 configurations with Overall computed from the two evaluation axes, while Table 3 reports the full leaderboard.The supplied passages identify the full configuration set and the square-root-mean aggregation, but not the table’s individual cell values.

C Leaderboard Uncertainty

A paired scene bootstrap finds a statistically reliable Overall and Latent Similarity lead for Astra over Sol-xhigh, but not a significant Dual VQA advantage.

  • C Leaderboard Uncertainty: 2.58 points is Astra’s Overall lead over Sol-xhigh, with a 95% interval of [0.72, 4.45].The paired bootstrap resamples 288 scene IDs 10,000 times and recomputes both axes and Overall.
  • C Leaderboard Uncertainty: Astra leads Sol-xhigh in Latent Similarity, whereas the 1.97-point Dual VQA difference has an interval crossing zero.The authors therefore treat Astra as the Overall and LS leader without claiming a statistically significant DV advantage.

D Derivation of the Square-Root Mean

The benchmark combines Dual VQA and Latent Similarity with a square-root mean designed to reward balanced performance across semantic and perceptual axes.

  • D Derivation of the Square-Root Mean: Overall increases whenever either axis improves, remains nonzero when one axis is zero, and lies between the geometric and arithmetic means.The aggregation choice does not materially change the headline ordering: the same three configurations remain top-ranked under arithmetic and geometric means.
  • D Derivation of the Square-Root Mean: Dual VQA measures conditional retention on questions the judge answers correctly from the source, using the same questions and sampled frames for source and reconstruction.The denominator includes 1,827 judge-correct questions out of 5,130 total, preventing source-judge failures from being attributed to reconstructions.
  • D Derivation of the Square-Root Mean: Latent Similarity compares source and rendered clips with a frozen V-JEPA 2.1 encoder using global representations and temporally pooled layout features.Both videos are sampled across their full temporal extent, and features are discarded after scoring.

G Additional Score Diagnostics

Additional diagnostics show that task difficulty, source domain, scene variability, and provider-specific effort all affect benchmark outcomes beyond aggregate leaderboard scores.

  • G Additional Score Diagnostics: Object size and route planning are retained more reliably than appearance order and object count, while scene-level Dual VQA comparisons contain substantial ties.The task profile and pairwise scene wins reveal structure that aggregate ranks alone do not capture.
  • G Additional Score Diagnostics: 22.6% of judge-failed multiple-choice questions are recovered on average from successful renders, below the 27.9% blended chance level.No configuration exceeds 25.1%, indicating that recovery outside the source-correct subset remains limited.
  • G Additional Score Diagnostics: Every shown configuration scores higher on ARKitScenes than ScanNet for both Dual VQA and Latent Similarity, indicating sensitivity to source domain.The collections differ in capture style, scene composition, and question mix, so the comparison is not attributable to one dataset factor alone.
  • G Additional Score Diagnostics: +0.65 points is the unweighted mean scene-level Dual VQA difference, versus +1.97 points for the pooled question-level leaderboard difference.These summaries answer different questions because scene-level denominators vary with the number of source-correct questions.
  • G Additional Score Diagnostics: Latent Similarity changes most monotonically with provider-defined effort, but additional runtime is neither necessary nor sufficient for an Overall gain.Some adjacent effort steps improve Overall while reducing runtime, whereas others add runtime with little or negative score change.
  • G Additional Score Diagnostics: Figures 21–22 illustrate localized semantic failures despite high perceptual similarity, including missed size, count, route, direction, and appearance-order facts.These cases demonstrate metric complementarity rather than estimating failure prevalence.

J Blind Human Ranking Study

The blind study compares anonymized reconstructions across object identity, spatial layout, camera path, and appearance order, with Latent Similarity closely matching human preference. Dual VQA shows a much weaker association, reflecting its role as a semantic-retention measure.

  • Study Design: Fifteen raters ranked five anonymized reconstructions per scene using object identity, spatial layout, camera path, and appearance order.Each rater saw nine scenes from a fixed pool of 24, producing 135 judgments per configuration.
  • Human Preference Correlation: Spearman ρ=0.83 links Latent Similarity with human preference across 120 scene-model observations.The correlations are descriptive and clustered within 24 scenes and five configurations.
  • Human Preference Correlation: Dual VQA correlates weakly with human preference at Spearman ρ=0.11 over 110 observations.Ten scene-model cases lacked judge-correct source questions and therefore had undefined conditional retention.
  • Configuration and Cost Comparison: Reconstruction cost ranges from $0.024 to $2.157 per scene and excludes Dual VQA and encoder scoring costs.The reported spend measures production cost from provider usage accounting.
  • Configuration and Cost Comparison: The blind study evaluates five configurations, while the frontier analysis identifies three DV-frontier and eight LS-frontier configurations.Astra reaches 53.7 DV and 88.6 LS at $1.258 per scene, whereas Sol-xhigh reaches 51.7 and 85.4 at $0.778.

L Limitations

BVB is scoped to indoor egocentric videos from three capture sources, and its evaluation also depends on programmatic expression, rendered-video metrics, and a fixed VLM judge. The shared cost ceiling standardizes comparisons but may limit further improvement.

  • Scope: BVB covers indoor egocentric videos from three capture sources; broader environments, outdoor scenes, and interactive editing remain future work.The benchmark’s environmental scope does not yet include those settings.
  • Evaluation Boundary: Performance depends on coding ability and Blender familiarity, so BVB measures video understanding through programmatic expression.This couples the evaluation with an agent’s ability to implement its understanding in Blender.
  • Evaluation Boundary: Both axes evaluate rendered video rather than underlying 3D geometry because aligned ground-truth geometry is unavailable.This matches the source-conditioned setting used by the benchmark.
  • Evaluation Dependence: Dual VQA depends on a VLM judge, mitigated by conditioning on judge-correct source questions and fixing the judge across configurations.The blind ranking study covers five configurations, while newer models can be added under the released protocol.
  • Budget Constraint: The shared per-scene cost ceiling keeps comparisons uniform and affordable, though larger budgets might improve individual configurations.The ceiling is a comparison constraint rather than a claim that each configuration has reached its maximum performance.
Loading 2609.15478v1…