PaperlayerView exact revision on arXiv

Source-linked AI summary

CineRecon: Benchmarking Frontier Models for Agentic Cinematic Reconstruction

Jialei Wang, Zehan Wang, XiaoYang Wu, Ziang Zhang, Siyu Chen, Jiaer Huang, Yujia Zhang, Tianyu Pang, Hengshuang Zhao

arXiv:2610.16577v1cs.CV

TL;DR

Editable dynamic 3D reconstructions could give filmmakers control over scene layout, subject actions, and camera behavior, but whether agents can faithfully reenact complete videos remains an open question. CineRecon benchmarks this task across representation settings and evaluation dimensions; simplifying appearance requirements improves some scores, while action timing, contact, and coordinated subject motion remain difficult.

  • Problem

    Whether agents can autonomously construct editable 3D scenes from complete videos and faithfully reenact their content and events remains an open question.

    Source: §1 · p. 2

  • Method

    CineRecon evaluates agents reconstructing 50 complete videos as editable Blender scenes across three representation settings, using entity- and time-linked checklists and aligned-frame judging.

    Source: §1 · p. 2

  • Results

    Simplifying appearance requirements improves camera and dynamic consistency for most models, but action timing, contact relations, and coordinated subject motion remain difficult under Proxy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Joint reconstruction of geometry, actions, and camera behavior remains a priority for graphics agents, and CineRecon provides a basis for measuring progress toward editable visual proxies.

    Source: §6 · p. 9

    Highest action-event scores are 42.7 in Proxy, 46.6 in Simplified, and 48.6 in Detailed, indicating incomplete reproduction of specific actions and interactions.

    Source: §4.2 · pp. 6–7

Abstract

from arXiv · show

Reconstructing reference videos as editable dynamic 3D scenes is an important path toward controllable video generation for filmmaking: these scenes provide modifiable, renderable visual proxies that enable explicit control over scene layout, subject actions, and camera behavior. To evaluate agents' ability to reconstruct editable dynamic 3D scenes from dynamic reference videos, we introduce CineRecon, a benchmark comprising 50 reference videos and 360 annotated evaluation clips. The reference videos span film, real-world footage, and games. Given a complete video, an agent uses tools to construct, animate, render, and refine its reconstruction. We evaluate this task under three representation settings: Proxy, Simplified, and Detailed. These settings increase geometric and appearance requirements while retaining the same requirements for spatial relations, camera behavior, and subject dynamics. This design tests whether agents can reproduce the events in a video when detailed appearance is unnecessary, and how performance changes as representation requirements increase. For evaluation, we construct checklists that associate observable properties and events with specific entities and times. A vision-language judge compares timestamp-aligned reference and rendered frames across 11 categories, grouped into camera, static, and dynamic consistency dimensions. We evaluate 12 models using their corresponding agent harnesses and find failures across all three dimensions. Subject motions and interactions remain difficult to reproduce even under the Proxy setting.

1. Introduction

Editable dynamic 3D scenes could give creators control over layout, actions, and camera behavior, but whether agents can faithfully reenact complete videos remains an open question. CineRecon benchmarks that capability and finds that simplifying appearance requirements does not eliminate important motion and interaction errors.

1. Introduction

  • Unlike prior video-reconstruction evaluations focused on indoor videos or dynamic phenomena, CineRecon tests joint cinematic reconstruction from complete videos without supplied assets or camera parameters.Cinematic reconstruction jointly requires scene content, subject actions and interactions, camera motion, and shot transitions.Source: §1 · p. 2
  • Agents receive complete videos and autonomously construct, animate, render, inspect, and revise editable Blender scenes to match content and events at corresponding timestamps.The benchmark comprises 50 reference videos spanning film, real-world footage, and games.Source: §1 · p. 2
  • Checklists link observable properties and events to entities and times, while a vision-language judge compares timestamp-aligned reference and rendered frames across camera, static, and dynamic consistency.The dimensions cover camera behavior and shot transitions; object presence, attributes, and spatial relations; and timed subject motion and action events.Source: §1 · p. 2
  • Proxy, Simplified, and Detailed vary geometric and appearance demands while keeping camera and dynamic checks—and spatial-relation requirements—unchanged across settings.This design compares performance as representation demands change and examines whether dynamic-event errors persist when appearance requirements are simplified.Source: §1 · p. 2
  • Most models improve camera and dynamic consistency with simpler appearance, yet action timing, contact, and coordinated motion remain difficult even under Proxy.These results identify shared weaknesses in spatial relations, specific action events, and camera changes.Source: §1 · p. 2

2. CineRecon

CineRecon evaluates agents that turn complete reference videos into editable Blender scenes and rendered animations. Its three representation settings scale geometric and appearance demands, while annotated clips and a vision-language judge assess reconstruction against the reference.

2.1. Task Formulation

  • Agents receive a complete reference video and representation setting, construct an editable Blender scene, and render an animation to match observable content at corresponding timestamps.Outputs must match the reference resolution, frame rate, frame count, and duration; multi-shot videos also require camera shot transitions.Source: §2.1 · p. 2

2.2. Representation Settings

  • Proxy, Simplified, and Detailed progressively require geometric stand-ins, recognizable structures, and reference-specific visible details, while retaining the same spatial, camera, motion, interaction, and timing requirements.Proxy omits category-specific appearance details; Simplified permits fine-detail simplification; Detailed requires distinctive local shapes and fine appearance features.Source: §2.2 · p. 3

2.3. Data Collection and Annotation

  • The dataset contains 50 videos spanning captured and rendered content, with 360 evaluation clips and checklists reviewed and refined by humans.Camera motion appears in 94% of videos, subject interaction in 58%, and shot transitions in 18%.Source: §2.3 · p. 4

2.4. Evaluation Protocol

  • A vision-language judge compares timestamp-aligned reference and rendered frames against checklists covering camera, static, and dynamic consistency.Scores average check items within clips and clips equally, then take the arithmetic mean of the three dimensions so annotation-heavy dimensions do not dominate.Source: §2.4 · pp. 4–5

3. Experiments

The experiments evaluate 12 multimodal coding models through their corresponding agent harnesses across three representation settings. Automatic scoring uses a vision-language judge, and a separate pairwise study compares model outputs with human preferences.

3.1. Experimental Setup

  • Twelve multimodal coding models are evaluated, with Opus 5.5 at low reasoning effort added as a thirteenth configuration for quantitative comparisons.Each configuration reconstructs all 50 videos under Proxy, Simplified, and Detailed settings.Source: §3.1 · pp. 5–6

3.2. Automatic Evaluation

  • GPT-5.6 Sol judges timestamp-aligned reference and rendered frames against applicable checklist items, which are scored 0, 0.5, or 1 and aggregated on a 0–100 scale.The judge receives the reference entity table and frames covering annotated observations, event intervals, and shot transitions.Source: §3.2 · p. 6

3.3. Human Evaluation

  • The human preference study comprises 1,440 pairwise comparisons, split equally across camera, static, and dynamic consistency, with ties counted as half a win.Each comparison presents a reference clip or frame and two anonymized reconstructions under the same representation setting.Source: §3.3 · p. 6

4. Analysis

Automatic scores align closely with human model rankings, but action events, spatial relations, and camera changes remain shared weaknesses. Simplifying appearance helps most models, while iteration and higher effort improve Opus 5.5’s reconstruction.

4.1. Alignment with Human Judgments

  • Automatic evaluation correlates with human preferences at 0.951 Spearman and 0.975 Pearson across 12 models; both rank Opus 5.5 and GPT-6 Astra first and second.The evaluations share three models in their top four.Source: §4.1 · p. 6

4.2. Performance and Limitations

  • The highest action-event scores are 42.7, 46.6, and 48.6 in Proxy, Simplified, and Detailed, while action events trail subject motion across configurations.These scores indicate that even stronger models incompletely reproduce specific actions and interactions.Source: §4.2 · pp. 6–7
  • Opus 5.5 has larger advantages in motion, actions, and viewpoint changes, though GPT-6 Astra reproduces ground support that Opus misses in one local-contact case.On Detailed, Opus 5.5 versus GPT-6 Astra scores are 70.3 versus 44.5 for subject motion, 74.9 versus 42.1 for camera rotation, and 48.6 versus 27.2 for action events.Source: §4.2 · p. 7

4.3. Representation Effects

  • Proxy raises mean camera and dynamic consistency by 7.4 and 4.1 points over Detailed across 13 configurations, but not uniformly across models.Proxy scores are higher for 12 of 13 configurations on camera consistency and 10 of 13 on dynamic consistency.Source: §4.3 · pp. 7–8, Figure 8 · p. 8
  • Even under Proxy, models make errors in action direction, timing, and phase transitions, including steering reversal and landing.GPT-6 Astra scores 44.6 on dynamic consistency in Proxy, while Opus 5.5 scores 60.8.Source: §4.3 · p. 8

4.4. Do Iteration and Effort Help?

  • Full Opus 5.5 scores 76.6 overall, exceeding No Refine by 10.2 points and Low Effort by 27.1; both variants lose most on dynamic consistency.The comparison indicates that iterative refinement and higher reasoning effort improve dynamic fidelity and visual-detail reproduction.Source: §4.4 · p. 8

5. Related Work

Prior work studies graphics agents, dynamic video reconstruction, and controllable 3D-guided video generation, often with narrower inputs or task scope. CineRecon focuses on jointly reconstructing cinematic content, actions, camera movement, and transitions.

5. Related Work

  • Graphics-agent research covers scene reconstruction from images and procedural modeling, while existing video benchmarks focus on narrower settings than CineRecon’s joint cinematic reconstruction task.The cited benchmarks include IR3D-Bench [29], 3DCodeBench [12], BVB [38], 4DCodeBench [36], SceneActBench [43], and VisPhyWorld [25].Source: §5 · p. 8
  • Explicit 3D structure and dynamics guide video generation through rendered scenes, editable meshes [32], camera-trajectory proxies [10], and simulated motion [31].CineRecon evaluates the upstream reconstruction of editable dynamic scenes from reference videos.Source: §5 · p. 8
  • Figure 9 compares six Proxy cases across articulated actions, hand–object interaction, steering, and landing using time-matched reference and model frames.Green and red boxes mark reproduced details and remaining errors.Source: Figure 9 · p. 9
  • Video evaluation spans quality, motion, temporal compositionality, and entity-action mismatches, while CineRecon ties checks to specific entities and times.Related approaches include video–language evaluation [6], CameraBench [26], and question-based image evaluation [8] [19].Source: §5 · pp. 8–9

6. Conclusion

CineRecon benchmarks editable Blender reconstructions of dynamic cinematic videos across camera, static, and dynamic checks and three representation settings. Across 12 models, simplifying appearance improves many scores, but action timing, contact, and coordinated motion remain difficult.

6. Conclusion

  • CineRecon evaluates 12 models and finds shared weaknesses in spatial relations, specific actions, and camera changes; simplification helps most models, but errors persist under Proxy.The conclusion identifies joint reconstruction of geometry, actions, and camera behavior as a priority for graphics agents.Source: §6 · p. 9
  • The benchmark covers dynamic cinematic videos as editable Blender scenes, assessed through camera, static, and dynamic checks under three representation settings.The paper presents CineRecon as a basis for measuring progress toward editable visual proxies for controllable video generation.Source: §6 · p. 9
  • Figure 10 compares Opus 5.5 scores across execution configurations and shows a Detailed steering sequence, marking reference details, reproduced steering, and errors.The marked errors distinguish a rotating display under No Refine from absent steering under Low Effort.Source: Figure 10 · p. 9

A.1. Dataset Statistics

The appendix details dataset and checklist scope, how evaluation clips and subjects are annotated, and an example where a recognizable pose fails a contact requirement.

A.1. Dataset Statistics

  • The dataset contains 50 videos and 360 clips; the three settings total 49,668 setting-specific checks, which merge to 35,836 released checklist records.The videos total 763.59 seconds, and clips average 1.41 seconds and last at most two seconds.Source: §A.1 · p. 12, Table 1 · p. 12

A.2. Evaluation Clip and Checklist Construction

  • Clips cover scene states, actions, interactions, camera changes, and transitions; check items tie assessable properties to entities and frames or intervals.Annotations receive multiple rounds of human review, and appearance-check granularity varies by setting while camera and dynamic checks remain consistent.Source: §A.2 · p. 12

A.3. Subject Identification and Temporal Annotation

  • Fixed subject identifiers preserve correspondence within each video, while first appearance marks the earliest identifiable visible part linked through the temporal sequence.Local frame intervals include both endpoints, and released records map local frames to source-video offsets.Source: §A.3 · p. 13

A.4. Annotation Example

  • In the 1.96-second example, Opus 5.5 matches a crouching pose but misses final hand-ground support, despite fully satisfying selected subject and clothing checks.The clip checklist contains 55 checks; the selected support relation and action requirement are rated unsatisfied.Source: §A.4 · p. 13, Figure 11 · p. 13, Table 3 · p. 14

B. Reconstruction Task and Configuration

The benchmark asks agents to reconstruct complete reference videos as editable, animated scenes through iterative inspection and refinement. Three settings progressively increase appearance and geometry requirements, while model-specific harnesses provide the tools and execution configuration.

B.1. Task Prompts for the Three Representation Settings

  • Agents receive the complete video and must construct, animate, inspect, and revise an editable scene while preserving subjects, spatial relations, actions, camera behavior, timing, and shot transitions.Deliverables include the editable Blender scene, reconstruction video, synchronized comparison materials, and a verification report.Source: §B.1 · pp. 14–15
  • Proxy permits simple geometric stand-ins, Simplified requires recognizable low-complexity objects, and Detailed requires reference-specific visible shape, appearance, and motion.All three settings retain the task's spatial, camera, and dynamic requirements; the prompts specify progressively richer appearance and geometry expectations.Source: §B.1 · p. 15

B.2. Agent Tools and Execution Environment

  • Agents work in Blender 5.2 with Blender MCP and supporting media tools, choosing their analysis frames, scene parameterization, animation, and intermediate render settings.The reconstruction loop includes reference inspection, scene editing, test rendering, comparison, and further editing; runs use Linux GPU workers and CUDA rendering instructions.Source: §B.2 · p. 15

B.3. Model Configuration

  • Each model uses its corresponding agent harness across all three settings, with specified reasoning configurations and a 32,768-token output budget per response.The reported comparisons cover 13 configurations; the Opus 5.5 ablation also includes a No Refine condition.Source: §B.3 · p. 15, Table 4 · p. 16

C. Automatic Evaluation Details

The evaluation aligns reference and reconstructed video evidence at annotated times, applies setting-specific checklist judgments, and aggregates validated scores across clips and consistency dimensions. Operational failures are recorded separately from visual mismatches.

C.1. Temporal Alignment of Reference and Reconstructed Videos

  • The pipeline maps annotated local frames to the complete reference timeline and selects mandatory evidence frames, retaining all mandatory frames if the request exceeds 50 aligned pairs.Additional frames are selected to maintain coverage between required observations, and paired reference and reconstruction frames are presented in increasing time order.Source: §C.1 · p. 16

C.2. VLM Judge Prompts and Rating Rules

  • A vision-language judge evaluates checklist statements at their specified times, maintains entity correspondence, and applies appearance requirements appropriate to Proxy, Simplified, or Detailed.Allowed simplifications incur no penalty; checks for presence, relations, camera, and dynamics are judged according to their statements.Source: §C.2 · pp. 16–17, Table 5 · p. 17

C.3. Score Aggregation and Failure Handling

  • Scores average item ratings within each clip and dimension, then equally weight applicable clips and the three dimensions on a 0–100 scale.Missing or undecodable videos and invalid judge responses are recorded as operational failures, separately from visual failures such as missing subjects or incorrect contact.Source: §C.3 · pp. 17–18

D.1. Sample Selection and Comparison Setup

The human study compares anonymized reconstructions on the same video, representation setting, and consistency dimension. Independent ratings are aggregated by majority vote, with unusable or split-vote tasks excluded.

D.1. Sample Selection and Comparison Setup

  • The study comprises 1,440 pairwise tasks across 50 videos and 12 models, with 480 tasks each for camera, static, and dynamic consistency.Sampling balances exposure across videos, models, and model pairs without using score magnitudes or differences to select tasks.Source: §D.1 · p. 18, Table 6 · p. 18

D.2. Annotation Procedure and Preference Aggregation

  • Three raters independently judge anonymized model pairs against the reference under one dimension and representation setting, and majority vote determines each task label.Tasks with issue reports or three different votes are excluded from both human and automatic aggregation.Source: §D.2 · pp. 18–19, §D.1 · p. 18

D.3. Agreement with Automatic Evaluation

Human and automatic model rankings align strongly across camera, static, and dynamic consistency. Win rates use the same valid comparison tasks, with ties counted as half wins and settings and dimensions weighted equally.

D.3. Agreement with Automatic Evaluation

  • Human and automatic win rates compare model pairs on the same valid tasks, count ties as half wins, and give equal weight to the three settings and dimensions.Automatic comparisons use scores for the same dimension, clip, and representation setting.Source: §D.3 · p. 19
  • Of 1,440 comparison tasks, 1,360 had valid preference labels; the exclusions were 19 issue reports and 61 three-way splits.The analysis measures rank and numerical association between human and automatic model win rates across the 12 models.Source: §D.3 · p. 19
  • Spearman correlations were 0.902, 0.965, and 0.832 for camera, static, and dynamic consistency, respectively; both evaluations ranked Opus 5.5 and GPT-6 Astra first and second across dimensions.Pearson correlations were 0.959, 0.951, and 0.904 for camera, static, and dynamic consistency, respectively.Source: §D.3 · p. 19, Table 7 · p. 19, Figure 12 · p. 20

E.1. Complete Quantitative Results

The appendix reports complete scores and category-level differences, then presents qualitative reconstructions and comparisons across representation settings, actions, and camera changes.

E.1. Complete Quantitative Results

E.2. Additional Reconstructions

E.3. Comparisons across Representation Settings

E.4. Actions and Camera Changes

Related papers

Loading 2610.16577v1…