PaperlayerView exact revision on arXiv

Source-linked AI summary

ROWBench: Do Video Models Render What the Program Specifies?

Zheng-Hui Huang, Guixu Lin, Yu-Ju Tsai, Jian-Kai Zhu, Fengbo Lan, Yu-Lun Liu, Yung-Yu Chuang, Kaipeng Zhang, Zhixiang Wang

arXiv:2610.02205v1cs.CV

TL;DR

Video world models may produce plausible scenes without faithfully rendering programmable rules and interactions, and existing benchmarks rarely test fine-grained program-specified events. PROWBench uses replayable world records and synchronized views and proxies to evaluate control, timeline adherence, and event realization. Its experiments find that prompted general-purpose models can follow prescribed trajectories and event logic well, while long-video persistence and camera following remain challenging.

  • Problem

    Existing benchmarks rarely assess fidelity to fine-grained program-specified events, leaving a gap in evaluating whether generated video realizes explicit rules and interactions.

    Source: §1 · p. 2

  • Method

    PROWBench records entity states and timestamped events, including off-camera events, and renders synchronized views and proxies for evaluation against program execution.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    With appropriate prompts, general-purpose video models follow prescribed entity trajectories and event logic well, but long-video camera following and entity persistence remain challenging.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Replayable records and synchronized proxies support controlled comparisons across representations, viewpoints, and time, while event metrics complement geometric and perceptual measures.

    Source: §5 · p. 15

    Long horizons, re-entering entities, and multi-view identity remain open problems in the challenge track.

    Source: §4.5 · p. 13

Abstract

from arXiv · show

Programmable world models separate executable dynamics from visual generation, offering a promising foundation for next-generation game engines. However, their visual adherence to explicit rules and interactions remains insufficiently evaluated. Existing benchmarks assess visual quality, controllability, and instruction or physical adherence, but rarely test fidelity to fine-grained, program-specified world events. We introduce PROWBench, comprising 170 programmatically constructed episodes and 600 proxy videos covering diverse scenes and interactions. PROWBench logs entity states and timestamped events, including those outside the camera's field of view, as replayable world records, from which it renders synchronized views and proxy representations. This enables generated videos to be checked against the observable consequences of program execution. An extensible framework constructs scenes, controls behaviors, and can render each camera view in different representations, such as coarse 3D, and bounding boxes. The benchmark covers first- and third-person perspectives, with synchronized multi-view observations available for a subset of episodes. Grounded in these records, PROWBench evaluates entity control, long-horizon memory, and, with two VLM-based metrics, Logic-Render Alignment and Interaction Success Rate, adherence to the prescribed timeline and the visual realization of timestamped engine-recorded events.

1 Introduction

Programmable world models make state evolution explicit, but correct program execution does not ensure that generated videos faithfully depict specified rules and interactions. PROWBench addresses this evaluation gap with replayable records, synchronized representations, and metrics for timeline adherence and interaction realization.

1 Introduction

  • Existing benchmarks rarely test fine-grained, program-specified events, and without replayable state and event records, video plausibility cannot be checked against program execution.Synchronized proxy representations and views are also uncommon, limiting controlled comparisons across representation and viewpoint.Source: §1 · p. 2
  • PROWBench logs entity states and timestamped events, including off-camera events, as replayable records for checking videos against program execution.The benchmark renders synchronized views and proxy representations from those records.Source: §1 · p. 2
  • With appropriate prompts, general-purpose video models can follow prescribed trajectories and event logic well, while task- or domain-specific training can improve some aspects.Long-video camera following and entity persistence remain challenging, and direct multi-view generation remains unsatisfactory.Source: §1 · p. 2
  • PROWBench comprises 170 dynamic scene episodes and 600 synchronized proxy videos spanning multiple perspectives, visual representations, and diverse interactions; its three synchronized representations derive from shared state, action, and interaction records.The representations are coarse 3D, semantic proxy, and colored oriented bounding boxes; some episodes additionally provide synchronized multi-view observations.Source: §1 · pp. 2–3
  • Entity control and long-horizon memory are evaluated against records, while Logic-Render Alignment and Interaction Success Rate assess timeline adherence and visible realization of recorded events.The metrics use engine-recorded events and states as grounding for evaluation.Source: §1 · pp. 2–3
  • The reusable generation framework supports extensions to scene layouts, actions, and visual representations while producing aligned multi-view data from shared records.It separates scene construction, behavior control, state recording, and rendering.Source: §1 · pp. 2–3

2 Related Work

Prior work develops video world models, structured visual controls, and benchmarks for quality, dynamics, and interaction. PROWBench focuses on whether rendered observations faithfully realize explicit states and events, with paired representations and synchronized perspectives supporting controlled comparisons.

2.1 Video World Models and Programmable World States

  • Video world models infer future observations from visual context and actions, whereas programmable models externalize state evolution in executable programs or persistent structured states.PROWBench evaluates whether observations faithfully realize those explicit states and interactions.Source: §2.1 · p. 3

2.2 Structured Representations for Visual Generation

  • Structured visual controls range from sparse object controls to dense geometric representations, but comparisons are often confounded by differing models, datasets, and tasks.PROWBench derives paired proxies from the same recorded trajectory and camera configuration to isolate representation effects.Source: §2.2 · p. 3

2.3 World Model Benchmarks and Evaluation

  • Existing benchmarks assess quality, physical plausibility, controllability, instruction adherence, and consistency, often evaluating complete pipelines rather than rendering fidelity alone.PROWBench uses frame-level engine states, rules, and timestamped interactions to compare synchronized perspectives and representations under the same trajectory.Source: §2.3 · p. 3

3 Benchmark

PROWBench constructs replayable programmable worlds, generates synchronized conditioning inputs and annotated observations, and evaluates control, event realization, and memory across benchmark settings. Its pipeline retains shared state records so views and representations can be compared against the same world evolution.

3.1 World Generation Pipeline

  • World synthesis configures environments, entities, controllers, seeds, and duration, then records entity states, controller events, and temporal links to video frames.Controllers coordinate phased interactions, locomotion or activity, and driving behaviors; saved states can be reconstructed from shared scene descriptions and dynamic records.Source: §3.1 · p. 4
  • Control compilation maps recorded states and camera trajectories to synchronized conditioning inputs across first- and third-person views and multiple proxy representations.Representations include coarse 3D, semantic proxy, and colored oriented bounding boxes; different views and representations share saved states and camera parameters.Source: §3.1 · pp. 4–5
  • The final process exports replayable scene and motion records, synchronized proxy videos, depth, and text and interaction labels, with validation across timelines, geometry, alignment, visibility, and media integrity.Provenance records link released observations to source records, rendering configurations, and verification reports.Source: §3.1 · p. 5

3.2 Evaluation Suite

  • Entity controllability measures trajectory alignment using BBox IoU and Center Error, while engine-recorded states and events ground adherence and perceptual metrics assess video quality.Entity detections are matched to engine-projected references; missed detections receive zero BBox IoU.Source: §3.2 · p. 6
  • Interaction Success Rate counts an event as successful only when both its prescribed action and expected end state are judged present.The judge evaluates sampled frames around registered event windows, including full-frame context and an engine-localized crop.Source: §3.2 · p. 6
  • Logic-Render Alignment measures whether actions in the engine-derived timeline are visibly realized within their corresponding time windows.The score is the fraction of timeline segments whose prescribed actions are visually consistent with engine-defined logic.Source: §3.2 · p. 6
  • Long-horizon memory tests whether entities reappear in the engine-prescribed location and state after leaving view, using reappearance IoU and state persistence.State persistence classifies returning people as walking or standing from post-return frames.Source: §3.2 · pp. 6–7

3.3 Benchmark Tracks

  • The main track tests five-second clips with and without a verified RGB first-frame appearance reference, while challenge settings test 30-second persistence and independently generated synchronized player views.The main-track settings use disjoint scenes; the long-horizon and multi-view challenge settings each contain ten recordings or scenes.Source: §3.3 · p. 7

4 Experiments

The experiments compare eleven video models across short clips, long-horizon sequences, and synchronized multi-view settings, scoring outputs against recorded engine states and events. Proxy-conditioned models generally place entities and render prescribed events better than camera-conditioned models, while challenge-track results expose persistent memory and identity-consistency gaps.

4 Experiments

  • The experiments separate five-second main-track clips from long-horizon and multi-view challenge settings to test prescribed trajectories, camera motion, event timelines and consistency.The main track includes Verified-FF and Unverified-FF settings with disjoint scenes; the challenge track tests 30-second recordings and independently generated synchronized views.Source: §4 · p. 7

4.1 Evaluated Methods

  • The evaluation covers eleven models in three control-input families, using native interfaces and common engine-recorded states, prompts and scoring references.Families are camera-conditioned world models, proxy-video-to-RGB renderers and video foundation models; PWM is evaluated only on the gunfight set.Source: §4.1 · pp. 7–8, §4.2 · p. 8

4.3 Main Track

  • In the main track, proxy-conditioned models lead entity control, interaction and logic metrics distinguish behavior from placement, and most methods’ camera errors are comparable to engine-render pose-estimator errors.LynnReal-Omni, MiniMax-H3 and CWM reach Verified-FF BBox IoU of 0.518, 0.494 and 0.454, versus 0.303–0.351 for camera-conditioned models.Source: §4.3 · p. 9

4.5 Discussion and Takeaways

  • Proxy-conditioned classes outperform camera-conditioned models in entity placement, while general-purpose models exceed them in Logic-Render Alignment across both settings and Interaction Success in Unverified-FF.The class-mean BBox IoU is 0.450 and 0.465 for task-specific and general-purpose models versus 0.322 in Verified-FF, and 0.376 and 0.373 versus 0.235 in Unverified-FF.Source: §4.5 · p. 10
  • On short single-camera clips, general-purpose models are competitive with task-specific renderers across several metrics, including BBox IoU, camera errors, event metrics and warping error.Their class-mean warping errors are 0.38 and 0.33, compared with 0.72 and 0.83 for task-specific renderers.Source: §4.5 · pp. 11–12
  • Fine-tuning yields mixed gains across models and objectives, while domain-specific gunfight events favor the dedicated PWM model over general-purpose pipelines.On gunfight scenes, MiniMax-H3 reaches BBox IoU 0.526 and Interaction Success Rate 0.600, while PWM reaches 0.602 and 0.850.Source: §4.5 · p. 12

4.4 Challenge Track

  • Over 30 seconds, BBox IoU is 0.175–0.280 and returning-entity Reappearance IoU is 0.053–0.131; multi-view identity consistency also varies with whether inputs bind descriptions to people.Even the best methods render a differently dressed player in 10–19% of judged view pairs, and the scores cover only observations the judge could assess.Source: §4.5 · pp. 13–14

5 Conclusion

PROWBench grounds programmable-world evaluation in replayable state and interaction records, with synchronized proxies and metrics for whether engine-defined rules and events are visibly realized. Experiments show that explicit entity control improves trajectory adherence, while fidelity can still fail across appearance, events, time, and viewpoints.

5 Conclusion

  • PROWBench grounds evaluation in replayable world states and interaction records, with synchronized proxies and metrics for visible realization of engine-defined rules and events.The proxies support controlled comparisons across representations, viewpoints, and time; Logic-Render Alignment and Interaction Success Rate complement geometric and perceptual metrics.Source: §5 · p. 15
  • Proxy-conditioned models adhere to prescribed trajectories more closely than camera-conditioned models, including without an RGB first-frame reference.Visual plausibility does not guarantee faithful world realization: appearance assignment, identity binding, and facing direction remain unreliable, while control and reappearance degrade over longer horizons.Source: §5 · p. 15
  • Programmable-world rendering is framed as preserving executable state across entities, events, time, and viewpoints, rather than optimizing visual quality alone.The paper identifies stronger entity–appearance binding, event-aware generation, long-horizon state memory, and joint multi-view consistency as requirements for reliable renderers.Source: §5 · p. 15

A Benchmark Data

This section reports dataset statistics and describes construction of the benchmark’s first frames and text prompts.

A Benchmark Data

  • The section covers dataset statistics and the construction of benchmark first frames and text prompts.The passage states the section’s scope without giving further details.Source: A Benchmark Data · p. 20

A.1 Data Statistics

PROWBench comprises 170 evaluation episodes, 200 selected views, and 600 proxy videos across four collections. Its records support varied scene and interaction annotations, synchronized representations, and both first- and third-person views.

A.1 Data Statistics

  • 170 evaluation cases provide 200 selected camera views and 600 proxy videos across four collections, with three synchronized representations per selected view.The 600 videos include 570 five-second clips and 30 thirty-second clips, all rendered at 30 FPS.Source: §A.1 · p. 20
  • The collections comprise 130 diverse-scenario episodes, ten Multi4 episodes, 20 Gunfight cases, and ten long-horizon episodes.Multi4 provides four synchronized following views per episode; the long-horizon collection uses one view per episode.Source: §A.1 · p. 20
  • The diverse-scenario collection covers seven scene domains and nine non-exclusive interaction categories, while the long-horizon episodes span 30 seconds.Gunfight cases derive from ten source recordings, so benchmark episode IDs do not necessarily correspond to independent recordings.Source: §A.1 · p. 20
  • Across 170 episodes, reviewed annotations yield 293 episode–tag assignments across nine non-exclusive interaction categories.The labeling policy excludes incidental background co-occurrence and does not imply independently verified physical contact or hit causality.Source: §A.1 · p. 20
  • The annotations cover 30 environment labels and 47 semantic action labels; seven requested visual styles span 200 views.The selected views include 173 third-person and 27 first-person perspectives; median camera path length is 6.11 m and median maximum angular deviation is 6.60°.Source: §A.1 · pp. 20, 22

A.2 First-Frame Construction

The benchmark constructs a shared RGB first-frame reference for first-frame-conditioned methods, then generates and filters candidate images against appearance and engine-record constraints. A separate prompt-based generation path uses proxy input without an RGB reference.

A.2 First-Frame Construction

  • Verified-FF supplies each selected view with a common RGB first frame, chosen to align with proxy geometry and satisfy appearance and geometric constraints.The 40 first frames—17 realistic and 23 stylized—pass automatic verification, and every method accepting a first frame receives the same image.Source: §A.2 · p. 22
  • For proxy-based construction, MiniMax-H3 receives the first 22 colored-OBB frames and a prompt combining layout, proxy-control, appearance, and style instructions, without an RGB reference.Three candidates are generated per view-specific instance using different random seeds.Source: §A.2 · p. 22
  • Candidate first frames undergo hard checks against the frame-0 engine record, including person count and box alignment.Detected person count must match the engine record; mean person-box IoU must reach 0.30, as must each visible animal’s IoU.Source: §A.2 · p. 22
  • Image checks assess target style and defects, including copied proxy geometry, inherited black surfaces, structural defects, and placeholder geometry.For vehicles and mounts with an unambiguous front direction, the visible heading must agree with the corresponding engine-box orientation.Source: §A.2 · pp. 22, 24
  • Candidates failing any check are regenerated with a new seed and targeted constraint sentences, retaining the best candidate that passes all hard checks.Source: §A.2 · p. 24

A.3 Text Prompts

PROWBench constructs reproducible per-view prompts from engine records, separating proxy-driven motion and geometry from appearance and style information. Deterministic conversions adapt the shared prompt content to different model interfaces and proxy representations.

A.3 Text Prompts

  • Except for the appearance block, prompts are generated deterministically from engine metadata, so fixed engine records yield reproducible inputs across methods.The appearance block is generated by a vision–language model from the RGB first frame and a proxy frame used only as a layout reference.Source: §A.3 · pp. 24–25
  • Engine records supply entity annotations and event timelines, augmented with measured path lengths, inter-entity relations, final object locations, and occlusions.Camera motion is provided through the control input, not described in the text prompt.Source: §A.3 · p. 25
  • Verified-FF derives appearance from the verified first frame, whereas Unverified-FF uses the authored scene description; both settings share the same structural prompt skeleton.The skeleton includes subject definitions, proxy bindings, the action timeline, and measured facts.Source: §A.3 · pp. 25–26
  • The base prompt combines a control block, subject definitions, a task summary, retention analysis, and a detailed description with a contiguous five-second timeline.The control block assigns geometry, camera, and motion to the proxy; style and appearance to the first frame; and articulated actions to timestamped text.Source: §A.3 · p. 25
  • An automatic validator checks subject coverage, permitted references, complete 0–5-second timelines, first-frame references, and required representation wording.A failed prompt is returned to the VLM for one regeneration attempt.Source: §A.3 · p. 25
  • Prompt conversions deterministically adapt shared semantic content to model interfaces, while system-role support and proxy-specific wording vary by method.The control block acts as a system prompt, but only CWM has a separate system role; representation-specific wording covers proxy semantics and subject-to-proxy binding.Source: §A.3 · pp. 25–26

B Metric Details

The metric suite measures entity and camera control, appearance across views, and visual or temporal quality, while distinguishing engine adherence from plausibility and consistency.

B Metric Details

  • Entity-control measures use label-compatible matching to compare generated entities with engine-projected references, assigning zero IoU to missed detections.Only dynamic or interaction-relevant entities visible in the reference view are evaluated; Center Error is conditional on detection.Source: B Metric Details · p. 29
  • Camera translation sequences are normalized by their own maximum norm, so TransErr and CamMC measure path shape rather than motion magnitude.RotErr is reported in degrees; normalization addresses monocular scale ambiguity.Source: B Metric Details · p. 29
  • MUSIQ, LAION aesthetic, Warp Error, Motion Smoothness, and CLIP-Score assess visual or temporal quality without engine-state references, measuring plausibility rather than adherence.Warp Error uses masked mean squared RGB error; CLIP-Score compares frame embeddings with the episode action title.Source: B Metric Details · p. 29
  • Appearance Compliance checks agreement with authored descriptions, while Cross-view Appearance Consistency checks whether participants look the same across synchronized views.A participant may be consistent yet non-compliant; unscored observations are excluded, so both measures should be read with the BBox miss rate.Source: B Metric Details · pp. 29–30
  • MEt3R compares generated-view image features after geometry-based alignment, measuring mutual consistency rather than agreement with engine geometry.It complements person-level appearance judgments rather than measuring adherence to an appearance description.Source: B Metric Details · p. 30

C Experiments

The appendix section provides implementation details and qualitative examples for Section 4.

C Experiments

  • This appendix section gives implementation details of the evaluated models and qualitative examples for Section 4.Source: C Experiments · p. 31

C.1 Model Configurations and Implementation Details

The model-configuration appendix summarizes conditioning and output specifications for the five-second main-track experiments, with a separate note on PWM evaluation scope.

C.1 Model Configurations and Implementation Details

  • Table C.1 summarizes conditioning inputs, spatial resolutions, frame rates, and delivered frame counts for the five-second main-track experiments.Figure C.1 shows one Unverified-FF test condition, while Figure C.2 shows conditions for the other settings.Source: §C.1 · p. 31, Table C.1 · p. 31
  • PWM is evaluated only on the gunfight set, and three listed world models generate the full clip without autoregressive rollout.EchoWM uses the Base checkpoint; SANA-WM and LingBot-World 2.0 use bidirectional checkpoints.Source: §C.1 · p. 31
  • AlayaWorld v1.1 uses the camera-controlled Stage2b AR teacher with the Stage3 DMD adapter because its released Stage1 bidirectional checkpoint lacks camera-trajectory control.The Stage2b configuration uses four denoising steps per rollout round.Source: §C.1 · p. 31

C.2 Qualitative Examples

The qualitative examples show that models can follow proxy-guided trajectories without a verified first frame, while leaving appearance, facing, viewpoint, and event outcomes vulnerable to errors.

C.2 Qualitative Examples

  • Without a verified first frame, most models align scenes and trajectories with the proxy, but entity facing and assignment of described appearances remain model-dependent.C2R renders the requested marker-sketch scene photorealistically, while the protagonist's climb remains proxy-aligned across methods.Source: §C.2 · p. 31
  • IoU 0.62–0.86 for proxy-conditioned models, versus 0.27 and 0.29 for camera-trajectory-only EchoWM, shows stronger car-box control.The example tracks two cars using engine-reference boxes.Source: Figure C.3 · p. 33
  • LRA 1.00 but ISR 0.67 for LingBot 2.0: it depicts all three actions but misses the first event's end state.LRA measures the fraction of timeline segments depicting the action; ISR measures the fraction of realized events.Source: Figure C.4 · p. 34
  • In the car-entry scene, proxy-conditioned methods drive away as recorded, while EchoWM and SANA-WM leave the protagonist outside and LingBot 2.0 does not drive.LingBot 2.0 opens the door, but the car remains stationary.Source: Figure C.6 · p. 36
  • The helicopter scene exposes viewpoint errors: EchoWM drops the cockpit, while Cosmos-Transfer2.5 and C2R render pilots inconsistent with the prescribed first-person view.The prescribed view is from a helicopter cockpit during ascent, forward flight, and descent.Source: Figure C.7 · p. 37
  • In the sitting-and-standing example, EchoWM loses the room and renders the protagonist against an empty background.The recorded sequence includes sitting, remaining seated, and standing again.Source: Figure C.8 · p. 38
  • Pair IoU 0.52–0.61 across methods in the stair scene accompanies proxy-aligned climbing, despite varying fidelity to the requested marker-sketch style.C2R renders photorealistically despite the style prompt.Source: Figure C.9 · p. 39
  • Appearance and facing can also be wrong: MiniMax-H3 swaps two outfits, Seedance 2.5 misassigns another and reverses a horse's facing.The outfit examples are from a supermarket scene; the facing error is from a horseback scene.Source: Figure C.10 · p. 39
Loading 2610.02205v1…