PaperlayerView exact revision on arXiv

Source-linked AI summary

HappyWorld-Bench

Zhiqi Bai, Junai Cai, Yixin Chen, Jingrun Du, Tao Feng, Wei Gong, Siyuan Huang, Xiao Lin, Jiaheng Liu, Jun Luo, Yongzhe Lyu, Liya Ma, Zenan Meng, Lin Qu, Wenbo Su, Jiaming Wang, Qinghe Wang, Shaofei Wang, Yanghai Wang, Zequn Wang, Ziming Wang, Hu Wei, Jiangtao Wu, Ruiqi Wu, Jiaxin Xie, Yuchi Xu, Ze Xu, Chengting Yu, Liangyu Yuan, Gang Zeng, Yawen Zeng, Xingyao Zhang, Zizheng Zhang, Bo Zheng, Jiancheng Zhu, Song-Chun Zhu

arXiv:2609.24308v1cs.CV

TL;DR

Existing world-model benchmarks are specialized, while reliable use requires worlds to hold up under interaction. HappyWorld-Bench evaluates video, spatial, and embodied models in a shared framework and finds reliability gaps across all three tracks.

  • Problem

    Existing benchmarks remain specialized to particular model forms or capability subsets, making evaluation under a shared framework difficult despite the need for worlds to remain reliable under interaction.

    Source: §1 · pp. 1–2

  • Method

    HappyWorld-Bench organizes video, spatial, and embodied evaluation in a shared W1–W6 framework, pairing human A/B Arena Elo ratings with automated capability metrics.

    Source: §1 · p. 3, §6 · p. 22

  • Results

    Across all three tracks, models show reliability gaps in video consistency, spatial physical plausibility and editability, and embodied state retention under multi-step actions; the highest spatial placement rate is 70.14%.

    Source: §1 · pp. 3–4, §4.3 · p. 14

  • Takeaways & Limitations

    World models should be assessed not only for generated-world quality but also for reliability under interaction, persistent state changes, and intervention.

    Source: §6 · p. 22

    The embodied benchmark operationalizes only W2–W4; W1, W5, and W6 remain framework-level capability definitions.

    Source: §5.1 · p. 16

Abstract

from arXiv · show

Evaluating world models requires assessing both the quality of the worlds they generate and their consistency and responsiveness under exploration, interaction, and modification. We introduce HappyWorld-Bench, a comprehensive benchmark that evaluates whether generated worlds remain reliable as agents interact with them. Our design is built on a hierarchical capability framework of six world capabilities (W1-W6), from generative construction to unified world modeling, instantiated across three independent evaluation tracks: video world models, spatial world models, and embodied world models. HappyWorld-Bench comprises 1,138 video prompts, 300 spatial scenes, and 254 embodied test cases. Across all three tracks, we build and operate HappyWorld-Arena to organize human A/B comparisons and derive model-level Elo ratings, which complement newly designed automated metrics that capture behavioral correctness. We evaluate 14 video world models, 9 spatial systems, and 8 embodied candidates under this unified framework. Results reveal remaining reliability gaps across all three tracks: video models exhibit reduced consistency during extended rollouts and revisits, spatial models achieve at best 70.14% placement accuracy and 73.33% edit execution, and embodied models struggle to preserve state across multi-step actions and respond precisely to altered action conditions and physical rules. These findings highlight the need to evaluate world models not only by visual quality, but also by state consistency and the correctness of their responses to actions and interventions.

1 Introduction

HappyWorld-Bench addresses the need to judge generated worlds by whether they remain reliable under interaction, not merely by visual quality, using a shared W1–W6 framework across video, spatial, and embodied tracks.

1 Introduction

  • Existing evaluations cover capabilities such as controllability, physical plausibility, memory, interaction, and long-horizon consistency, but remain specialized across model forms and evaluation settings.This fragmentation makes it difficult to compare systems under a shared capability framework (Zheng et al., 2025; Duan et al., 2025; Xu et al., 2026a; Fang et al., 2026; Ying et al., 2026; Ding et al., 2026; Duggal et al., 2025; Tam et al., 2025).Source: §1 · pp. 1–2
  • W1–W6 progress from perceptual coherence and action-responsive simulation through persistent state, causal programmability, scalable multi-agent worlds, and unified universal modeling.The hierarchy provides a common vocabulary for evaluating world models across video generation, 3D reconstruction, and embodied control.Source: §1 · pp. 2–4
  • The video track tests spatial and temporal coherence, the spatial track tests whether scenes support physical operations, and the embodied track tests action-conditioned changes from an egocentric view.All three tracks use the W1–W6 taxonomy while covering different subsets according to model form and evaluation interface.Source: §1 · p. 3
  • 1,138 video prompts, 300 spatial scenes, and 254 embodied test cases make up the 1,692-instance benchmark.The spatial set includes 266 scenes for visual quality, physical usability, consistency, and editing, plus 34 for spatial expansion; data span nine domains, with rollouts up to 60 seconds and first-frame resolutions below 1 MP to above 8 MP.Source: §1 · pp. 3–4
  • HappyWorld-Arena organizes human A/B comparisons into within-track Elo ratings, complemented by domain-specific automated metrics for behavioral correctness.The benchmark applies those metrics to perception, consistency, causality, and controllable interaction in video, and to visual quality, usability, consistency, and editing in spatial scenes.Source: §1 · pp. 3–4
  • 1263 and 1206 are the two leading video systems’ Arena Elo ratings; consistency and action response, spatial physical plausibility and editing, and embodied state preservation remain challenging.Embodied models also struggle to respond precisely to altered action conditions and physical rules.Source: §1 · pp. 3–4

2 Related Work

Related work traces world models from generation toward interactive, persistent environments, while benchmarks increasingly test behavior and reliability beyond appearance. HappyWorld-Bench organizes these concerns across video, spatial, and embodied worlds, including navigation, consistency, editing, and action-conditioned behavior.

2.1 World Model

  • Video world models progress from controllable generation toward longer, more persistent and general interactions, including scene manipulation and multimodal control.Approaches add explicit keyboard, mouse, and camera controls, history conditioning, long-context memory, and extended rollouts.Source: §2.1 · p. 4
  • Spatial world models combine image or panorama generation with 3D reconstruction, while complementary approaches transfer video priors into geometry and support authoring or executable scenes.Systems also target persistence during exploration and efficiency in direct 3D Gaussian Splatting generation.Source: §2.1 · p. 4
  • Embodied world models predict action-driven observations using latent or pixel-space representations, with applications spanning manipulation, driving, and navigation.Research connects video prediction to action generation, planning, physical priors, and foundation platforms.Source: §2.1 · p. 5

2.2 World Model Benchmarks

  • Video benchmarks have expanded from perceptual quality and local controllability to multi-turn interaction, memory, physics, and sustained goal-directed behavior.Recent protocols assess extended persistence, scene and subject memory, closed-loop interaction, and state evolution within and beyond the current view.Source: §2.2 · p. 5
  • Spatial benchmarks assess assets, scene plausibility, and camera-conditioned generation; this track tests construction, navigation, consistency, editing, and expansion in explicit environments.Its common protocol combines rendered observations, exported geometry, and paired operations, including preservation of non-target content and existing regions or paths.Source: §2.2 · p. 5
  • Embodied benchmarks progress from video appearance to action-conditioned reliability and downstream utility; this track evaluates visual quality and behavior through a generative-only interface.Candidates receive one egocentric frame and an action prompt; evaluation covers atomic responses, ordered multi-step state persistence, and matched-pair interventions that edit an action or physical condition.Source: §2.2 · p. 6

3 Video World Model Track

HappyWorld-Bench evaluates video world models across perception, consistency, causality, and interaction, finding that overall human preference and capability-level performance do not always align. Its cases probe natural dynamics, controlled actions, long-term persistence, and persistent rule edits.

3.1 Data Construction

  • Controls vary by capability: W1 tests uncontrolled evolution, W2–W3 specify directions and timing, W4 adds interventions, and W5–W6 can target multiple subjects.Cases also vary scenes, viewpoints, and control patterns to test natural dynamics, appearance and spatial structure, movement responses, persistence, and interaction or rule editing.Source: §3.1 · pp. 6–7
  • The benchmark contains 1,138 W1–W5 cases and 74 facets across five capability dimensions, including consistency, causality, interaction, and multi-subject coordination.The five dimensions cover perception and representation (342 cases), consistency and state retention (228), causality and causal rollout (299), controllable interaction and counterfactuals (212), and multi-subject coordination (57).Source: §3.1 · p. 7

3.2 Evaluation Metrics

  • Metrics assess perception and representation, consistency and state retention, causality and causal rollout, and controllable interaction and counterfactuals.They include image and motion measures, geometric and identity consistency, physical and content causality, and action execution or intervention validity and state fidelity.Source: §3.2 · pp. 7–8

3.3 Experimental Results

  • HappyOyster leads Arena preference at Elo 1263, while Genie 3 leads W1 at 82.8; HappyOyster leads W2 at 78.0, W3 at 76.8, and W4 at 66.7.Genie 3 scores 1206 in Arena and HappyOyster scores 82.4 on W1; W4 results compare only HappyOyster and Lingbot-World-v2, which scores 59.3.Source: Table 2 · p. 8, §3.3 · p. 8
  • W1 results show that strong perceptual quality and instruction following do not necessarily yield stable geometry, appearance, or world state over time.The benchmark treats perception and consistency as complementary capabilities rather than interchangeable ones.Source: Table 3 · p. 9, §3.3 · p. 9
  • W2 distinguishes action execution from transition correctness: control-following scores do not guarantee coherent state preservation or causally consistent outcomes.Models with similar control-following performance can differ in state consistency and content causality.Source: Table 4 · p. 9, §3.3 · pp. 9–10
  • W3 tests whether layout, appearance, temporal continuity, state, and identity persist during extended interaction and revisits; W2 and W3 use different case pools.Thus aggregate W2–W3 score differences should not be interpreted as a pure duration effect.Source: Table 5 · p. 10, §3.3 · p. 10
  • W4 evaluates whether natural-language interventions are executed, preserve unrelated content, and govern subsequent world evolution rather than causing only transient visual changes.Causal evaluation uses the edited rule as the reference for later behavior, such as motion after a gravity intervention.Source: Table 6 · p. 10, §3.3 · p. 10

4 Spatial World Model Track

The spatial track evaluates whether renderable environments support exploration and physical interaction while preserving consistency through edits and expansion; its static-scene benchmark reveals capability-specific reliability gaps beyond visual quality.

4.1 Data Construction

  • The final dataset contains 300 scenes: 266 snapshots and 34 expansion cases, with 424 annotated objects, support surfaces in 72 scenes, and 30 edit pairs.Those edit pairs cover 18 object-level and 12 global edits; scenes and annotations are selected for task-specific needs.Source: §4.1 · pp. 11–12

4.2 Evaluation Metrics

  • The track scores W1–W5, leaving W6 unscored, using rendered camera trajectories alongside geometry measurements and physical simulation.Its measures cover visual construction, navigation and support, scene and object persistence, editing with preservation, and spatial expansion with regional preservation.Source: §4.2 · p. 12

4.3 Experimental Results

  • Marble leads capability-adjusted Arena Elo at 1308, followed by GPT-6-Astra at 1252 and HYWorld-2.0 at 1230; GPT-6-Astra leads W3–W5 despite lower W1 performance.Marble and HYWorld-1.0 achieve the highest W1 scores, while HYWorld-2.0 leads W2; FlashWorld and Lyra variants score only 0.00–1.36 on W2.Source: Table 7 · p. 13, §4.3 · p. 13
  • Marble and HYWorld-1.0 achieve similar leading W1 scores, while Marble leads Q-Align, HPS, and VLM and GPT-6-Astra achieves the highest TOPIQ score.HYWorld-1.0 lifts a generated panorama into a mesh; GPT-6-Astra constructs meshes directly, with lower PI and HPS scores than its TOPIQ result.Source: Table 8 · p. 13, §4.3 · p. 13
  • HYWorld-2.0 has the highest placement rate at 70.14%, while WorldGen approaches the highest navigable ratio but trails HYWorld-2.0 substantially in placement.FlashWorld and Lyra variants provide almost no navigable area and no successful placements under this protocol.Source: Table 9 · p. 14, §4.3 · p. 14
  • GPT-6-Astra leads both scene-level persistence metrics and all three object composites, whereas Matrix-3D’s stable brightness and hue mask its lowest Track F1 and VLM F1.This separates scene appearance stability from whether objects remain identifiable and intact across viewpoints.Source: Table 10 · p. 15, §4.3 · p. 15
  • HYWorld-2.0 leads edit success, but GPT-6-Astra leads preservation and both joint edit scores; GPT-6-Astra also leads spatial growth, junction improvement, and both preservation scores.Matrix-3D has a higher area score than HYWorld-2.0 and Marble but the lowest new-region quality and old-region preservation, while Marble has the highest-quality new views.Source: Table 11 · p. 15, §4.3 · pp. 15–16, Table 12 · p. 15

5 Embodied World Model Track

HappyWorld-Bench evaluates embodied video models on action-conditioned world evolution, separating visual perception from consistency, causality, and controllability. Results show strong visual fidelity but persistent failures in action grounding, state persistence, and response to changed conditions.

5.2 Evaluation Metrics

  • Four separate dimensions—perception, consistency, causality, and controllability—distinguish recognizable scenes, persistent state, plausible causes, and faithful action execution.The framework evaluates action-conditioned egocentric visual predictions and supports a future extension to discrete robot actions.Source: §5 · p. 16

5 Embodied World Model Track

  • Embodied evaluation spans W2 atomic actions, W3 persistent multi-step rollouts, and W4 paired interventions that change an action condition or physical rule.The current benchmark operationalizes W2–W4; W1, W5, and W6 remain framework-level definitions.Source: §5 · p. 16, §5.1 · p. 16

5.1 Data Construction

  • The formal set contains 254 cases: 62 W2, 100 W3, and 92 W4, including 40 action-condition and 52 physical-rule pairs.W4 branches share an initial frame to isolate the effect of the intervention.Source: §5.1 · p. 17

5.3 Experimental Results

  • MiniMax-H3 leads overall capability at W2–W4 and Arena ELO, while Cosmos 3 and Sora 2 remain below 75 at every level.Arena ELO broadly follows the automatic evaluation, with differences mainly among closely performing models.Source: §5.3 · p. 18, Table 14 · p. 18
  • At W2, models often imitate requested motions but fail to establish intended contact, affect the correct object, or reach the prescribed terminal state.Representative failures include grasping beside a target, acting on the wrong object, and introducing uncaused scene or viewpoint changes.Source: §5.3 · p. 19, Figure 7 · p. 19
  • At W3, consistency degrades over longer action chains when initial or intermediate states fail to transfer, preventing accumulated effects from reaching the required terminal state.Perception remains largely intact while later stages lose the state preconditions established by earlier actions.Source: §5.3 · p. 20, Figure 8 · p. 21
  • At W4, physical-rule edits often leave dynamics unchanged, while action edits may alter outcomes without faithfully changing the prescribed manner of interaction.Across W2–W4, the gap between generally strong visual fidelity and faithful action-conditioned dynamics becomes more pronounced as interaction requirements increase.Source: §5.3 · pp. 20–22

6 Conclusion · A Video World Model Track

HappyWorld-Bench evaluates video, spatial, and embodied world models within a shared W1–W6 framework, assessing generated-world quality alongside reliability under interaction, persistent state changes, and intervention. Across the tracks, it combines automated capability metrics with human-comparison Elo ratings to support shared requirements for dependable world models.

6 Conclusion

  • HappyWorld-Bench brings video, spatial, and embodied world models into a shared W1–W6 capability framework.Its central premise is to evaluate both generated-world quality and reliability under interaction, persistent state changes, and intervention.Source: §6 · p. 22
  • Across all three tracks, HappyWorld-Arena derives model-level Elo ratings from human A/B comparisons to complement automated capability metrics.The benchmark combines a common capability vocabulary with automated evaluation and human preference assessments.Source: §6 · p. 22
  • The shared framework is intended to help video, spatial, and embodied world-model research converge on common requirements for useful systems.The authors hope it will guide integration of advances in generation, simulation, and embodied prediction.Source: §6 · p. 22
  • The intended outcome is systems that support dependable exploration, planning, and action in complex environments.This goal follows from evaluating world models for reliability under interaction and intervention, not only generated-world quality.Source: §6 · p. 22

A.1 Inference Protocols and Model Configurations

The video-track protocols translate benchmark cases into model-specific image, text, and control inputs, with differing rollout procedures and interface limitations. Those differences shape which actions and interventions each evaluated configuration can execute.

Genie 3 via Project Genie

  • Genie 3 is evaluated through Project Genie with browser-scheduled controls, opaque default generation settings, and support limited to W1–W3 cases.Controls run in wall-clock time, so prescribed durations need not match encoded video duration; unsupported intervention or event cases are recorded as unsupported rather than approximated. Results characterize the deployed Genie 3-based system, not an isolated checkpoint.Source: §A.1 · p. 31

HappyOyster

  • HappyOyster uses one shared configuration across cases, causal generation over temporal segments, and joint frame-aligned controls when actions are specified.Its canvas preserves reference aspect ratio under an approximately 832 × 480-pixel budget; cases without action sequences use no-input rollouts.Source: §A.1 · pp. 31–32

Trajectory-conditioned configurations

Action-conditioned configurations

Yume-1.5

  • Yume-1.5 expresses controls as language tokens in 29-frame chunks and does not evaluate cases without discrete controls or natural-language interventions.Its descriptions are refined once by an InternVL captioner (Chen et al., 2024), and each chunk is conditioned on preceding frames; timed W4 interventions are not implemented.Source: §A.1 · pp. 32–33

A.2 Detailed Metrics

Video metrics assess four dimensions of world-model behavior, with normalized-score and aggregation conventions defining how results are interpreted.

A.2 Detailed Metrics

  • Video evaluation covers perception and representation, consistency and state retention, causality and causal rollout, and controllable interaction.Normalized scores lie in [0, 1], percentage scores equal 100 times normalized values, and higher is better; inapplicable cases and missing measurements are excluded rather than scored as zero. Frames are denoted I_t and cos(x, y) denotes cosine similarity.Source: §A.2 · p. 35

A.2.1 Perception and Representation … A.2.4 Controllable Interaction and Counterfactuals

The benchmark’s video metrics separate visual quality and stability from state consistency, causal correctness, and action responsiveness. Together, they assess both what remains stable and whether changes follow instructions, causes, and interventions.

A.2.1 Perception and Representation

  • VQ measures frame-level technical quality, while HPS estimates broader visual preference; neither establishes temporal, semantic, physical, or instruction-following correctness.HPS emphasizes visual appeal, composition, naturalness, and acceptability, whereas MUSIQ emphasizes technical quality.Source: §A.2.1 · p. 35
  • Brightness consistency, color-temperature consistency, and sharpness retention form IS, tracking exposure, hue, and edge changes; coarse brightness histograms ignore layout and may penalize requested lighting transitions.Sharpness retention uses a cumulative noise gate, and IS averages the three normalized components.Source: §A.2.1 · pp. 35–36
  • Dyn measures visible motion magnitude rather than whether motion is commanded or smooth, while IF checks prompt requirements and evaluates W4 initial conditions before intervention.For W4, an explicitly requested later attribute change is not an IF failure.Source: §A.2.1 · p. 36

A.2.2 Consistency and State Retention

  • Background consistency combines adjacent- and first-frame similarity with subjects masked, while GC measures depth-reprojection geometry and TC measures aligned surface appearance.The finalized masked-background version is not numerically comparable to standard full-frame CLIP background consistency.Source: §A.2.2 · pp. 36–37
  • SC evaluates persistence of object states and justified changes, while SuC tracks preservation of subject identity and appearance; SC scores must not be mixed with WorldMark’s different procedure.SuC combines adjacent-frame DINOv2 identity continuity with long-range CLIP similarity to the first frame.Source: §A.2.2 · p. 37

A.2.3 Causality and Causal Rollout

  • PC scores visibly assessable physical processes, CC checks trigger-to-consequence order and persistence, and IP checks relevant interactions for penetration or invalid contact.IP is a visibility-based judgment, not a complete 3D collision simulation; invisible prerequisites may be inapplicable for CC, but visible incorrect behavior remains applicable.Source: §A.2.3 · p. 37

A.2.4 Controllable Interaction and Counterfactuals

  • For W2–W3, TA evaluates commanded translation direction against estimated camera motion, while AE checks visible action completion in the correct temporal segments.Zero motion counts as incorrect for TA; AE counts missing targets, absent actors, incorrect actions, and unmet target states as failures.Source: §A.2.4 · p. 38
  • For W4, IV assesses whether interventions follow physically and causally admissible processes, while SF separately measures target achievement, persistence, and preservation of untouched content.Subclass-balanced SF aggregation prevents numerous achievement items from overwhelming persistence or preservation.Source: §A.2.4 · p. 38

B Spatial World Model Track: Detailed Metrics

The appendix defines the Spatial World Model Track’s metrics, evaluation procedures, and aggregation rules, including which metrics are retained and how capabilities are scored.

B Spatial World Model Track: Detailed Metrics

  • The track defines 39 quantities and retains 31 via F1 composites within consistency–coverage families; it scores W1–W5, leaves W6 unscored, and assigns operation-specific preservation to W4/W5 rather than W3.Metrics cover observable quality, physical usability, scene- and object-level consistency, controlled editing, and expansion and preservation.Source: B Spatial World Model Track: Detailed… · p. 38

B.1 Shared Notation, Actions, and Evaluation Inputs … B.4 W3: State Persistence

The appendix defines shared rendering, geometry, and camera conventions, then specifies metrics for generative appearance, interactive usability, and cross-view state persistence.

B.1 Shared Notation, Actions, and Evaluation Inputs

  • The shared camera convention exposes observer pose as part of the state, while prescribed-pose rendering does not independently measure pose-estimation error, action latency, or learned physical evolution.Action timestamps, requested and realized poses, and controller settings are needed to distinguish camera-interface behavior from scene limitations.Source: §B.1 · p. 39
  • Navigation preprocessing crops and cleans geometry, transforms it to simulator coordinates, and estimates XY footprints on a 0.05 m grid using one original scale and grid origin for before/after operations.Source: §B.1 · p. 39

B.2 W1: Generative Construction

  • W1 evaluates rendered appearance and conditioning alignment with ten prescribed origin-view metrics, including learned TOPIQ quality regression.TOPIQ is a no-reference quality regressor whose outputs are averaged across preprocessed frames.Source: §B.2 · p. 40
  • W1 object coverage measures whether each requested category is detected in any of six origin probes, assessing named-category presence rather than exact instance counts.An empty annotation is inapplicable, while an annotated but absent target counts as a failure.Source: §B.2 · p. 41

B.3 W2: Interactive Simulation

  • W2 navigation measures the largest four-connected NavMesh component’s area relative to the scene footprint, with corner-touching cells treated as disconnected.Vertically overlapping surfaces share horizontal grid cells; a valid scene without a navigable component receives a zero numerator.Source: §B.3 · p. 41
  • W2 tabletop readiness tests whether a plate remains on annotated support surfaces after independent releases at selected placement locations.Trials fail under specified penetration, support-boundary, or final-height conditions; unusable collision geometry also fails.Source: §B.3 · p. 42

B.4 W3: State Persistence

  • W3 evaluates observations across camera motion and viewpoint changes using 15 consistency quantities: six scene-level and nine object-level measurements.Source: §B.4 · p. 42

B.4.1 Scene-Level (Place) Metrics

The six quantities measure spatial validity, short-range visual continuity, and location robustness within W3 place-level consistency.

B.4.1 Scene-Level (Place) Metrics

  • M12 uses six-face probes and depth clearance as a depth-and-coverage proxy for invalid camera positions.When metric scale is available, the clearance test fails if more than half of valid-depth pixels are closer than 0.10 m; without a filled footprint, the outside-scene test uses the hole-rate threshold alone.Source: §B.4.1 · p. 42
  • Coarse histogram continuity can remain high despite object rearrangement; the described hue-histogram procedure converts RGB frames to OpenCV HSV’s 0–179 hue convention and measures palette continuity, not kelvin temperature.Source: §B.4.1 · pp. 42–43
  • M15 is the population standard deviation of HPSv3 ring means at successfully scored accepted locations plus the origin; rejected candidates are excluded, and a scored origin and at least one scored additional location are required.M38 has eight location slots, two at each of four radii; the origin is not a ninth recall slot.Source: §B.4.1 · p. 43
  • M39 combines inverse HPSv3 dispersion with reported recall as a harmonic consistency–coverage score, but cannot distinguish consistently good from consistently poor images.Low dispersion with low recall is penalized; zero reported recall yields a zero composite, while missing recall yields a missing composite.Source: §B.4.1 · p. 43

B.4.2 Object-Level Metrics

B.4.2 defines object-level measures from fixed 24-view orbits, combining detected or judged visibility with perceptual quality, tracking, and visible-integrity assessments. The measures are proxies with specific limitations, including endpoint-only tracking and benchmark-specific visual rubrics.

B.4.2 Object-Level Metrics

  • Fixed-radius 24-view orbits are used without camera filtering or radius search, so missing observations may reflect geometry, occlusion, camera placement, or segmentation.The subgroup tests W3 observation persistence and return correspondence.Source: §B.4.2 · p. 44
  • obj_sam_recall measures detected-view coverage, while obj_hpsv3_std measures perceptual-score dispersion across valid masked object crops; their composite is not true-positive precision and depends on frozen HPSv3.Masks under 64 pixels are excluded; dispersion is undefined when fewer than two valid crops have scores, and the proxy may reward uniformly low quality.Source: §B.4.2 · p. 44
  • obj_track_accuracy tracks 64 sampled target points from a visible start and compares only repeated-start endpoints, so intermediate drift can be concealed rather than testing full-trajectory stationarity.The composite combines endpoint accuracy and SAM view coverage, with zero recall contributing zero and positive recall without track accuracy excluded.Source: §B.4.2 · p. 44
  • obj_vlm_recall counts views judged to show the target present and unoccluded from outside the scene; unlike SAM recall, it has no two-view minimum and is not ground-truth detector recall.The Qwen3-VL prompt and parser are frozen.Source: §B.4.2 · p. 45
  • obj_vlm_artifact_free and obj_vlm_completeness assess visible defects and missing or damaged solid parts on valid views, using visual judgments rather than reference-scan surface recall.Empty valid-view sets are undefined; conditional quality components are combined with valid-view coverage, and the rubric and harmonic composite are benchmark-specific.Source: §B.4.2 · p. 45
  • The procedure renders the fixed orbit, computes SAM/HPS and tracking measures when evaluable, judges view validity and integrity, and stores all nine outputs with annotations and error states.It sets the tracking component to zero under the zero-recall rule and the VLM composite to zero when no view is valid.Source: §B.4.2 · p. 45

B.5 W4: Programmable Dynamics · B.6 W5: Scalable Shared World

The W4 and W5 evaluations test restricted edit behavior and world expansion through paired before/after views, with explicit limits on what their metrics establish. W4 measures edit success and preservation, while W5 measures spatial growth, coverage, and view quality.

B.5 W4: Programmable Dynamics

  • W4 evaluates a restricted intervention interface with operation-specific allowed-change masks; physical-law, causal-mechanism, behavior, and task-constraint programming require separate verification.For addition, deletion, replacement, and modification, the mask uses the corresponding after, before, before/after union, and before/after target regions; evaluation falls back to the whole image if no target mask is generated.Source: §B.5 · p. 45
  • W4 edit success is the mean of per-case indicators that any selected after-view satisfies the edit instruction.Object edits use up to six detectable orbit views after six origin probes; global edits use six origin views. Before judgments do not gate improvement, so an already-satisfied no-op can pass.Source: §B.5 · p. 45
  • W4 reports object-edit and global-edit joint scores separately, using different preservation functions and case populations.The object score is zero when edit success fails; with success, unavailable preservation yields a missing joint score. The global score combines edit success with depth agreement over common-valid geometry and inherits the no-op limitation.Source: §B.5 · pp. 46–47

B.6 W5: Scalable Shared World

  • For W5 expansion, paired before/after worlds use common-frame footprints and identical origin, junction, and connecting-path cameras; sampling is centered on newly occupied cells.The expansion camera height is the expanded ground height at the area-weighted centroid of new cells plus the original camera height.Source: §B.6 · p. 47
  • W5 footprint area ratio compares after-area with before-area, while junction coverage gain measures paired opacity-coverage change under shared junction cameras.The area ratio is averaged across pairs, not computed from pooled areas; it is not the new-cell area divided by the old footprint, and loss of old cells can coexist with growth.Source: §B.6 · p. 47
  • W5 junction HPSv3 scores are absolute quality at proxy views, whereas origin and path HPSv3 metrics compare paired after-minus-before view-quality differences.The path metric samples 16 frames between the original camera and the center of the newly occupied region; equal quality rewards do not prove unchanged geometry or identity.Source: §B.6 · pp. 47–48
  • W5 expansion tables report normalized scores rather than raw area ratios or signed HPS changes.The normalization mappings are specified in Section B.8.Source: §B.6 · p. 48

B.7 Aggregation and Reproducibility · B.8 Normalized World-Level Summary

These sections specify how scores and missing cases are aggregated, how model-level metrics and Arena ratings are normalized, and how the resulting scores should be interpreted.

B.7 Aggregation and Reproducibility

  • Aggregation proceeds from frames or trials to object or case components, then composites, then model means; metric-specific valid counts and failure statuses remain distinct.The nominal 266 snapshot scenes, 424 annotated objects, and 72 support scenes are not interchangeable denominators, and N/A is distinct from a measured zero.Source: §B.7 · p. 48
  • The implementation fixes evaluation settings for standard deviation, histograms, navigation grids, placement, object quality, editing, and expansion sampling.Per-case reports retain component scores, valid counts, and failure statuses.Source: §B.7 · p. 48
  • Reported results are descriptive model averages of scene quality and specified spatial operations, not direct measurements of downstream agent task success.Source: §B.7 · p. 48

B.8 Normalized World-Level Summary

  • Raw model-level metric means are mapped to [0, 1] and multiplied by 100 for display, while Arena Elo ratings retain their original scale.The paper tables display metric values to two decimal places.Source: §B.8 · p. 48
  • Arena Elo combines human pairwise votes with deterministic capability-support comparisons, assigning supported models wins over unsupported ones and “both bad” when neither supports the capability.The ratings use Bradley–Terry maximum likelihood estimation, normalize the mean to 1000, and give added comparisons the same weight as human votes.Source: §B.8 · p. 48
  • F1-only selection leaves 10, 2, 7, 7, and 5 retained metrics for W1–W5; group means weight metrics equally regardless of valid sample counts.FlashWorld uses nine W1 metrics because it lacks PI; empty groups are shown as a dash, W6 is unscored, and explicit failure scores remain included.Source: §B.8 · pp. 48–49
  • Fixed transforms map raw metric summaries to [0, 1] and clip outputs to that range; selected metrics use frozen reference-percentile bounds that are not refitted when methods are added.The frozen constants use a baseline-by-case reference distribution for the 0827_validated_eval set.Source: §B.8 · p. 49
  • W1–W5 averages summarize different metric groups and are not directly comparable; a displayed value of 50 is neutral for junction coverage gain and the two HPSv3 change scores.Expansion area scores of zero include unchanged or reduced footprint area; positive values indicate increased raw area. HPSv3 preservation measures view-quality changes, not unchanged geometry or identity.Source: §B.8 · p. 49

C Embodied World Model Track: Detailed Metrics … C.6 Metric Scoring and Overall Aggregation

The embodied track defines a 17-metric scoring system across perception, consistency, causality, and controllability. Task-specific applicability and weighted aggregation produce dimension scores and an overall score that assigns the greatest weights to causality and controllability.

C Embodied World Model Track: Detailed Metrics

  • The embodied track scores 17 metrics: four for perception, five for consistency, three for causality, and five for controllability, with all scores normalized to [0, 100].Higher scores indicate better performance.Source: C Embodied World Model Track: Detailed… · p. 50

C.1 Metric Inventory and Applicability

  • Table 21 summarizes each metric’s evidence and applicability across W2–W4.Source: §C.1 · p. 50

C.2 Perception

  • Perception metrics assess scene and robot-subject fidelity, frame-level visual quality, and sharpness retention throughout generated rollouts.Scene Fidelity and Subject Fidelity use reference-image assertions; Perceptual Quality evaluates sampled frames, while Sharpness Retention tracks sharpness relative to the first generated frame.Source: §C.2 · pp. 50–51

C.3 Consistency

  • Consistency metrics assess subject appearance (SAC), static geometry (GC), persistent scene and object states (SSC), temporal coherence (TC), and motion smoothness (MS).SAC combines feature similarity with valid-mask coverage, lowering scores when the subject cannot be localized in many frames; GC excludes entities expected to change, while SSC checks task-specific assertions at fixed checkpoints.Source: §C.3 · pp. 51–52

C.4 Causality

  • Causality metrics score physical plausibility, cause–effect temporal order, and whether salient state changes have their specified causal triggers.Assertions may concern task-specific physical constraints; order examples include grasp before transport, and trigger checks include displacement caused by contact.Source: §C.4 · p. 52

C.5 Controllability

  • Controllability metrics assess goal-state achievement, atomic action compliance, functional interaction success, multi-action ordering, and fidelity to altered action or physical-rule conditions.Condition-Branch Fidelity evaluates each branch’s action or condition and expected outcome, then whether paired outcomes exhibit the required difference; variants can change direction, target, action type, or physical rules.Source: §C.5 · pp. 52–53

C.6 Metric Scoring and Overall Aggregation

  • Soverall = 0.20Sperc + 0.20Scons + 0.30Scaus + 0.30Sctrl.Applicable metrics are averaged within each dimension, inapplicable metrics and assertions are omitted, and predefined importance weights combine assertion judgments.Source: §C.6 · p. 53

Related papers

Loading 2609.24308v1…