PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity

Yuxue Yang, Shuyao Shang, Jiahe Wang, Zitong Zhou, Liang Tan, Junhan Zeng, Ruizhi Li, Junyan Li, Yu Liu, Xiao Yang, Yong Li, Jun Zhu, Hongsheng Li, Tieniu Tan, Lue Fan, Zhaoxiang Zhang

arXiv:2608.02603v1cs.CV

TL;DR

Existing benchmarks mainly test visual quality and explicit instruction fulfillment, leaving world models’ ability to infer unstated, scene-conditioned consequences underexamined. WorldExam evaluates this capability alongside three other levels and finds paradigm-dependent trade-offs, with no model consistently strong across broad task coverage.

  • Problem

    Existing benchmarks mainly assess visual quality or specified instruction outcomes, leaving models’ ability to infer unstated consequences implied by the initial scene underexamined.

  • Method

    WorldExam is a hierarchical diagnostic benchmark with four evaluation levels, eight tasks, and unified testing of camera-, action-, and language-driven models.

  • Results

    Evaluation reveals paradigm-dependent trade-offs: camera-driven models excel at camera control, action-driven models control subjects more precisely, and language-driven models perform better on interaction.

  • Takeaways & Limitations

    No model combines broad task coverage with consistently strong performance, so high visual quality and explicit instruction fulfillment do not guarantee inherent reactivity.

  • Takeaways & Limitations

    Strong visual quality, control adherence, and spatial consistency do not guarantee successful scene-conditioned reactions or goal execution.

Abstract

from arXiv · show

Controllable video generation models are increasingly being developed as world models. Accordingly, evaluating them in this role extends beyond the apparent appearance of generated videos to the inherent reactivity of the worlds they depict: the ability to infer from the scene state how the world should react and to generate plausible consequences not explicitly described in the input. Yet existing benchmarks mainly assess visual quality or explicit instruction fulfillment by checking whether requested actions and interaction outcomes are realized, leaving inherent reactivity underexamined. We introduce WorldExam, a hierarchical diagnostic benchmark spanning four levels: Visual Quality, Control Adherence, Spatial Consistency, and World Reactivity. It comprises 1,474 cases across eight dedicated tasks and supports unified evaluation of camera-, action-, and language-driven model paradigms. The World Reactivity level evaluates scene-conditioned reactions and goal-directed behaviors beyond what is explicitly specified in the input. Evaluation of 20 representative models reveals a clear capability split. Camera-driven models excel at camera control, but their interfaces do not support dynamic interaction; action-driven models control subjects more precisely but often leave the world unresponsive; and language-driven models perform better on interaction but follow complex controls less faithfully. No model combines broad task coverage with consistently strong performance, showing that high visual quality and explicit instruction fulfillment do not guarantee inherent reactivity.

1 Introduction

WorldExam evaluates controllable video models not only by visual quality and explicit instruction fulfillment, but also by inherent reactivity: generating plausible scene-conditioned consequences left unstated in the input. Its hierarchical benchmark spans four diagnostic levels and eight tasks across camera-, action-, and language-driven interfaces, revealing capability splits without a universally strong model.

  • Motivation: WorldExam targets inherent reactivity by testing whether models infer and generate plausible scene-conditioned consequences beyond explicitly specified controls or goals.Examples include terrain-adapted motion, obstacle contact or avoidance, and plausible responses from other agents.
  • Benchmark: WorldExam comprises 1,474 cases across four diagnostic levels and eight tasks, supporting unified evaluation of camera-, action-, and language-driven model paradigms.The levels are Visual Quality, Control Adherence, Spatial Consistency, and World Reactivity.
  • Benchmark Design: WorldExam adapts atomic control units to each model’s native interface while leaving expected scene-conditioned reactions unstated in World Reactivity cases.Interfaces use SE(3) camera trajectories, discrete action sequences, or natural-language prompts, depending on the paradigm.
  • Evaluation Protocol: Separate static-scene and dynamic-interaction tracks prevent unsupported capabilities and differing scene assumptions from being treated as failures or averaged together.The static-scene track is available to all paradigms, whereas dynamic interaction is evaluated only on compatible action- and language-driven models.
  • Findings: 20 representative models exhibit paradigm-dependent trade-offs: camera-driven models excel at camera control, action-driven models control subjects more precisely, and language-driven models perform better on interaction tasks.Camera-driven interfaces do not support dynamic interaction; action-driven models often leave the world unresponsive; language-driven models follow complex controls less faithfully.
  • Findings: No model combines broad task coverage with consistently strong performance, so high visual quality and explicit instruction fulfillment do not guarantee inherent reactivity.Aggregate scores can obscure the capability splits across paradigms and diagnostic levels.

2 Related Work

Video world models are progressing from short open-loop synthesis toward controllable, persistent, interactive simulation across camera-, action-, and language-driven paradigms. Existing benchmarks assess complementary capabilities, while WorldExam addresses inherent reactivity through native-interface controls, scene-conditioned reactions, and goal-directed behaviors beyond explicit inputs.

  • World Model Paradigms: Video world models are broadly grouped into camera-, action-, and language-driven paradigms based on their primary control interfaces.These models support controllable generation for gaming, robotics, embodied AI, and open-world simulation.
  • World Model Paradigms: Across paradigms, video world models are evolving from short open-loop synthesis toward controllable, persistent, and interactive environment simulation.Heterogeneous interfaces complicate direct comparison, while controllability, long-term memory, and inherent reactivity remain key challenges.
  • Related Benchmarks: Existing benchmarks span perceptual and temporal quality, compositionality, world knowledge, implicit rules, future-state reasoning, physics, geometry, generalization, and embodied action fidelity.This benchmark landscape evaluates complementary aspects of video world modeling rather than a single unified capability.
  • Related Benchmarks: General benchmarks improve comparison through camera-trajectory control, action control, revisit consistency, and standardized or unified action representations.Examples include WorldScore, MIND, WorldMark, and iWorld-Bench, while Omni-WorldBench evaluates prompt-specified interaction.
  • WorldExam’s Distinction: WorldExam adapts atomic control units to each model’s native interface and evaluates inherent reactivity through scene-conditioned reactions and goal-directed behaviors beyond explicit inputs.This design targets reactions that are not explicitly specified in the input.

3 WorldExam

WorldExam provides a unified, four-level diagnostic framework for evaluating video world models across camera-, action-, and language-driven interfaces. It isolates World Reactivity—the ability to infer scene-conditioned consequences and goal-directed behavior beyond explicit input—from visual quality, control, and spatial consistency.

  • Unified evaluation: WorldExam evaluates camera-, action-, and language-driven models by adapting shared control cases to camera trajectories, discrete actions, or natural-language prompts.The benchmark represents controllable behavior as ordered compositions of atomic control units and maps each shared case to the model’s native interface.
  • Four-level hierarchy: The diagnostic hierarchy comprises Visual Quality, Control Adherence, Spatial Consistency, and World Reactivity, progressing from apparent appearance to inferred behavior.Visual Quality measures perceptual plausibility, temporal stability, and aesthetics; Control Adherence measures following input controls; Spatial Consistency measures coherence after viewpoint revisits.
  • World Reactivity: World Reactivity evaluates scene-conditioned reactions and goal-directed behaviors that are not explicitly specified in the input.Reaction cases specify an initiating behavior while requiring the model to infer consequences from terrain, obstacles, nearby agents, objects, or physical processes.
  • Diagnostic separation: Strong Visual Quality, Control Adherence, and Spatial Consistency do not guarantee successful scene-conditioned reactions or goal execution.WorldExam therefore reports the four levels separately to localize failures in generation quality, explicit control, spatial persistence, and inferred behavior.
  • Evaluation tracks: The benchmark reports separate static-scene and dynamic-interaction tracks so models are not penalized for tasks unsupported by their interfaces.The static-scene track contains Camera Control and Scene Revisit for all paradigms, while the dynamic-interaction track contains Subject Control and five World Reactivity tasks; Goal Completion is language-only.

4 Evaluation Protocol and Metrics

WorldExam evaluates generated videos through reconstructed 3D geometry and GPT-5.5 VLM judgments, with task-specific metrics for camera control, scene revisit, and general consistency. Increasing image-space displacement makes controlled, spatially consistent generation harder, reducing Camera Control, Scene Revisit, and general-metric performance.

  • Evaluation framework: WorldExam lifts Camera Control, Scene Revisit, Subject Control, and Terrain Interaction videos into 3D for geometry-based evaluation, while four remaining tasks use GPT-5.5 as a VLM judge.Camera- and scene-revisit metrics use recovered camera trajectories; subject and terrain tasks use recovered 3D trajectories and terrain geometry.
  • Camera Control: Camera Control reports Scam = 100√stsr after normalizing translation and rotation errors against prescribed motion directions and temporal progression.The normalized terms are st = max(0, 1 − et/0.5) and sr = max(0, 1 − er/15), with rotation error measured in degrees.
  • Scene Revisit: Scene Revisit combines binary return-to-pose success with PSNR, LPIPS, and SSIM, producing a high score only when the camera returns and the scene view remains consistent.Translation revisits require minimum distance within 10% of outgoing displacement, while rotation revisits require minimum angular distance below 5°.
  • General metrics: General evaluation reports five task-agnostic metrics across static-scene videos, including 3D Consistency, Photometric Consistency, and Temporal Flickering.3D Consistency uses cycle reprojection error from recovered geometry, while Photometric Consistency uses forward-backward optical-flow cycle error measured by AEPE.
  • Displacement calibration: Increasing input translation multipliers to create larger image-space displacements causes declines in Camera Control, Scene Revisit, and general-metric performance.WorldExam calibrates camera-driven models because identical SE(3) translation values can produce different image-space displacements across pose-conditioning interfaces.

5 Experiments

Experiments on 20 video world models show a clear capability split across camera control, subject control, spatial consistency, and world reactivity. Diagnostic task scores reveal that visual quality and explicit control adherence do not reliably predict scene-conditioned interaction or goal-directed behavior.

  • Evaluation setup: 20 models are evaluated across camera-driven, action-driven, and language-driven paradigms, with dynamic interaction tested only for compatible action- and language-driven models.All models are evaluated on the static-scene track; the dynamic-interaction track includes WorldPlay, LingBot-World, and all seven language-driven models.
  • Camera control and spatial consistency: NeoVerse leads Camera Control at 97.33 and Scene Revisit at 89.25, while WorldPlay leads action-driven models at 92.74 Camera Control and 72.51 Scene Revisit.NeoVerse and InSpatio-World both achieve 1.000 Revisit Success, whereas WorldPlay and Hailuo 2.3 achieve 0.790 and 0.505, respectively.
  • Camera control and spatial consistency: 0.10× to 2.00× translation scaling lowers NeoVerse’s Camera Control from 98.25 to 95.32, Scene Revisit from 90.98 to 88.37, and General average from 80.42 to 75.05.Photometric Consistency falls from 80.17 to 62.45 as image-space displacements increase.
  • World reactivity: Action-driven models provide stronger Subject Control, with LingBot-World scoring 55.47 and WorldPlay 49.75 versus Veo 3.1’s best language-driven score of 37.28.Failures often convert requested subject motion into camera motion or leave the scene static.
  • World reactivity: Language-driven models lead World Reactivity tasks: Vidu Q3 and Hailuo 2.3 score 64.39 and 61.57 on Terrain Interaction, while Veo 3.1 scores 75.96 on Object Interaction and 85.10 on Social Interaction.Hailuo 2.3 leads Physical Reaction at 63.84, while HappyHorse 1.0 and Veo 3.1 lead Goal Completion at 85.33 and 85.30.
  • Capability split: Language-driven General averages span only 79.64–81.04, while Task averages span 39.85–65.02, showing that visual quality or control adherence does not guarantee World Reactivity.Camera-driven models have the strongest camera control and scene revisiting but do not support dynamic interaction; action-driven models control subjects more precisely without consistently inducing scene-conditioned reactions.

6 Conclusion · Appendix

WorldExam provides a unified hierarchical benchmark for evaluating video world models beyond visual quality and explicit instruction fulfillment, including scene-conditioned reactivity. Results across 20 models reveal a capability split among camera-, action-, and language-driven systems, while robustness checks support the findings and scope limitations remain.

  • 6 Conclusion: WorldExam diagnoses video world models beyond visual quality and explicit instruction fulfillment.The benchmark is presented as a unified hierarchical framework.
  • 6 Conclusion: WorldExam distinguishes explicit control fulfillment from inferred scene-conditioned reactions through four diagnostic levels, eight tasks, and 1,474 test cases.It adapts shared cases to camera-, action-, and language-driven model interfaces.
  • 6 Conclusion: 20 representative models reveal a capability split: camera-driven models excel at camera control, action-driven models control subjects precisely, and language-driven models perform better on interaction.Action-driven models often leave environmental elements unresponsive, while language-driven models follow composed controls less faithfully.
  • 6 Conclusion: No evaluated model combines broad task coverage with consistently strong performance.This result separates high visual quality or explicit instruction fulfillment from comprehensive inherent reactivity.
  • Appendix: Appendix evaluations recompute static-scene metrics with DA3 while keeping benchmark inputs and model outputs fixed, and recompute selected dynamic-interaction metrics with unchanged checklist tasks.Aggregation and highlights follow the corresponding main-table procedures.
  • 6 Conclusion: Alternative reconstruction-backend evaluations preserve stable model rankings, supporting the reliability of the findings.The conclusion also references VLM checklist scores as part of this support.
  • 6 Conclusion: The benchmark scope is bounded by available model interfaces: dynamic interaction requires reliable third-person subject control, and Goal Completion is limited to language-driven models.Its metrics evaluate observable end-to-end behavior rather than locating reasoning or proving an internal causal representation in the generator.

A Gallery

The WorldExam gallery showcases diverse test cases across viewpoints, subject types, scene settings, and terrain. This breadth spans both first- and third-person perspectives and indoor and outdoor environments.

  • Test cases span first- and third-person viewpoints.
  • The gallery includes human, animal, vehicle, and robot subjects.
  • Cases cover diverse indoor and outdoor scenes and a range of terrain types.

B Per-Model Inference Settings

The benchmark preserves each model’s native inference settings where applicable, while constraining streaming outputs to 100–200 frames. Dynamic-interaction eligibility depends on reliable third-person subject control, which excludes most action-driven models and all camera-driven interfaces.

  • Inference settings: Models use their default resolution, video length, and other inference settings whenever applicable.This preserves each model’s native performance during evaluation.
  • Inference settings: 100–200 frames are used for streaming models with flexible generation lengths to limit quality drift in excessively long outputs.Table 8 reports the resulting resolution and frame count for each model.
  • Dynamic-interaction eligibility: Only WorldPlay [38] and LingBot-World [40] qualify among action-driven models because dynamic interaction requires reliable control of a visible third-person subject.The other five action-driven models lack third-person subject control or cannot provide it reliably.
  • Dynamic-interaction eligibility: All evaluated language-driven models accept third-person subject-motion prompts, whereas camera-driven interfaces control only the camera.Goal Completion is language-only within the dynamic-interaction track.

C Qualitative Examples of the Eight Tasks

The qualitative examples present all eight tasks in a shared visual format, pairing an initial frame with four generated frames and the corresponding control intent or goal. For checklist-based tasks, the checklist explains evaluation but is withheld from the model input.

  • Shared Visual Format: All eight tasks use a shared format with one initial frame, four temporally ordered generated frames, and a bottom-panel control intent or high-level goal.The initial frame is highlighted, while the generated frames follow it in temporal order.
  • Evaluation Protocol: For checklist-based tasks, checklists are shown only to explain the evaluation protocol and withheld from the input.This separates the displayed evaluation guidance from the information provided to the model.

C.1 Camera Control

Camera Control evaluates whether ordered camera motions follow prescribed directions and temporal order without unintended drift. The example requires a downward tilt, leftward pan, and leftward movement, occupying 0.27, 0.42, and 0.31 of the video, respectively.

  • C.1 Camera Control: Camera Control measures whether ordered camera motions follow prescribed directions and temporal order without unintended drift.The evaluated motion sequence consists of a downward tilt, a leftward pan, and leftward movement.
  • C.1 Camera Control: 0.27, 0.42, and 0.31 of the video are occupied by the three atomic camera controls, respectively.The controls correspond to tilting down, panning left, and moving left.

C.2 Subject Control

Subject Control evaluates whether a designated third-person subject can execute atomic directional controls in sequence while remaining visually identifiable. The video allocates 0.25 to Forward, 0.25 to Right, and 0.50 to Left.

  • Subject Control: Subject Control requires a designated third-person subject to move forward, right, and left in order while remaining visually identifiable.The task applies atomic controls to the subject.
  • Subject Control: 0.25 Forward, 0.25 Right, and 0.50 Left define the control durations in the video.The Forward, Right, and Left controls occupy 0.25, 0.25, and 0.50 of the video, respectively.

C.3 Scene Revisit … C.8 Goal Completion

The benchmark’s later tasks test whether generated worlds preserve scene state, infer unstated physical and social responses, and execute high-level goals. They progress from camera-memory consistency through terrain, object, social, autonomous physical, and goal-directed interactions.

  • C.3 Scene Revisit: Scene Revisit couples a round-trip camera motion with spatial memory, requiring return to the initial viewpoint while preserving a consistent scene.The camera tilts down and then up, so success requires both motion execution and recovery of the revisited view.
  • C.4 Terrain Interaction: Terrain Interaction specifies only horizontal motion, requiring the model to infer vertical adaptation for traversing stairs while continuing forward.The Forward control remains active throughout, while stair climbing is implied by the visible terrain rather than stated.
  • C.5 Object Interaction: Object Interaction evaluates whether contact with a designated target produces a physically consistent response, such as a cart pushing a lightweight sign aside.The checklist assesses approach and contact location, hinged-sign motion, visible mobility, continued pushing, and whether displacement enables onward movement.
  • C.6 Social Interaction: Social Interaction tests whether nearby pedestrians and a shopping cart respond plausibly when a sedan enters their crossing and safety space.Only the car’s Forward control is specified; evaluation examines changes in cart path or walking pace near the car’s front end.
  • C.7 Physical Reaction: Physical Reaction evaluates autonomous temporal development: a towel-loaded basket should rotate about the washer edge and fall while the woman remains stationary.The process must proceed without a new touch, grab, or bump, with the overhanging load moving outward and downward after support is lost.
  • C.8 Goal Completion: Goal Completion supplies a high-level goal, requiring scene grounding and coherent execution steps to loosen the blue toy car’s battery-compartment screw.Evaluation checks tool choice, correct target and alignment, screw-loosening motion, avoidance of distractors, and a visibly loosened or openable cover.
Loading 2608.02603v1…